Python and SQL; strong experience with data engineering libraries (e.g., Pandas, PySpark, Dask). Deep knowledge of ETL/ELT frameworks and orchestration tools (e.g., Airflow, Azure Data Factory, Dagster). Proficient in cloud platforms (preferably Azure) and services such as Data Lake, Synapse, Event Hubs, and Functions. Authoring reports and dashboards with either open source or commercial products More ❯
Python and SQL; strong experience with data engineering libraries (e.g., Pandas, PySpark, Dask). Deep knowledge of ETL/ELT frameworks and orchestration tools (e.g., Airflow, Azure Data Factory, Dagster). Proficient in cloud platforms (preferably Azure) and services such as Data Lake, Synapse, Event Hubs, and Functions. Authoring reports and dashboards with either open source or commercial products More ❯
together data from all kinds of sources, whether that's batch files or real-time streams. You'll have set up and worked with ETL and ELT tools like Dagster, AWS Glue, Azure Data Factory, Airflow or dbt, and you can decide what tools are right for the job. You'll have an understanding of how Node.js and TypeScript More ❯
and development of a new global data platform (PaaS) Ensure scalable storage solutions (data lakes, data warehouses) to handle structured and unstructured data. Implement ETL/ELT pipelines using Dagster, Airflow, or similar tools. Optimize performance and scalability for large data volumes. Govern data security, compliance, and access controls. Development & DevOps: Strong programming and scripting skills in Python. Knowledge More ❯
and rebuilding our data platform to ensure we stay ahead in a rapidly evolving landscape. You'll be hands-on, using the latest tools like Apache Arrow, Polars, and Dagster to craft a data architecture that aligns with our innovative investment strategies. In this position, you'll be deeply involved in building and optimizing data pipelines that are reliable More ❯
Defender suite, Microsoft Entra, Microsoft Purview and Azure security services (e.g. Key Vault, NSGs, WAF, etc.). Experience with Kubernetes (Azure Kubernetes Service) and data platforms (e.g. Databricks, Snowflake, Dagster). Proven understanding of security risk management. Excellent understanding of common security controls, in particular cloud security controls. Understanding of threat modelling. Knowledge of ISO 27001 and other commonly More ❯
At least 4 years of experience in industry working with the above tools and technologies. Nice to Have Experience with other parts of our tech stack (Clickhouse, Postgres, Datadog, Dagster, Temporal, AWS, Gitlab and RunAI) Any domain knowledge within Life Sciences and Biotech/Techbio, specifically around high throughput sequencing, ML models, and bioinformatics. What we can offer in More ❯
In details, the position encompasses duties and responsibilities as follows: An experienced Data Engineer is required for the Surveillance IT team to develop ingestion pipelines and frameworks across the application portfolio, supporting Trade Surveillance analysts with strategy and decision-making. More ❯
robust data pipelines, automate data cleaning, and deliver scalable, production-grade Python code using best practices such as typing, testing, and modular design. Integrate models into orchestration frameworks (e.g. Dagster) and cloud-based deployment pipelines following CI/CD principles. Use analysis and visualisation to identify high-value opportunities, measure product adoption, and quantify business impact. Partner with stakeholders … experiment tracking (e.g. MLflow), and containerisation (e.g. Docker). Familiarity with CI/CD tools (e.g. GitHub Actions), model/data versioning (e.g. DVC), and orchestration frameworks (e.g. Airflow, Dagster). Skilled in testing (unit, integration, end-to-end), and visualising outcomes with tools like seaborn. Ability to translate complex business problems into data science solutions and effectively communicate More ❯
We are looking for brilliant engineers to join our team at Magentic. We're pushing the boundaries of AI with next-generation agentic systems that can manage entire workflows. We're focusing on a three trillion dollar market of supply More ❯
