with Azure Data Factory, Azure Functions, and Synapse Analytics. Proficient in Python and advanced SQL, including query tuning and optimisation. Hands-on experience with big data tools such as Spark, Hadoop, and Kafka. Familiarity with CI/CD pipelines, version control, and deployment automation. Experience using Infrastructure as Code tools like Terraform. Solid understanding of Azure-based networking and More ❯
and delivering end-to-end AI/ML projects. Nice to Have: Exposure to LLMs (Large Language Models), generative AI , or transformer architectures . Experience with data engineering tools (Spark, Airflow, Snowflake). Prior experience in fintech, healthtech, or similar domains is a plus. More ❯
City of London, London, United Kingdom Hybrid / WFH Options
Hlx Technology
data infrastructure or data platforms, with proven ability to solve complex distributed systems challenges independently Expertise in large-scale data processing pipelines (batch and streaming) using technologies such as Spark, Kafka, Flink, or Beam Experience designing and implementing large-scale data storage systems (feature stores, timeseries databases, warehouses, or object stores) Strong distributed systems and infrastructure skills (Kubernetes, Terraform More ❯
data infrastructure or data platforms, with proven ability to solve complex distributed systems challenges independently Expertise in large-scale data processing pipelines (batch and streaming) using technologies such as Spark, Kafka, Flink, or Beam Experience designing and implementing large-scale data storage systems (feature stores, timeseries databases, warehouses, or object stores) Strong distributed systems and infrastructure skills (Kubernetes, Terraform More ❯
london, south east england, united kingdom Hybrid / WFH Options
Hlx Technology
data infrastructure or data platforms, with proven ability to solve complex distributed systems challenges independently Expertise in large-scale data processing pipelines (batch and streaming) using technologies such as Spark, Kafka, Flink, or Beam Experience designing and implementing large-scale data storage systems (feature stores, timeseries databases, warehouses, or object stores) Strong distributed systems and infrastructure skills (Kubernetes, Terraform More ❯
london (city of london), south east england, united kingdom Hybrid / WFH Options
Hlx Technology
data infrastructure or data platforms, with proven ability to solve complex distributed systems challenges independently Expertise in large-scale data processing pipelines (batch and streaming) using technologies such as Spark, Kafka, Flink, or Beam Experience designing and implementing large-scale data storage systems (feature stores, timeseries databases, warehouses, or object stores) Strong distributed systems and infrastructure skills (Kubernetes, Terraform More ❯
How did you hear about this job? Select Have you worked with Python/Java/Scala/Go? Select Do you have experience with Big Data Technologies like Spark, Databricks,andAirflow? Do you have any experience with Microservices architecture and API development? If yes, please describe briefly. What are you salary expectations? Do you require visa sponsorship to More ❯
practices for data infrastructure, fostering a culture of collaboration and knowledge sharing. (Required) Kubernetes and Orchestration: Manage and optimize Kubernetes clusters, specifically for running critical data processing workloads using Spark and Airflow. (Required) Cloud Security: Implement and maintain robust security measures, including cloud networking, IAM, encryption, data isolation, and secure service communication (VPC peering, PrivateLink, PSC/PSA). More ❯
e.g Pandas, Numpy, Scikit learn etc.) Have experience productionising machine learning models Are an expert in one of predictive modeling, classification, regression, optimisation or recommendation systems Have experience with Spark Have knowledge of DevOps technologies such as Docker and Terraform and ML Ops practices and platforms like ML Flow Have experience with agile delivery methodologies and CI/CD More ❯
We use a variety of 3rd party packages, including MLFlow, Seldon for ML model tracking and deployment, Kubernetes for hosting models, Argo and Git for CI/CD automation, Spark for big data processing. This is a rapidly changing field and we are deeply involved in open source community to help shape the technology evolution and are constantly looking … for components to adopt in order to enhance our platform. What you'll do: Develop across our evolving technology stack - we're using Python, Java, Kubernetes, ApacheSpark, Postgres, ArgoCD, Argo Workflow, Seldon, MLFlow and more. We are migrating into AWS cloud and adopting many services that are available in that environment. You will have the opportunity to … solutions. Strong analytical skills and desire to write clean, correct and efficient code. Sense of ownership, urgency and pride in your work. Experience with Python, Java, Docker, Kubernetes, Argo, Spark and AWS cloud services a plus. Exposure to Machine Learning practices a plus. We strive to create an accessible and inclusive experience for all candidates. If you need a More ❯
with libraries such as Pandas, NumPy, and FastAPI. Experience with weather and climate datasets and tooling (e.g., Copernicus, Xarray, Zarr, NetCDF). Experience with ETL tools and frameworks (e.g., Apache Airflow, Apache NiFi, Talend). Strong understanding of relational databases and SQL. Experience with cloud platforms (e.g., AWS, GCP, Azure) and their data services. Familiarity with data warehousing … skills and attention to detail. Excellent communication and collaboration skills. Preferred Qualifications: Experience with containerization and orchestration tools (e.g., Docker, Kubernetes). Knowledge of big data technologies (e.g., Hadoop, Spark). Experience in commodities (Agriculture, Natural Gas, Power). For more information about DRW's processing activities and our use of job applicants' data, please view our Privacy Notice More ❯
technologies, including Java, SQL Server/Snowflake databases, Python and C#. We are in the process of migrating more of our data to Snowflake, leveraging technologies like AWS Batch, Apache Flink and AWS Step functions for orchestration and Docker containers. These new systems will respond in real-time to events such as position and price changes, trades and reference … well as complex stored procedures and patterns, preferably in SQL Server. Snowflake Database experience can be valuable and would help the team in the data migration process. Knowledge of Apache Flink or Kafka highly desirable or similar technologies (e.g. ApacheSpark) Skills in C# WPF or Javascript GUI development beneficial, but not essential. Excellent communication skills. Mathematical. More ❯
Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, ApacheSpark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯
of the AEC industry and its specific data processing challenges Experience scaling ML training and data pipelines for large datasets Experience with distributed data processing and ML infrastructure (e.g., ApacheSpark, Ray, Docker, Kubernetes) Experience with performance optimization, monitoring, and efficiency in large-scale ML systems Experience with Autodesk or similar products (Revit, Sketchup, Forma) The Ideal Candidate More ❯
Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, ApacheSpark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯
to refine and monitor data collection systems using Scala and Java. Apply sound engineering principles such as test-driven development and modular design. Preferred Background Hands-on experience with Spark and Scala in commercial environments. Familiarity with Java and Python. Exposure to distributed data systems and cloud storage platforms. Experience designing data schemas and analytical databases. Use of AI More ❯
to refine and monitor data collection systems using Scala and Java. Apply sound engineering principles such as test-driven development and modular design. Preferred Background Hands-on experience with Spark and Scala in commercial environments. Familiarity with Java and Python. Exposure to distributed data systems and cloud storage platforms. Experience designing data schemas and analytical databases. Use of AI More ❯
and problem-solving skills Comfort working within a globally-distributed team A background in some of the following a bonus: Java experience Python experience Ruby experience Big data technologies: Spark, Trino, Kafka Financial Markets experience SQL: Postgres, Oracle Cloud-native deployments: AWS, Docker, Kubernetes Observability: Splunk, Prometheus, Grafana For more information about DRW's processing activities and our use More ❯
hold or gain a UK government security clearance. Preferred technical and professional experience Experience with machine learning frameworks (TensorFlow, PyTorch, scikit-learn). Familiarity with big data technologies (Hadoop, Spark). Background in data science, IT consulting, or a related field. AWS Certified Big Data or equivalent. IBM is committed to creating a diverse environment and is proud to More ❯
experience of utilising strong communication and stakeholder management skills when engaging with customers Significant experience of coding (ideally Python) Experience with big data processing tools such as Hadoop or Spark Cloud experience; GCP specifically in this case, including services such as Cloud Run, Cloud Functions, BigQuery, GCS, Secret Manager, Vertex AI etc. Experience with Terraform Prior experience in a More ❯
communication and stakeholder management skills when engaging with customers Significant experience of coding in Python and Scala or Java Experience with big data processing tools such as Hadoop or Spark Cloud experience; GCP specifically in this case, including services such as Cloud Run, Cloud Functions, BigQuery, GCS, Secret Manager, Vertex AI etc. Experience with Terraform Prior experience in a More ❯
programming languages such as Python, Java and Scala, and experience with ML frameworks like TensorFlow, PyTorch, and scikit-learn. Experience with cloud platforms (e.g., AWS), big data technologies (e.g., Spark) as well as other technologies used to deploy models to production (e.g., Kubernetes, GHA, Airflow, Docker etc.). Accommodation requests If you need assistance with any part of the More ❯
large structured and unstructured data sets. Hands on experience with the set up and maintenance of bronze, silver, and gold layers in a big data platform, ideally Databricks and Apache Spark. And experience building and maintaining DBT models. A desire and passion for transforming raw data into structured tables which can answer common business questions. Proven experience working with More ❯
dedicated to building and maintaining our cutting-edge media data lake on Microsoft Azure. This role will focus on developing and supporting data pipelines within a medallion architecture, utilizing Spark and Scala to process and transform large volumes of media data. The successful candidate will be passionate about data, eager to learn, and contribute to a high-performing engineering … WILL DO Develop, test, and deploy data ingestion, transformation, and processing pipelines using Azure services (Azure Data Factory, Azure Data Lake Storage). Write efficient and maintainable code in Spark and Scala for data manipulation and analysis. Contribute to the implementation and maintenance of the medallion architecture (Bronze, Silver, Gold layers). Collaborate with senior engineers, architects, and analysts … Creativity. Create an Open environment by Balancing People and Client Experiences by Cultivating Trust . Lead Optimistically by Championing Growth and Development to Mobilize the Enterprise . Proficiency in ApacheSpark and Scala programming. Experience with cloud platforms, preferably Microsoft Azure (Azure Data Factory, ADLS Gen2, Azure Synapse Analytics). Understanding of ETL/ELT processes and data More ❯
