1 to 25 of 59 Apache Spark Jobs in the UK

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

partnership with Inspire People, is hiring a Lead Data Engineer with strong SQL and Python expertise, experience across modern cloud data platforms including Databricks, Apache Spark and lakehouse architectures, and a proven ability to design, build and lead delivery of reliable data products. This is a permanent role … Strong proficiency in SQL and Python - Experience with modern cloud data platforms and lakehouse architectures - Hands-on experience with big data technologies such as Apache Spark or Databricks - Strong knowledge of data warehousing concepts - Experience with build, test and deployment automation - Proven leadership capability leading or mentoring data ...

requirements 3+ years of Python in production environments Strong AWS experience across core services Strong SQL skills for querying, transformation, and optimisation Experience with Apache Spark or serverless architectures Core responsibilities and skills Build and maintain real-time and batch ETL pipelines using Python on large, complex datasets … across large data volumes Model and transform data into well-structured, reusable datasets for downstream use Work with distributed data processing tools such as Apache Spark to handle high-volume workloads Improve existing pipelines by identifying bottlenecks and refactoring code and queries Develop and maintain data workflows, scheduling ...

Data Engineer, working with large-scale, complex datasets in regulated environments Scala development experience supporting analytics or data-processing solutions Hands-on experience with Apache Spark for large-scale distributed data processing Experience with model calibration, threshold management, and score explainability to meet AML governance and regulatory standards … Analyst or Data Engineer, working with large, complex datasets in regulated environments Solid Scala development experience, writing and supporting scalable data and analytics solutions Apache Spark experience, including batch and/or dynamic processing of high-volume data About Us FDM is an award-winning global leader ...

Python for data engineering tasks. * Experience with cloud platforms such as AWS, Azure, or GCP. * Hands-on experience with data technologies including Apache Spark, Databricks, Kafka, Airflow, Snowflake, and Hadoop ecosystems. * Experience designing and maintaining data warehouses and data lakes. * Knowledge of DevOps and CI/CD tooling ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

Collaborate with senior developers and quantitative teams on new analytics capabilities Participate in debugging, testing, and production support activities Technology Environment Python Databricks/Apache Spark SQL Pandas/NumPy Git REST APIs Cloud technologies (Azure or AWS) Numerix (or equivalent pricing/risk platform) Jupyter/notebook … commercial Python development experience Strong understanding of data structures and software engineering fundamentals Experience working with data engineering or analytics tooling Exposure to Databricks, Spark, or cloud-based data platforms Understanding of SQL and relational databases Excellent problem-solving and communication skills STEM or Computer Science degree preferred Nice ...

Data Engineer (Streaming) – Requirements Strong coding in Java (preferred) or Python Experience with Kafka, Flink, or Spark (Flink/Kafka ideal) Solid understanding of stream processing (event time, state, backpressure) Experience building ETL/ELT or real-time data pipelines Knowledge of microservices and distributed systems Tech & Cloud Experience ...

capacity planning, skills development, and vendor contributions The individual: Expertise in Azure data services (ADLS, ADF, Databricks, Synapse, Event Hub/Kafka), Python, SQL, Spark/PySpark. Experience with data modeling, metadata and lineage tooling; performance & cost optimization. Proven leadership in engineering teams; Agile/DevOps ways of working ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active SC Clearance ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will ...

promote DevOps. Support internal and external stakeholders, including customers and regulatory authorities. Tools, Systems & Continuous Improvement Manage and optimise the use of tools (e.g. Apache Spark, AWS Glue, Databricks or equivalent). Support system enhancements of data processing and transformation, migrations, and process improvements. Contribute to the development ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms (e.g., Snowflake, Redshift, BigQuery, Spark). Engineering Practices & Tooling Experience with CI/CD pipelines and automated build and deployment processes. Strong experience with version control systems (Git). ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms (e.g., Snowflake, Redshift, BigQuery, Spark). Engineering Practices & Tooling Experience with CI/CD pipelines and automated build and deployment processes. Strong experience with version control systems (Git). ...

Strong analytical and problem-solving skills Excellent communication and stakeholder engagement skills Technology Stack Python, SQL, Databricks, PyTorch, Transformers/LLMs, Prompt Engineering, Pandas, Spark, AWS, Azure, PostgreSQL, FastAPI, Git, CI/CD, Knowledge Graphs What's on Offer Fully remote UK-based role Opportunity to work on cutting ...

such as Docker and orchestration tools like Kubernetes Strong problem-solving skills and attention to detail Desirable Skills Experience with big data technologies (e.g., Spark, Hadoop) Knowledge of MLOps practices and tools (e.g., MLflow, Kubeflow) Experience working with NLP, computer vision, or recommendation systems Understanding of data governance ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...