1 to 25 of 113 Apache Spark Jobs in the UK

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Data Engineer Salary up to £60,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via chelsea.hackett@searchability.com ABOUT THE CLIENT Due to continued growth, we’re seeking a skilled Data … optimise scalable data pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

looking for a Test Engineer to join a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB. You will own testing across automated pipelines, data workflows, and cloud infrastructure - identifying risks, championing test … Development Identify, evaluate, and implement new test frameworks Enhance existing frameworks to improve testing confidence and coverage Pipeline & Data Testing Validate data pipelines using Apache Airflow, AWS Glue, Athena, and EMR Ensure data integrity, transformation accuracy, and performance under load Analyse data in multiple formats to validate new functionality ...

Software Developer Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required. The Role … looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data-driven applications and pipelines in a collaborative Dev and DevOps environment. The role focuses on writing and improving application code while working closely with DevOps ...

scale data processing. • Proficiency in Python/SQL for data engineering and scripting. • Familiarity with risk analytics environments or financial data. • Strong experience with Apache Spark (Spark Engine), including performance optimization and distributed data processing. • Strong experience ingesting data from diverse sources including relational databases, REST APIs ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

requirements 3+ years of Python in production environments Strong AWS experience across core services Strong SQL skills for querying, transformation, and optimisation Experience with Apache Spark or serverless architectures Core responsibilities and skills Build and maintain real-time and batch ETL pipelines using Python on large, complex datasets … across large data volumes Model and transform data into well-structured, reusable datasets for downstream use Work with distributed data processing tools such as Apache Spark to handle high-volume workloads Improve existing pipelines by identifying bottlenecks and refactoring code and queries Develop and maintain data workflows, scheduling ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

where needed. Qualification Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc) Experience architecting CI/CD and MLOps frameworks (Azure ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

experience in data engineering or large-scale data platforms Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark, with a preference for streaming technologies Solid understanding of real-time data processing concepts such as event time, state handling, and backpressure Experience building ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Master's degree in Computer Science, Engineering, Information Technology, or a related field. ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

Collaborate with senior developers and quantitative teams on new analytics capabilities Participate in debugging, testing, and production support activities Technology Environment Python Databricks/Apache Spark SQL Pandas/NumPy Git REST APIs Cloud technologies (Azure or AWS) Numerix (or equivalent pricing/risk platform) Jupyter/notebook … commercial Python development experience Strong understanding of data structures and software engineering fundamentals Experience working with data engineering or analytics tooling Exposure to Databricks, Spark, or cloud-based data platforms Understanding of SQL and relational databases Excellent problem-solving and communication skills STEM or Computer Science degree preferred Nice ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

analytical data modeling in BigQuery using dbt . Build reliable data pipelines utilizing SQL, Python, and distributed processing frameworks (Apache Spark, Scio, Beam, or Flink). Develop clear, story-driven dashboards using tools like Looker or Tableau . Champion data quality , implement CI/CD for dbt models ...

experience working with data warehouses. Solid understanding of ETL/ELT design and data pipeline optimisation. Experience with big data frameworks such as Apache Spark, and exposure to streaming technologies like Kafka. Strong problem-solving skills and the ability to collaborate effectively across teams. What's on offer ...

Microsoft Azure, Amazon Web Services, or Google Cloud Platform Familiarity with data visualization tools (e.g., Power BI, Tableau) Experience with big data technologies (e.g., Spark, Hadoop) Knowledge of MLOps practices and model deployment Understanding of UK data regulations (e.g., GDPR) Skills Machine Learning & Predictive Modeling Data Analysis & Statistical Modeling ...

experience building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

focus on production-quality code • Solid understanding of ETL processes and best practices • Experience with SQL/NoSQL databases and big data tools (e.g. Spark, Pulsar) • Hands-on experience with cloud platforms (AWS, Azure, or GCP) • Experience developing and deploying data services end-to-end • Strong stakeholder communication ...

focus on production-quality code • Solid understanding of ETL processes and best practices • Experience with SQL/NoSQL databases and big data tools (e.g. Spark, Pulsar) • Hands-on experience with cloud platforms (AWS, Azure, or GCP) • Experience developing and deploying data services end-to-end • Strong stakeholder communication ...