1 to 25 of 119 Apache Spark Jobs in the UK excluding London

deliver reliable, high-performance data solutions using Databricks and modern cloud technologies. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Delta Lake Build and optimize ETL/ELT workflows for batch and streaming data Develop data models to support analytics, reporting … data pipelines, architectures, and best practices Required Qualifications 3+ years of experience in data engineering or a similar role Strong experience with Databricks and Apache Spark Proficiency in Python and SQL Experience with Delta Lake, data modelling, and performance tuning Hands-on experience with cloud platforms (AWS, Azure ...

keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue, EMR, or Spark. Ensure high … data Billing & revenue assurance Build models and pipelines to support network analytics, customer 360, churn prediction, fraud detection , etc. 7. Performance Optimization & Monitoring Tune Spark/Glue jobs for performance and cost. Monitor Redshift/Athena/S3 efficiency and implement best practices. Perform data quality checks and validation ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

Experience building and deploying cloud data platforms Infrastructure-as-Code and CI/CD practices (GitHub, CodeBuild, pipelines) Strong coding experience in Python (Pandas), Apache Spark, and/or Apache Iceberg Expertise in SQL and advanced data manipulation and analysis Experience with data warehouse design and data ...

delivering large-scale data integration, engineering, and analytics projects Strong expertise in cloud platforms (e.g. Azure, MS Fabric, AWS) and modern data tools (e.g. Apache Spark, Hadoop) Proficiency in programming languages (e.g. Python, SQL, Java) Extensive experience in stakeholder management and collaboration across health and local government ...

architectures. Hands‐on experience with Microsoft Fabric (Data Engineering, Data Science and Real‐Time Analytics) and Azure ML, including MLflow. Strong proficiency with Apache Spark, Azure Data Factory, Git, Python and SQL Experience developing, training and deploying machine learning models using TensorFlow, PyTorch or Scikit‐learn. Solid understanding ...

architectures. Hands‐on experience with Microsoft Fabric (Data Engineering, Data Science and Real‐Time Analytics) and Azure ML, including MLflow. Strong proficiency with Apache Spark, Azure Data Factory, Git, Python and SQL Experience developing, training and deploying machine learning models using TensorFlow, PyTorch or Scikit‐learn. Solid understanding ...

architectures. Hands‐on experience with Microsoft Fabric (Data Engineering, Data Science and Real‐Time Analytics) and Azure ML , including MLflow. Strong proficiency with Apache Spark, Azure Data Factory, Git, Python and SQL Experience developing, training and deploying machine learning models using TensorFlow, PyTorch or Scikit‐learn. Solid understanding ...

share knowledge with team members. Required Qualifications Proven experience in designing and building data pipelines (ETL/ELT) using modern technologies (Python, SQL, Spark, Airflow, etc.). Strong analytical skills with the ability to interpret complex data and deliver business value. Experience integrating data from multiple sources and systems. ...

Java. Hands on experience with at least one major cloud platform (Azure, AWS or GCP). Exposure to big data technologies such as Spark or Hadoop. Experience with ETL/orchestration tools like Airflow, Talend or Azure Data Factory. Understanding of data warehousing solutions (Snowflake, BigQuery, Redshift or similar ...

similarly regulated industry. Strong coding skills in Python or Java (Spring Boot); React experience is a plus. Proficiency with modern data tools: Airflow, Spark, Kafka, dbt, Snowflake or similar. Experience with cloud platforms (AWS, Azure, GCP), containerization (Docker, Kubernetes), and CI/CD. Data Quality: Proven ability to validate ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services (AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage). Exposure to workflow orchestration tools (Apache Airflow, Prefect, or Dagster). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

Charity Volunteer Days Cycle to work scheme And More.. DATA ENGINEER – ESSTENTIAL SKILLS Proven experience building data pipelines using Databricks . Strong understanding of Apache Spark (PySpark or Scala) and Structured Streaming . Experience working with Kafka (MSK) and handling real-time data . Good knowledge of Delta ...

lakehouse and datalake architectures Familiarity with modern data engineering patterns (ETL/ELT, medallion architecture, observability) Excellent Python and SQL skills Experience working with Apache Spark and large-scale data processing For a full consultation on this exciting new role, please get in touch with ARC IT Recruitment ...

individuals’ desires and capabilities. Essential Skills Proven experience as a Data Architect (or similar/related role) Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience with Big Data Experience with Enterprise level Data sets Experience of using programming languages such ...

experience and familiarity with frameworks (e.g. Flask, Django) Experience building and orchestrating ETL pipelines (e.g. Airflow, Luigi, Argo) Exposure to big data technologies (e.g. Spark, Kafka, Hadoop) Experience working in Agile environments Solid understanding of data architecture and data quality best practices ...

experience and familiarity with frameworks (e.g. Flask, Django) Experience building and orchestrating ETL pipelines (e.g. Airflow, Luigi, Argo) Exposure to big data technologies (e.g. Spark, Kafka, Hadoop) Experience working in Agile environments Solid understanding of data architecture and data quality best practices ...

deliver intelligent data products, leveraging modern cloud and AI technologies. Key Responsibilities Design and implement robust data pipelines and ML workflows using Python, SQL, Spark, and Databricks. Develop and deploy machine learning models (including NLP, deep learning, and agentic AI) in production environments. Integrate data from diverse sources, including ...

technology (AWS or Azure). Strong experience with creation of data ingestion pipeline and ET process. Strong knowledge of big data tool such as Spark, Databricks and Python. Strong understanding of common machine learning techniques and frameworks (e.g. mlflow). Strong knowledge of Natural language processing (NPL) concepts. Strong ...

modern cloud engineering practices. Wherever you land, youll be working with some of the biggest datasets in the UK using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Design and deploy machine learning models for fraud detection, credit … risk, customer segmentation, and behavioural analytics using scalable frameworks like TensorFlow, PyTorch, and XGBoost. Engineer robust data pipelines and ML workflows using Apache Spark, Vertex AI, and CI/CD tooling to ensure seamless model delivery and monitoring. Apply advanced techniques in deep learning, natural language processing ...

modern cloud engineering practices. Wherever you land, youll be working with some of the biggest datasets in the UK using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Design and deploy machine learning models for fraud detection, credit … risk, customer segmentation, and behavioural analytics using scalable frameworks like TensorFlow, PyTorch, and XGBoost. Engineer robust data pipelines and ML workflows using Apache Spark, Vertex AI, and CI/CD tooling to ensure seamless model delivery and monitoring. Apply advanced techniques in deep learning, natural language processing ...

modern cloud engineering practices. Wherever you land, youll be working with some of the biggest datasets in the UK using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Design and deploy machine learning models for fraud detection, credit … risk, customer segmentation, and behavioural analytics using scalable frameworks like TensorFlow, PyTorch, and XGBoost. Engineer robust data pipelines and ML workflows using Apache Spark, Vertex AI, and CI/CD tooling to ensure seamless model delivery and monitoring. Apply advanced techniques in deep learning, natural language processing ...

modern cloud engineering practices. Wherever you land, youll be working with some of the biggest datasets in the UK using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Design and deploy machine learning models for fraud detection, credit … risk, customer segmentation, and behavioural analytics using scalable frameworks like TensorFlow, PyTorch, and XGBoost. Engineer robust data pipelines and ML workflows using Apache Spark, Vertex AI, and CI/CD tooling to ensure seamless model delivery and monitoring. Apply advanced techniques in deep learning, natural language processing ...

modern cloud engineering practices. Wherever you land, youll be working with some of the biggest datasets in the UK using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Design and deploy machine learning models for fraud detection, credit … risk, customer segmentation, and behavioural analytics using scalable frameworks like TensorFlow, PyTorch, and XGBoost. Engineer robust data pipelines and ML workflows using Apache Spark, Vertex AI, and CI/CD tooling to ensure seamless model delivery and monitoring. Apply advanced techniques in deep learning, natural language processing ...

modern cloud engineering practices. Wherever you land, youll be working with some of the biggest datasets in the UK using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Design and deploy machine learning models for fraud detection, credit … risk, customer segmentation, and behavioural analytics using scalable frameworks like TensorFlow, PyTorch, and XGBoost. Engineer robust data pipelines and ML workflows using Apache Spark, Vertex AI, and CI/CD tooling to ensure seamless model delivery and monitoring. Apply advanced techniques in deep learning, natural language processing ...