1 to 25 of 62 Apache Spark Jobs in the UK

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

evolution of data architecture and help shape the organisations data engineering capabilities. Key Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark and Python. Develop and maintain robust ELT processes to ingest, transform and integrate data from multiple sources. Contribute to the design and evolution … platform performance. Essential Skills & Experience Strong hands-on experience with Databricks . Proven experience designing, implementing and optimising data pipelines. Strong knowledge of Apache Spark and Python . Understanding of Lakehouse architecture and modern data engineering principles. At least five years experience across ELT, data engineering and data ...

Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platform Develop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta Lake Contribute to platform architecture, engineering standards and best practices across the data estate Work with Product, Analytics and Data Science … continuous improvement across the engineering team Job Requirements: Significant experience building and supporting scalable, production-grade data platforms Strong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQL Good understanding of cloud-based data engineering, ideally within Microsoft Azure environments Experience with Infrastructure as Code ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

hands-on Senior Databricks Solutions Architect to support a complex, high-performance data platform within a global organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements … Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

Senior Data Engineer Role (Python/Spark/Databricks) - Hybrid - Contract - Banking Role - Senior Data Engineer Rate - £850 p/d (Inside IR35) Duration - 6 months with very likely extension Location - Hybrid/Liverpool Street (London) - 3 days per week in a Liverpool Street office About the Role Global … delivering critical data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands-on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud-native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version control, CI/CD, testing ...

Python and SQL; APIs; test automation; data structures; distributed processing; Git; CI/CD; containers; infrastructure awareness; production debugging and performance optimization. Data engineering: Spark or Databricks, Kafka or equivalent streaming, ETL/ELT, batch and real-time pipelines, data contracts, lineage, cataloguing, quality rules, and scalable cloud storage. ...

collaboration skills. Desirable Experience Experience with AWS services such as S3, Glue, Lambda, Step Functions, Redshift or related data engineering technologies. Familiarity with Apache Spark, Airflow, distributed processing, data lakes and data warehouses. Experience with infrastructure as code, Docker or automated deployment pipelines. Background in government, defence, national ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic ...