1 to 25 of 48 Apache Spark Jobs in the UK excluding London

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google ...

aiming for a career in data engineering, it can feel like youre staring at a never-ending list of tools and technologies SQL, Python, Spark, Kafka, Airflow, dbt, Snowflake, Redshift, Terraform, Kubernetes, and the list goes on. Scroll job boards and LinkedIn, and its easy to conclude that unless ...

consultancy or client delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality ...

platforms (AWS, GCP, or Azure) Experience with data warehousing and lake architectures ETL/ELT pipeline development SQL and NoSQL databases Distributed computing frameworks (Spark, Kinesis etc) Software development best practices including CI/CD, TDD and version control Strong understanding of data modelling and system architecture Excellent problem ...

optimising ETL/ELT pipelines. Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures. Experience with Azure Data Factory, Azure Synapse, and Apache Spark. Knowledge of data modelling, semantic models, and enterprise reporting solutions. Experience implementing data governance, security, and quality frameworks. Strong understanding of modern data engineering ...

range of projects that include internal and external suppliers. Essential Skills Proven experience as a Data Engineer Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience of using programming languages such as Python, Scala and SQL. Demonstrable knowledge of data modelling ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

Strong background in data engineering and software development Proven experience working with Databricks (essential) Hands-on expertise with tools such as Python, PySpark, Spark, and SQL Experience building and maintaining data pipelines and data platforms Solid understanding of data modelling and data architecture principles Experience working with cloud platforms ...

Strong background in data engineering and software development Proven experience working with Databricks (essential) Hands-on expertise with tools such as Python, PySpark, Spark, and SQL Experience building and maintaining data pipelines and data platforms Solid understanding of data modelling and data architecture principles Experience working with cloud platforms ...

prem environments. Hard Skills Experience in data engineering, ideally aerospace/defence or industrial sectors. ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka). Strong Python and SQL; C++, C#, or MATLAB a plus. Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB. Data Platform Capabilities ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will ...

modern data engineering best practices You have experience with Databricks, ideally also Azure You have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and Pandas You're excited to join a start-up in a role that you can shape and influence ...

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...

Senior Data Engineer (Scala/Spark) London | Hybrid Working | Initial 12 month contract | £450-550pd PAYE + holiday pay We are partnering with a world-leading technology organisation seeking an experienced Senior Data Engineer to join a high-performing engineering team responsible for building and operating large-scale … machine learning workflows at scale. As a Senior Data Engineer, you will design, build and operate large-scale batch data pipelines using Scala and Apache Spark. Working alongside Machine Learning Engineers, Researchers and Platform Engineers, you will help develop the infrastructure and tooling required to process vast datasets ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

data modelling and processing but focuses on practical ability to implement solutions using a range of technologies and languages including SQL, Python, Databricks, Apache Spark, Lakehouse and various RDBMS. This is a senior, hands-on role. You will develop and support more junior team members whilst working ...

scaling data platforms, with a strong focus on Azure Cloud technologies (e.g., Azure Data Factory, Azure Synapse, Azure Data Lake, Azure SQL Database, Databricks, Apache Spark). Proven experience leading the implementation of data platform strategies and managing large‐scale data infrastructure initiatives using Azure. Demonstrated experience implementing ...

/ELT design and implementation) Strong SQL & Data Modeling (query optimization, schema design) Programming Skills (Python/Scala for data processing) Big Data Processing (Spark/Databricks) Cloud Data Platforms (Azure/AWS/GCP - especially data services) Secondary skills API-based Data Integration & Microservices exposure Banking/Financial ...

monitoring Experience with cloud platforms (e.g. AWS, Azure, or GCP) and containerisation (Docker, Kubernetes) Experience building data pipelines and working with large datasets (SQL, Spark or similar) Strong understanding of machine learning algorithms and statistical modelling techniques Experience collaborating across data science and engineering teams Machine Learning Engineer, Python ...

/ELT design and implementation) Strong SQL & Data Modeling (query optimization, schema design) Programming Skills (Python/Scala for data processing) Big Data Processing (Spark/Databricks) Cloud Data Platforms (Azure/AWS/GCP - especially data services ...

/ELT design and implementation) Strong SQL & Data Modeling (query optimization, schema design) Programming Skills (Python/Scala for data processing) Big Data Processing (Spark/Databricks) Cloud Data Platforms (Azure/AWS/GCP - especially data services) Secondary s click apply for full job details ...