26 to 50 of 138 Apache Spark Jobs in England

Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/or Amazon Athena ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure as Code. Git/GitLab and modern software development practices. ...

Data Engineer (Streaming) – Requirements Strong coding in Java (preferred) or Python Experience with Kafka, Flink, or Spark (Flink/Kafka ideal) Solid understanding of stream processing (event time, state, backpressure) Experience building ETL/ELT or real-time data pipelines Knowledge of microservices and distributed systems Tech & Cloud Experience ...

Knowledge of data governance, security, and access frameworks Experience working in cloud environments (e.g., Azure, AWS, or GCP) Familiarity with big data tools (e.g., Spark, Databricks) Key Competencies Problem-solving mindset with strong analytical skills Ability to work collaboratively across multiple teams Strong communication and stakeholder management skills Proactive ...

years (Senior) of data engineering experience Hands-on experience with Palantir Foundry and/or Foundry Data Platform (FDP) Strong proficiency in: Python SQL Spark (PySpark or SparkSQL) Experience building production data pipelines in cloud environments (AWS, Azure, or GCP) Understanding of data modeling, schema design, and metadata management ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

disparate sources to build meaningful datasets for analytics and reporting, focusing on consolidating data from various systems. Implement big-data technologies (e.g., Redshift, EMR, Spark, SNS, SQS, Kinesis) to optimize processing of large datasets. Develop and maintain the team's data platform, including infrastructure-as-code using AWS CDK. … permissions Experience with non-relational databases/data stores (object storage, document or key-value stores, graph databases, column-family databases) Experience with Apache Spark/Elastic Map Reduce ...

clouds (AWS, Azure, GCP) and cloud data warehousing solutions like Snowflake and Databricks. Certifications with each are a plus. Understanding of SQL, Python, R, Apache Spark or similar product or language are also a plus.* Strong knowledge of data analytics, data science, ETL processes, AI/ML workflows ...

with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark or other big-data tools Hybrid working – 3 days per week in a central London office (Tues - Thurs) If you’re excited ...

capacity planning, skills development, and vendor contributions The individual: Expertise in Azure data services (ADLS, ADF, Databricks, Synapse, Event Hub/Kafka), Python, SQL, Spark/PySpark. Experience with data modeling, metadata and lineage tooling; performance & cost optimization. Proven leadership in engineering teams; Agile/DevOps ways of working ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

multiples sources (APIs, BDD, streams, fichiers). Qualité, Performance & Fiabilité Déployer des mécanismes de Data Quality (contrôles, monitoring, alerting). Optimiser les traitements Spark (partitioning, tuning, formats de données). Assurer la supervision et la résolution des incidents en production. CI/CD & Gouvernance Industrialiser les développements ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

strong focus on scalability, resilience, and performance. Develop and optimise scalable data pipelines , supporting both batch and streaming workloads, using technologies such as Apache Spark . Work extensively with data technologies , leveraging Python and SQL to deliver high-quality analytical and data-driven solutions. Lead the design ...

experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active SC Clearance ...

Software Engineer on large software applications. Proficient in technologies including Python, REST, PyTorch, TensorFlow, Docker, FastAPI, Selenium, React, TypeScript, Redux, GraphQL, Kafka, and Apache Spark. Experience working with one or more of the following database systems: DynamoDB, DocumentDB, MongoDB. Demonstrated expertise in unit testing and tools such as JUnit ...

have experience and expertise in following technologies: Java, Python, Data engineering, Data and Database engineering, Agentic AI, Gen AI, Machine Leaning, Generative AI, Scripting, Spark, Hadoop, SQL, NoSQL, PyTorch and one of the cloud technologies like AWS, Azure or GCP. Should have worked in at least two or more ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

improve data engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance … data pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will ...

looking for Strong Python + SQL (essential) 5+ years in data engineering/platform engineering AWS experience in production data environments Hands-on with Spark/Kafka/Snowflake/Databricks (or similar) Experience with data lakes/lakehouse/distributed systems Background in financial services, trading, or quantitative ...

scalable data pipelines, managing large datasets, and enabling engineering and AI teams to deliver high-performance data-driven products. Key Technical Requirements • Python • SQL • Spark • Airflow • Data pipelines • AWS or GCP Experience Required • 4+ years in data engineering • Experience working with large-scale structured and unstructured datasets • Strong ...

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...