51 to 75 of 199 Apache Spark Jobs in England

Python and SQL Hands-on experiences with Data Architecture, including: Cloud platforms and orchestration tools (e.g. Dagster, Airflow) AI/MLOps: Model deployment, monitoring, lifecycle management. Big Data Processing: Spark, Databricks, or similar. Bonus: Knowledge Graph engineering, graph databases, ontologies. Located in London And ideally you... Are a zero-to-one builder who thrives on autonomy and ambiguity. Are More ❯

Python data science stack Knowledge of OO programming, software design, i.e., SOLID principles, and testing practices. Knowledge and working experience of AGILE methodologies. Proficient with SQL. Familiarity with Databricks, Spark, geospatial data/modelling and insurance are a plus. Exposure to MLOps, model monitoring principles, CI/CD and associated tech, e.g., Docker, MLflow, k8s, FastAPI etc are desirable More ❯

and orchestration tools like Kubernetes. Familiarity with Infrastructure as Code (IaC) tools such as Terraform or CloudFormation. Knowledge of data engineering and experience with big data technologies like Hadoop, Spark, or Kafka. Experience with CI/CD pipelines and automation, such as using Jenkins, GitLab, or CircleCI. As an equal opportunities employer, we welcome applications from individuals of all More ❯

e.g Pandas, Numpy, Scikit learn etc.) Have experience productionising machine learning models Are an expert in one of predictive modeling, classification, regression, optimisation or recommendation systems Have experience with Spark Have knowledge of DevOps technologies such as Docker and Terraform and ML Ops practices and platforms like ML Flow Have experience with agile delivery methodologies and CI/CD More ❯

scalable pipelines, data platforms, and integrations, while ensuring solutions meet regulatory standards and align with architectural best practices. Key Responsibilities: Build and optimise scalable data pipelines using Databricks and Apache Spark (PySpark). Ensure performance, scalability, and compliance. Collaborate on requirements, design, and backlog refinement. Promote engineering best practices including CI/CD, code reviews, and testing. Research … experience: Experience with efficient, reliable data pipelines that improve time-to-insight. Knowledge of secure, auditable, and compliant data workflows. Know how on optimising performance and reducing costs through Spark and Databricks tuning. Be able to create reusable, well-documented tools enabling collaboration across teams. A culture of engineering excellence driven by mentoring and high-quality practices. Preferred Experience … Databricks in a SaaS environment, Spark, Python, and database technologies. Event-driven and distributed systems (Kafka, AWS SNS/SQS, Java, Python). Data Governance, Data Lakehouse/Data Intelligence platforms. AI software delivery and AI data preparation. More ❯

able to work across full data cycle. Proven Experience working with AWS data technologies (S3, Redshift, Glue, Lambda, Lake formation, Cloud Formation), GitHub, CI/CD Coding experience in Apache Spark, Iceberg or Python (Pandas) Experience in change and release management. Experience in Database Warehouse design and data modelling Experience managing Data Migration projects. Cloud data platform development … the AWS services like Redshift, Lambda,S3,Step Functions, Batch, Cloud formation, Lake Formation, Code Build, CI/CD, GitHub, IAM, SQS, SNS, Aurora DB Good experience with DBT, Apache Iceberg, Docker, Microsoft BI stack (nice to have) Experience in data warehouse design (Kimball and lake house, medallion and data vault) is a definite preference as is knowledge of … other data tools and programming languages such as Python & Spark and Strong SQL experience. Experience is building Data lake and building CI/CD data pipelines A candidate is expected to understand and can demonstrate experience across the delivery lifecycle and understand both Agile and Waterfall methods and when to apply these. Experience This position requires several years of More ❯

UK and potentially international sources, enabling advanced analytics and data-driven insights to inform strategic decisions. This role is highly technical and data-focused, requiring strong experience in Databricks , Spark , and related data engineering tools. While stakeholder engagement is minimal, you will need to align technical solutions with business objectives and ensure data integrity and compliance throughout. Key Requirements … Active SC Clearance (used within the last 12 months) Proven experience with Databricks (including notebooks, clusters, and job orchestration) Strong knowledge of Apache Spark , PySpark , and distributed data processing Experience building and optimising ETL pipelines and data workflows Familiarity with Delta Lake , SQL , and data modelling best practices Ability to work with large, complex datasets from multiple sources More ❯

for best practice and technical excellence and be a person that actively looks for continual improvement opportunities. Knowledge and skills Experience as a Data Engineer or Analyst Databricks/Apache Spark SQL/Python BitBucket/GitHub. Advantageous dbt AWS Azure Devops Terraform Atlassian (Jira, Confluence) About Us What's in it for you... Healthcare plan, life assurance More ❯

requirements and provide technical guidance. Troubleshoot and resolve complex issues related to Databricks infrastructure. Essential Skills & Experience Proven experience working with Databricks in a similar role. Strong knowledge of Spark, Scala, Python, and SQL. Experience with cloud platforms such as AWS or Azure. Ability to work collaboratively in a fast-paced environment. Excellent problem-solving and communication skills. Desirable More ❯

GCP including BigQuery, Pub/Sub, Cloud Composer and IAM You have strong Python, SQL and PySpark skills You have experience with real-time data streaming using Kafka or Spark You have a good knowledge of Data Lakes, Data Warehousing, Data Modelling You're familiar with DevOps principles, containerisation and CI/CD tools such as Jenkins or GitHub More ❯

data reliability, scalability, and performance across the platform 🧩 The Tech Environment You’ll likely be working with some combination of: Languages: Python, Scala, Go Streaming: Kafka/Flink/Spark Structured Streaming Workflow orchestration: Airflow/Prefect/Dagster Data storage & processing: Snowflake/Detabricks/BigQuery/Redshift Infrastructure: Docker/Kubernetes/Terraform/dbt Monitoring: Prometheus More ❯

data reliability, scalability, and performance across the platform 🧩 The Tech Environment You’ll likely be working with some combination of: Languages: Python, Scala, Go Streaming: Kafka/Flink/Spark Structured Streaming Workflow orchestration: Airflow/Prefect/Dagster Data storage & processing: Snowflake/Detabricks/BigQuery/Redshift Infrastructure: Docker/Kubernetes/Terraform/dbt Monitoring: Prometheus More ❯

stakeholders in a fast-paced environment Experience in the design and deployment of production data pipelines from ingestion to consumption within a big data architecture, using Java, Python, Scala, Spark, SQL. Experience performing tasks such as writing scripts, extracting data using APIs, writing SQL queries etc. Experience in processing large amounts of structured and unstructured data, including integrating data More ❯

coding standards, and documentation to improve data engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance standards. Work with cloud platforms … CI/CD processes for data pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ETL/ELT development and orchestration … tools (e.g., Apache Airflow, dbt, Prefect). Knowledge of data modelling, data warehousing, and lakehouse architectures. Familiarity with DevOps practices, CI/CD pipelines, and infrastructure-as-code. Strong problem-solving skills and the ability to work in fast-paced environments. Excellent communication and stakeholder management skills. Preferred Qualifications: Experience with machine learning data pipelines and MLOps practices. Knowledge More ❯

coding standards, and documentation to improve data engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance standards. Work with cloud platforms … CI/CD processes for data pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ETL/ELT development and orchestration … tools (e.g., Apache Airflow, dbt, Prefect). Knowledge of data modelling, data warehousing, and lakehouse architectures. Familiarity with DevOps practices, CI/CD pipelines, and infrastructure-as-code. Strong problem-solving skills and the ability to work in fast-paced environments. Excellent communication and stakeholder management skills. Preferred Qualifications: Experience with machine learning data pipelines and MLOps practices. Knowledge More ❯

teams to build scalable data pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation Contribute to containerised deployments (Docker/… within Agile teams and support DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/Kubernetes) is a plus Excellent communication and stakeholder engagement skills Experience working within More ❯

teams to build scalable data pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation Contribute to containerised deployments (Docker/… within Agile teams and support DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/Kubernetes) is a plus Excellent communication and stakeholder engagement skills Experience working within More ❯

a focus on having delivered in Microsoft Azure Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure Data Engineering certifications Databricks certifications What's in it for you: 📍Location: London More ❯

a focus on having delivered in Microsoft Azure Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure Data Engineering certifications Databricks certifications What's in it for you: 📍Location: London More ❯

best practices for data security and compliance. Collaborate with stakeholders and external partners. Skills & Experience: Strong experience with AWS data technologies (e.g., S3, Redshift, Lambda). Proficient in Python, Apache Spark, and SQL. Experience in data warehouse design and data migration projects. Cloud data platform development and deployment. Expertise across data warehouse and ETL/ELT development in More ❯

Proficiency in one or more programming languages including Java, Python, Scala or Golang. Experience with columnar, analytical cloud data warehouses (e.g., BigQuery, Snowflake, Redshift) and data processing frameworks like Apache Spark is essential. Experience with cloud platforms like AWS, Azure, or Google Cloud. Strong proficiency in designing, developing, and deploying microservices architecture, with a deep understanding of inter More ❯

It, Express>It, Metadata Hub, and PDL. Hands-on experience with SQL , Unix/Linux shell scripting , and data warehouse concepts . Familiarity with big data ecosystems (Hadoop, Hive, Spark) and cloud platforms (AWS, Azure, GCP) is a plus. Proven ability to troubleshoot complex ETL jobs and resolve performance issues. Experience working with large-scale datasets and enterprise data More ❯

management skills and the ability to influence A drive to learn new technologies and techniques Experience/aptitude towards research and openness to learn new technologies Experience with Azure, Spark (PySpark) Kubeflow - desirable Benefits We pay competitive salaries based on experience of the candidates, along with this, you will be entitled to an award-winning range of benefits including More ❯

science team, mentoring junior colleagues and driving technical direction. Experience working with Agile methodologies in a collaborative team setting. Extensive experience with big data tools, such as Hadoop and Spark, for managing and processing large-scale datasets. Extensive experience with cloud platforms, particularly Microsoft Azure, for building and deploying data science solutions. Why Join? You'll be joining a More ❯

analysis hierarchically. Robust knowledge of statistical concepts, accompanied by expertise with a set of analytical tools ranging from databases (e.g. SQL, BigQuery, noSQL) to programming languages (e.g. R, Python, Spark), and from data visualisation (e.g. PowerBI, Tableau) to machine learning. Deep expertise in advanced analytics, data science and traditional analytics. High-level understanding of data engineering and data strategy. More ❯