1 to 25 of 69 Apache Spark Jobs in London

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend ...

complex legacy data pipelines into production-grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high-quality testing frameworks covering performance, data quality and reliability Leveraging Agentic AI development tools to improve … consumed We're looking for someone with experience in Staff, Principal or Lead Data Engineering roles Python and expert-level PySpark AWS Glue Apache Spark Apache Airflow Cloud-native Data Engineering architectures Large-scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing, validation ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. You'll also play a key role in pre-sales activities, solution design, and helping to drive strategic Databricks initiatives … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

data lakes, operational data stores, event streaming platforms, and real-time data pipelines. Architect and document solutions using technologies such as Hive, Impala, HBase, Spark, NiFi, Flink, Kafka, and equivalent AWS services. Design Kafka-based streaming applications, including event ingestion, topic design, consumer patterns, error handling, replay, scalability …/CDP or equivalent big data platforms. Strong demonstrable experience with some of the distributed data technologies (if not all) such as: HBase Spark NiFi Kafka Flink Hive Impala Experience designing and delivering large-scale data ingestion, transformation, storage, and consumption patterns. Experience with Kafka consumer architecture, including consumer ...

domain-driven approaches CI/CD for data pipelines, version control discipline, infrastructure-as-code basics (Terraform) Streaming and big data exposure (Kafka, Kinesis, Spark, Flink) is a plus Data quality and observability tooling (Great Expectations, Monte Carlo, Soda, dbt tests) is a plus Engineering & Soft Skills: Strong software ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and Lakehouse solutions … forums, architecture review boards, design authority meetings, and client stakeholder workshops. Your Skills: Strong hands-on experience with Databricks on AWS. Strong experience with Apache Spark/PySpark. Excellent programming skills in Python. Strong SQL skills including complex queries, joins, CTEs, window functions, and query optimization. AWS Secrets ...

consultancy or client delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality ...

monitoring Strong background in CI/CD pipeline design, automation, and maintenance Hands‐on experience with data engineering tools and storage solutions (e.g., ADLS, Spark, Databricks, SQL/NoSQL databases) Experience with distributed computing and big data processing frameworks such as PySpark Knowledge of infrastructure‐as‐code tools such ...

Databricks, Azure Data Factory (ADF), and ADLS Gen2. Expertise in designing scalable, resilient, and enterprise-grade Azure data solutions. Advanced proficiency in PySpark, Python, Spark SQL, and SQL. Strong experience with SQL optimization, performance tuning, joins, window functions, and stored procedures. Extensive experience in developing and optimizing large-scale ...

version control. Data Engineering Stack Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL. Orchestration Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect. Data Quality and Lineage Familiarity with tools ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

Exceptional programming skills in Python, with a command of data science libraries (e.g., Pandas, NumPy, Scikit‐learn). Experience with big data technologies (e.g., Spark, Trino, Hadoop) and ideally cloud platforms (e.g., AWS, GCP, Azure) for scalable data processing, storage, and model deployment. Ideally having understanding and practical experience ...

Exceptional programming skills in Python, with a command of data science libraries (e.g., Pandas, NumPy, Scikit‐learn). Experience with big data technologies (e.g., Spark, Trino, Hadoop) and ideally cloud platforms (e.g., AWS, GCP, Azure) for scalable data processing, storage, and model deployment. Ideally having understanding and practical experience ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

data models and ETL processes, to support the construction of scalable and efficient data systems. Familiarity with big data tools like DuckDB, Polars or Apache Spark for processing out-of-memory datasets" . Proven experience collaborating with first-team stakeholders within a professional sports environment, demonstrating strong interpersonal ...

business value in collaboration with an Account Executive and a Senior Solutions Architect. Gain excitement from clients about Databricks through hands‐on evaluation and Spark programming, integrating with the wider cloud ecosystem and 3rd party applications. Contribute to building the Databricks technical community through engagement at workshops, seminars ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

sophisticated proofs‐of‐concept and public cloud platform(s). Coding expertise in a core programming language (Python, Java, Scala). Foundational understanding of Apache Spark architecture is preferable; hands‐on skills will be advantageous. Key Notes for the role: Location for the role needs to be within ...

concept and public cloud platform(s). Coding expertise in a core programming language (i.e., Python, Java, Scala). A foundational understanding of Apache Spark architecture is preferable; hands‐on skills will be advantageous for the role. We will be testing your ability to utilise AI‐assisted tools ...

Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial ...

Financial Services, multi-market experience Data Integration & System Engineers Build scalable data pipelines supporting decisioning systems. Key Skills: GCP (BigQuery) & Hadoop/Cloudera (mandatory) Spark, Scala, Python, SQL Data pipelines (batch & near real-time) Data modelling, governance & CI/CD Pega CDH integration experience QA/Test Specialists Ensure ...

will work in a modern, cloud-first environment built on Google Cloud Platform, with workloads deployed on GKE and leveraging technologies such as Spark, Python, SQL, Trino, Databricks SQL, Apache Arrow, Apache Kafka and Iceberg. The role involves close collaboration with other engineering and data teams … technical guidance, coaching and support to help them grow and deliver high-quality solutions. Design, build and maintain scalable data pipelines and workflows using Spark, Python and SQL to support analytics, reporting and data products. Oversee the implementation and optimisation of data processing solutions on cloud infrastructure, particularly within ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive ...