1 to 25 of 164 Apache Spark Jobs in the UK

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend ...

through large-scale distributed processing, structured storage and Data LakeHouse design, to curated consumption by Data Scientists, Analysts, Engineers and AI/ML platforms. Spark and distributed data processing sit at the heart of this architecture; deep fluency in these domains is essentia l. What You’ll Do:Architectural … Leaders hipDefine and evolve the enterprise data architecture strategy, with a primary focus on Cloud-scale Distributed Data Processing using Apache Spark and PySpark at the core of the platf ormEstablish and maintain architectural artefacts, reference architectures, Architecture Decision Records (ADRs), technical standards, and architecture roadmaps ...

Highly competitive Status: Inside IR35 Working pattern: Full time We are seeking an experienced Technical Delivery Lead with strong expertise in Python, Databricks, and Apache Spark to lead the successful delivery of enterprise-scale data engineering initiatives.. This is a hands-on leadership role combining technical delivery, solution … Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern data architecture and Lakehouse principles. ...

Data Virtualization) TIBCO Data Virtualization Red Hat JBoss Data Virtualization (Teiid) Microsoft Fabric Databricks Snowflake Collibra Microsoft Purview Informatica Azure Data Factory AWS Glue Apache Spark Apache Kafka Cloud Technologies Experience with one or more cloud platforms, including: Amazon Web Services (AWS) Microsoft Azure Google Cloud Platform ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. Key Responsibilities Design and architect Databricks-based data platforms and solutions. Build scalable data lakehouse and data warehouse architectures. Design … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. You'll also play a key role in pre-sales activities, solution design, and helping to drive strategic Databricks initiatives … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

technical direction within your team Required Experience: Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

NRequired Experience:\ N\n\nStrong analytic skills related to working with unstructured datasets\ N\n\nPython (PySpark, Pandas, PyArrow)\ N\n\nDistributed data processing (Apache Spark)\ N\n\nData ETL(Apache Airflow, AWS Step Functions, Apache NiFi)\ N\n\nCloud services (AWS, Azure ...

data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/ ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

domain-driven approaches CI/CD for data pipelines, version control discipline, infrastructure-as-code basics (Terraform) Streaming and big data exposure (Kafka, Kinesis, Spark, Flink) is a plus Data quality and observability tooling (Great Expectations, Monte Carlo, Soda, dbt tests) is a plus Engineering & Soft Skills: Strong software ...

where needed. Qualification Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

nCore Data Engineering\ N\n\nStrong programming proficiency in Java (preferred) or Python.\ N\n\nHands-onexperience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming).\ N\n\nSolid understanding of stream processing concepts (e.G., event time, state, backpressure).\ N\n\nUnderstanding ...

Django, Docker \n Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo \n Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. \n Data acquisition and development of data sets and improving data quality \n Preparing data for predictive and prescriptive modelling ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

Flask, Tornado or Django, Docker\ NExperience working with ETL pipelines is desirable e.G. Luigi, Airflow or Argo\ NExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.\ NData acquisition anddevelopment of data sets and improving data quality\ NPreparing data for predictive and prescriptive modelling\ NHands ...

Java or Scala, and SQL. \n Hands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). \n Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. \n Experience working with cloud ...

Python, Java or Scala, and SQL. Hands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop).Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools.Experience working with cloud‐native data platforms, with ...

Python, Java or Scala, and SQL. Hands on experience building and operating large scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. Experience working with cloud native data ...

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

Django, Docker \n Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo \n Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. \n Data acquisition and development of data sets and improving data quality \n Preparing data for predictive and prescriptive modelling ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

Flask, Tornado or Django, Docker \ NExperience working with ETL pipelines is desirable e.G. Luigi, Airflow or Argo \ NExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.\ NData acquisition anddevelopment of data sets and improving data quality \ NPreparing data for predictive and prescriptive modelling \ NReporting tools ...

Python, Java or Scala, and SQL.\ NHands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.G. Apache Spark, Hadoop).\ NExperience designingand maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools.\ NExperience working with cloud‐native data platforms ...

Python, Java or Scala, and SQL.\ NHands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.G. Apache Spark, Hadoop).\ NExperience designingand maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools.\ NExperience working with cloud‐native data platforms ...