1 to 25 of 141 Apache Spark Jobs in the UK

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend ...

through large-scale distributed processing, structured storage and Data LakeHouse design, to curated consumption by Data Scientists, Analysts, Engineers and AI/ML platforms. Spark and distributed data processing sit at the heart of this architecture; deep fluency in these domains is essentia l. What You’ll Do:Architectural … Leaders hipDefine and evolve the enterprise data architecture strategy, with a primary focus on Cloud-scale Distributed Data Processing using Apache Spark and PySpark at the core of the platf ormEstablish and maintain architectural artefacts, reference architectures, Architecture Decision Records (ADRs), technical standards, and architecture roadmaps ...

Highly competitive Status: Inside IR35 Working pattern: Full time We are seeking an experienced Technical Delivery Lead with strong expertise in Python, Databricks, and Apache Spark to lead the successful delivery of enterprise-scale data engineering initiatives.. This is a hands-on leadership role combining technical delivery, solution … Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern data architecture and Lakehouse principles. ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. Key Responsibilities Design and architect Databricks-based data platforms and solutions. Build scalable data lakehouse and data warehouse architectures. Design … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. You'll also play a key role in pre-sales activities, solution design, and helping to drive strategic Databricks initiatives … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

NRequired Experience:\ N\n\nStrong analytic skills related to working with unstructured datasets\ N\n\nPython (PySpark, Pandas, PyArrow)\ N\n\nDistributed data processing (Apache Spark)\ N\n\nData ETL(Apache Airflow, AWS Step Functions, Apache NiFi)\ N\n\nCloud services (AWS, Azure ...

data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/ ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

domain-driven approaches CI/CD for data pipelines, version control discipline, infrastructure-as-code basics (Terraform) Streaming and big data exposure (Kafka, Kinesis, Spark, Flink) is a plus Data quality and observability tooling (Great Expectations, Monte Carlo, Soda, dbt tests) is a plus Engineering & Soft Skills: Strong software ...

nCore Data Engineering\ N\n\nStrong programming proficiency in Java (preferred) or Python.\ N\n\nHands-onexperience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming).\ N\n\nSolid understanding of stream processing concepts (e.G., event time, state, backpressure).\ N\n\nUnderstanding ...

Django, Docker \n Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo \n Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. \n Data acquisition and development of data sets and improving data quality \n Preparing data for predictive and prescriptive modelling ...

Flask, Tornado or Django, Docker\ NExperience working with ETL pipelines is desirable e.G. Luigi, Airflow or Argo\ NExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.\ NData acquisition anddevelopment of data sets and improving data quality\ NPreparing data for predictive and prescriptive modelling\ NHands ...

Java or Scala, and SQL. \n Hands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). \n Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. \n Experience working with cloud ...

Java or Scala, and SQL. \n Hands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). \n Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. \n Experience working with cloud ...

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

Django, Docker \n Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo \n Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. \n Data acquisition and development of data sets and improving data quality \n Preparing data for predictive and prescriptive modelling ...

Flask, Tornado or Django, Docker \ NExperience working with ETL pipelines is desirable e.G. Luigi, Airflow or Argo \ NExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.\ NData acquisition anddevelopment of data sets and improving data quality \ NPreparing data for predictive and prescriptive modelling \ NReporting tools ...

Python, Java or Scala, and SQL.\ NHands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.G. Apache Spark, Hadoop).\ NExperience designingand maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools.\ NExperience working with cloud‐native data platforms ...

Python, Java or Scala, and SQL.\ NHands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.G. Apache Spark, Hadoop).\ NExperience designingand maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools.\ NExperience working with cloud‐native data platforms ...

/CD, and Infrastructure as Code. Desirable: * Experience within UK Defence, Intelligence, or National Security sectors. * Knowledge of big data tools such as Kafka, Spark, or Hadoop. * Support for machine learning or advanced analytics workloads. * Experience working with classified datasets and Elastic Stack (ELK). If you're ready ...

able to guide how one could deploy infrastructure into different environments. \n Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. \n Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes ...

able to guide how one could deploy infrastructure into different environments.\ NKnowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop.\ NGood understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes)\ NAbility to create ...

consultancy or client delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality ...

production environments Familiarity with Azure Machine Learning and AI services Exposure to data visualisation tools (e.g. Power BI) Experience with big data frameworks (Spark, Kafka) Knowledge of data governance, lineage, and metadata tooling SHIFT & WORKING PATTERN Standard business hours, with participation in an on-call rota as required Occasional ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...