1 to 25 of 152 Apache Spark Jobs in the UK

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend ...

design, build and operate reliable, scalable data pipelines and lakehouse solutions that enable analytics, AI and GenAI. You will work hands-on with Databricks, Apache Spark and the Azure data ecosystem to ingest, transform and serve trusted data products, taking ownership from development through to production support … Databricks focus, you will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage Databricks, Apache Spark and Azure services to develop and optimise batch and streaming pipelines, implement Delta Lake lakehouse patterns, and deliver well-governed data sets that ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. Design … build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Design and oversee the delivery of secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure ...

create scalable how-to guides Integrate Databricks platform capabilities with client systems and existing data infrastructure Advise on Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and MLflow best practices Deliver training and enablement sessions to customer data teams Contribute to internal knowledge sharing and best-practice documentation … Must-Have Skills & Experience 10+ years working with Big Data technologies (Apache Spark, Kafka, Data Lakes) in a customer-facing, post-sales, or consulting capacity 4+ years as a Solutions Architect designing and delivering Big Data solutions for enterprise clients Deep expertise in Apache Spark ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. What … which Azure data focused components to use for which purpose. Microsoft certified architect Databricks Design and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Ability to Design, Build and deploy ...

through large-scale distributed processing, structured storage and Data LakeHouse design, to curated consumption by Data Scientists, Analysts, Engineers and AI/ML platforms. Spark and distributed data processing sit at the heart of this architecture; deep fluency in these domains is essentia l. What You’ll Do:Architectural … Leaders hipDefine and evolve the enterprise data architecture strategy, with a primary focus on Cloud-scale Distributed Data Processing using Apache Spark and PySpark at the core of the platf ormEstablish and maintain architectural artefacts, reference architectures, Architecture Decision Records (ADRs), technical standards, and architecture roadmaps ...

Highly competitive Status: Inside IR35 Working pattern: Full time We are seeking an experienced Technical Delivery Lead with strong expertise in Python, Databricks, and Apache Spark to lead the successful delivery of enterprise-scale data engineering initiatives.. This is a hands-on leadership role combining technical delivery, solution … Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern data architecture and Lakehouse principles. ...

technologies, share knowledge, and contribute to a culture of excellence and innovation. Key responsibilities include: • Design & Build: Develop and optimise data pipelines using Databricks, Apache Spark, and Python. • Develop & Deliver: Create scalable data solutions on AWS leveraging S3, Glue, Lambda, and related services. • Integrate & Automate: Implement ETL processes … Hands-on commercial expertise with Databricks. You should have: • Proven experience as a Data Engineer working with large, complex datasets. • Hands-on expertise with Apache Spark, and SQL. • Strong proficiency in Python (PySpark preferred). • Experience with AWS cloud services including S3, Glue, Lambda, IAM. • Familiarity with ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. Key Responsibilities Design and architect Databricks-based data platforms and solutions. Build scalable data lakehouse and data warehouse architectures. Design … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. You'll also play a key role in pre-sales activities, solution design, and helping to drive strategic Databricks initiatives … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

pipeline and working with stakeholders to understand requirements. What You’ll Do as Data Engineer: Work on complex data projects, using tools such as Apache Spark, Athena and AWS. Review and measure the performance of the data pipeline, identifying improvements in design, architecture and tooling. Work with stakeholders … What We’re Looking For: 4+ years' experience in a Data Engineer role, with proven experience on AWS. Strong skills in SQL, Python, and Apache Spark (Scala or Python). Knowledge of software development best practices and ability to problem-solve whilst working on large, complex data sets. ...

data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/ ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

data warehouses and data lakes. (Preferred) Experienced in data ingestion, processing, transformation, and delivery using batch, streaming, and real-time methods. (Preferred) Experience supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting, and AWS CLI. (Preferred) Great background in building and managing AWS platforms for analytics and data lake workloads, including application and framework management. (Preferred ...

domain-driven approaches CI/CD for data pipelines, version control discipline, infrastructure-as-code basics (Terraform) Streaming and big data exposure (Kafka, Kinesis, Spark, Flink) is a plus Data quality and observability tooling (Great Expectations, Monte Carlo, Soda, dbt tests) is a plus Engineering & Soft Skills: Strong software ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and Lakehouse solutions … forums, architecture review boards, design authority meetings, and client stakeholder workshops. Your Skills: Strong hands-on experience with Databricks on AWS. Strong experience with Apache Spark/PySpark. Excellent programming skills in Python. Strong SQL skills including complex queries, joins, CTEs, window functions, and query optimization. AWS Secrets ...

learning or data science. Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials. Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow ...

aiming for a career in data engineering, it can feel like youre staring at a never-ending list of tools and technologies SQL, Python, Spark, Kafka, Airflow, dbt, Snowflake, Redshift, Terraform, Kubernetes, and the list goes on. Scroll job boards and LinkedIn, and its easy to conclude that unless ...

consultancy or client delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality ...

production environments Familiarity with Azure Machine Learning and AI services Exposure to data visualisation tools (e.g. Power BI) Experience with big data frameworks (Spark, Kafka) Knowledge of data governance, lineage, and metadata tooling SHIFT & WORKING PATTERN Standard business hours, with participation in an on-call rota as required Occasional ...

automation & configuration management; • Ansible (plus Puppet, Saltstack), Terraform, CloudFormation; • NodeJS, REACT/MaterialUI (plus Angular), Python, JavaScript; • Big data processing and analysis, e.g. Apache Hadoop (CDH), Apache Spark; • RedHat Enterprise Linux, CentOS, Debian or Ubuntu. • Java 8, Spring framework (preferably Spring boot), AMQP - RabbitMQ, • Open source technologies ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...