1 to 25 of 31 Apache Spark Jobs in the South East

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend ...

design, build and operate reliable, scalable data pipelines and lakehouse solutions that enable analytics, AI and GenAI. You will work hands-on with Databricks, Apache Spark and the Azure data ecosystem to ingest, transform and serve trusted data products, taking ownership from development through to production support … Databricks focus, you will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage Databricks, Apache Spark and Azure services to develop and optimise batch and streaming pipelines, implement Delta Lake lakehouse patterns, and deliver well-governed data sets that ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. Design … build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Design and oversee the delivery of secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. What … which Azure data focused components to use for which purpose. Microsoft certified architect Databricks Design and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Ability to Design, Build and deploy ...

data warehouses and data lakes. (Preferred) Experienced in data ingestion, processing, transformation, and delivery using batch, streaming, and real-time methods. (Preferred) Experience supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting, and AWS CLI. (Preferred) Great background in building and managing AWS platforms for analytics and data lake workloads, including application and framework management. (Preferred ...

domain-driven approaches CI/CD for data pipelines, version control discipline, infrastructure-as-code basics (Terraform) Streaming and big data exposure (Kafka, Kinesis, Spark, Flink) is a plus Data quality and observability tooling (Great Expectations, Monte Carlo, Soda, dbt tests) is a plus Engineering & Soft Skills: Strong software ...

learning or data science. Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials. Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow ...

consultancy or client delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality ...

production environments Familiarity with Azure Machine Learning and AI services Exposure to data visualisation tools (e.g. Power BI) Experience with big data frameworks (Spark, Kafka) Knowledge of data governance, lineage, and metadata tooling SHIFT & WORKING PATTERN Standard business hours, with participation in an on-call rota as required Occasional ...

native solutions that support analytics, automation, and digital transformation. Data Engineering, Full Stack & Platform Development Design and implement scalable data pipelines using tools like Apache Spark, Airflow, or dbt.Build and maintain data lakes, warehouses, and real-time streaming solutions. Develop APIs and microservices to expose data securely ...

optimising ETL/ELT pipelines. Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures. Experience with Azure Data Factory, Azure Synapse, and Apache Spark. Knowledge of data modelling, semantic models, and enterprise reporting solutions. Experience implementing data governance, security, and quality frameworks. Strong understanding of modern data engineering ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

What we’re looking for: Strong Python development and software engineering experience Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments ...

workflows. Key Requirements Passion for building scalable, resilient cloud data platforms. Strong experience with Databricks or Snowflake on AWS. Proven Python skills, including Spark and Airflow expertise. Advanced SQL skills and end-to-end data modelling experience. Experience building batch and real-time data integrations. Hands-on CICD skills ...

data pipeline orchestration tools Containerisation (Docker, EKS, AKS, Kubernetes) Proficiency in CI/CD principles and tools Familiarity with open-source data tools (e.g., Spark, Kafka, PostgreSQL) Infrastructure as Code (Terraform, Ansible) Air-gapped development Automation Client/stakeholder management Public sector domain understanding Technical architecture Experience: You will ...

data modernisation approach es Technical Expert iseStrong hands-on/architectural expertise in:Azure data ecosystem (ADF, Synapse, Fabric, AD LS)Databricks (Delta Lake, Spark, ML workflo ws)Power BI (enterprise analytics & semantic lay er)Strong grounding in:Data modelling (dimensional, domain-driv en)Data governance, lineage, catalogu ingIntegration ...

Analytical B2B solution providers. Proficiency in SQL, Python, or other data-focused programming languages and data modeling principals. Experience with big data technologies (e.g. Spark) and distributed data processing.Good experience managing cloud-based data platforms, ideally Azure. Experience developing using agile software development methodologies principles such as DevOps ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

monitoring Experience with cloud platforms (e.g. AWS, Azure, or GCP) and containerisation (Docker, Kubernetes) Experience building data pipelines and working with large datasets (SQL, Spark or similar) Strong understanding of machine learning algorithms and statistical modelling techniques Experience collaborating across data science and engineering teams Machine Learning Engineer, Python ...

unstructured datasets. Strong stakeholder management and communication skills. Desirable Insurance or financial services experience, ideally within pricing, underwriting, fraud, or claims. Experience with Spark, Hadoop, NoSQL, or modern big data platforms. Agile delivery experience and collaborative software development practices. Experience leading analytics teams and driving organisational change. Robert Walters ...

Experience with: Relational databases (SQL Server, PostgreSQL, Oracle, MySQL) Data visualisation tools (Power BI, Tableau, Spotfire) Cloud platforms, particularly Microsoft Azure Big data technologies (Spark, Elasticsearch, Hadoop) Statistical modelling and advanced analytics Machine learning and pattern recognition techniques Web technologies such as HTML and JavaScript ...

Limited (DDIL) communications environments, supporting local processing at the edge Building streaming ingestion, event pipelines, and batch/stream processing workloads using technologies like Apache Flink and/or Apache Spark Implementing metadata, lineage, and catalogue capabilities along with policy-based security and governance Job Requirements: Extensive ...