16 of 16 Apache Spark Jobs in the Midlands

should have specialist experience in one or more of the following technologies**Azure Databricks*** Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.* Experience of Databricks ML and Azure … develop predictive models and drive business insights.* Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices.* Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory.* Azure Data Engineer Associate and Databricks ...

Generative AI initiatives by developing scalable, secure and reliable data platforms. Key Responsibilities Design, build and maintain high-throughput data pipelines using Apache Spark and cloud-native technologies. Develop scalable ETL/ELT workflows for structured and unstructured datasets. Build and optimise data-processing solutions within AWS environments. … standards. Troubleshoot complex data-processing and distributed-system issues within production environments. Essential Experience 5+ years of commercial Data Engineering experience. Strong hands-on Apache Spark development experience. Strong programming experience with Java and/or Scala. Strong AWS experience, ideally including technologies such as S3, EMR, Glue ...

/CD platform (Github Actions and Admin), Password/Secret Management (hashicorp vault). Strong Data related programming skills SQL/Python/Spark/Scala. Database technologies in relation to Data Warehousing/Data Lake/Lake housing patterns and relevant experience when handling structured and non-structured ...

with Microsoft Azure (a plus): Azure SQL Database, Analysis Services, Databricks, Data Lake, Logic Apps and Data Factory. Understanding of big data technologies (Hadoop, Spark). Able to create rich visualisations that appropriately represent the underlying datasets. Review and align the reporting attributes against the enterprise and conceptual data ...

alignment with business priorities and best practice Implementing robust ETL/ELT pipelines using Azure Data Factory, Synapse Pipelines and Azure Databricks (PySpark/Spark SQL) to support batch and streaming data processing Designing and delivering real-time and batch data ingestion solutions using Azure Event Hubs, Kafka, Databricks ...

drift monitoring; versioning and release management. Cloud architecture (Azure/AWS/GCP), Kubernetes/Docker, serverless, IAM and network security. Data engineering depth (Spark/Databricks; ETL/ELT); cloud‐native data + AI architectures. Enterprise integration and SRE principles (SLIs/SLOs, runbooks, rollback). Consulting leadership ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Unity Catalog migration across key data domains. Define and implement modern orchestration patterns, replacing legacy workflows. Improve performance and reduce costs across Databricks clusters, Spark jobs, storage, and scheduling. CI/CD Testing Monitoring and observability Production readiness Review architecture decisions and critical pull requests. Mentor engineers and provide ...

with cloud‐based data warehouses (e.g., Snowflake, GBQ). Familiarity with cloud‐based data pipeline services (e.g. AWS Glue) or distributed processing frameworks like Apache Spark. Equal Opportunity Employment Goldman Sachs is an equal opportunity employer and does not discriminate on the basis of race, colour, religion, sex, national ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

engines. Optimize processing engines for scalability and latency Expand capabilities to support new databases and cloud platforms Tech stack: Java, Python, Angular, Kubernetes, Spark, AWS, Azure, GCP AI Governance: Build features for AI compliance, connecting disparate systems and simplifying governance processes. Develop customizable platforms for managing AI compliance ...

Senior Data Engineer 6 Month contract with extension Full-time Inside IR35 Hybrid 1 day every two weeks in the office Nottingham - NG1 4BX Join our client in embarking on an ambitious data transformation journey ...

Kafka Connect, Schema Registry, Cluster Linking and ksqlDB.Some other highly valued skills may include: Experience with streaming technologies such as Kafka Streams, Flink or Spark Streaming. Experience working with cloud-native technologies and Kubernetes-based platforms in enterprise environments. Effective collaboration, communication and problem-solving skills, with the ability ...

Location: Northampton, UK --------------------------------------- JOB DETAILS Role Title: Sr Data Engineer • Minimum 6-8 years of experience working as Sr Data Engineer • Bigdata Technologies – PySpark, Spark • Coding Languages – Python, SQL • AWS services – Glue, Cloud formation, Lake Formation, IAM, Lambda, Redshift • Exposure to cloud technologies – Snowflake, Redshift, Apache Flink … Apache Kafka. • Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives. • Understand project requirements and translate them into technical solutions which meets the project quality standards • Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/ ...