1 to 25 of 96 Apache Spark Jobs in the UK

Data Engineer Salary up to £60,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via chelsea.hackett@searchability.com ABOUT THE CLIENT Due to continued growth, we’re seeking a skilled Data … optimise scalable data pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

capabilities across the organization. Key Responsibilities Design and implement data architectures on the Databricks Lakehouse Platform Build and optimize ETL/ELT pipelines using Apache Spark and Databricks Implement Delta Lake and medallion architecture (Bronze, Silver, Gold) Ensure data governance, security, and performance optimization Integrate Databricks with cloud … stakeholders to deliver scalable data solutions Required Skills & Qualifications Databricks Certification (e.g., Databricks Certified Data Engineer Associate/Professional) Strong experience with Databricks, Apache Spark, and Delta Lake Proficiency in Python, SQL, and data pipeline development Experience with cloud platforms (Azure, AWS, or GCP) Solid understanding of modern ...

code and sharing knowledge across the team Supporting production systems when needed The platform uses a mix of modern data tooling including AWS and Apache Spark, so there's plenty of opportunity to work with big data technologies. What they're hoping you'll bring … complex data problems An interest in improving pipelines, tooling and processes A collaborative approach and willingness to share ideas Exposure to tools like Apache Spark, modern data platforms, or AI tools for development productivity would be a bonus but isn't essential. The setup The team works ...

code and sharing knowledge across the team Supporting production systems when needed The platform uses a mix of modern data tooling including AWS and Apache Spark, so there's plenty of opportunity to work with big data technologies. What they're hoping you'll bring … complex data problems An interest in improving pipelines, tooling and processes A collaborative approach and willingness to share ideas Exposure to tools like Apache Spark, modern data platforms, or AI tools for development productivity would be a bonus but isn't essential. The setup The team works ...

scalability Build and integrate RESTful microservices within a complex distributed architecture Work with large-scale data sets, leveraging SQL and big data tools (eg, Spark, Kafka) Influence system design and contribute to the technical roadmap Collaborate closely with a highly skilled engineering team to deliver innovative solutions What … experience building and consuming RESTful services Solid SQL skills and familiarity with large-scale data processing Knowledge of big data technologies such as Spark and Kafka Experience in finance is preferred, but not essential Strong academic background, ideally in Computer Science or a related discipline Skills Representational State Transfer ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark Very useful - Python, SQL, JSON, LakeFS. ...

with cross-functional teams to optimise data workflows and performance * Utilising programming languages such as Java, Python, and SQL, alongside tools like LakeFS and Apache Spark, to process and manage data efficiently What you'll bring: * Proven expertise as an Elastic Search SME, with a deep understanding … with data systems, from collection to analysis * Strong programming skills in Java, Python, and SQL * Familiarity with data processing tools such as LakeFS and Apache Spark (desirable) * Ability to work effectively in a collaborative environment, with excellent problem-solving skills * Must have a minimum of 5 years ...

direction within your team Required Skills and Experience Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training simulation ...

Time series, NLP, recommendation systems (as applicable) Programming Python (NumPy, Pandas, Scikit learn) Optional: R, SQL Data Platforms Relational & NoSQL databases Big data frameworks (Spark, Hive, Databricks) MLOps & Deployment Model lifecycle management CI/CD pipelines Containerization (Docker, Kubernetes desirable) Model packaging and REST APIs Cloud & Tools (Any combination ...

data platform and engineering capability as the business continues to scale its trading operations. They are looking for a Data Engineer with strong Apache Airflow experience to help design and build the pipelines that power trading, analytics and commercial decision-making. This is a great opportunity to join … platform and the opportunity to work with modern open-source technologies . What You’ll Be Doing Designing and building data pipelines orchestrated with Apache Airflow Developing ETL/ELT workflows in Python and SQL to process trading and operational data Building reliable, scalable data infrastructure to support analytics ...

downstream data services across the organisation. Key Responsibilities Design, build, and maintain data pipelines using Databricks Develop ETL/ELT processes with PySpark and Spark SQL Transform and model structured and semi-structured datasets Improve performance, reliability, and cost efficiency of data workloads Ensure compliance with data governance, security … practice routes Required Skills & Experience Active SC Clearance - mandatory and non-negotiable Strong hands-on experience with Databricks in production environments Expert knowledge of Apache Spark , including PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure ...

logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across Cloud (Azure, AWS, GCP) and Big Data (Spark, Kafka) stacks. Collaboration: Work closely with business teams to define data flows and ensure security protocols (encryption/authentication) are met. Technical Toolkit Languages ...

Production Hands-On Experience with Deep Learning Frameworks (TensorFlow, PyTorch, XGBoost) Deep Knowledge of Data Warehousing, ETL/ELT Architectures & Heavy Analytic Environments (Apache Beam, Hadoop, Spark, Hive) Responsibilities: Leading Customer-Facing AI Deployments across Europe, from the Initial Consulting Phase right through to the Live Implementation … Production ML Systems Architecting & Optimising Scalable Data Pipelines, ETL/ELT Solutions & Distributed ML Environments (using Hadoop, Spark & Apache Beam). Deploying Recommendation Engines & Deep Learning Models (TensorFlow, PyTorch) to solve specific Client Problems. Running Technical Design Sessions, Tailoring ML Ops Architectures using Vertex AI & Gen AI. Writing ...

within scalable cloud -based architectures. Robust understanding of data governance, data quality, and metadata management. Desirable skills Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience in using Git, Azure DevOps, or GitHub Actions for version control, CI/ ...

within scalable cloud -based architectures. Robust understanding of data governance, data quality, and metadata management. Desirable skills Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience in using Git, Azure DevOps, or GitHub Actions for version control, CI/ ...

Ecosystem (Preferred/Good to Have) Azure data services experience in one or more: Azure Synapse Analytics/Dedicated SQL Pools Azure Databricks/Spark ADLS Gen2, Azure SQL DB, Managed Instance Event Hub/Kafka, Stream Analytics (if real-time involved) Monitoring & observability: Azure Monitor, Log Analytics, Application ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

deliver high-quality, consumable datasets while contributing to best practices across the wider platform. You'll be working heavily with Python , SQL and Spark , using tools such as Databricks , Airflow , dbt and Kafka within AWS . Experience with modern data stacks is important, along with a solid understanding ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

workflow management tools Strong expertise in SQL, data modelling, and database optimisation (relational, vector and NoSQL) Proficiency in distributed data processing frameworks (e.g., Spark, Flink, Hadoop) Deep knowledge of cloud data platforms (AWS, Azure, or GCP) and associated services, with proven data governance and security practices Strong programming skills ...

client schemas (local, global, corporate). The new platform targets Aurora PostgreSQL, with several modules already migrated. Pipelines must bridge both. • Databricks/Apache Spark — a lakehouse POC is being evaluated with Unity Catalog and Delta Sharing via AWS PrivateLink. Experience with Databricks or equivalent analytics platforms … important for assessing and scaling the right approach • Apache Kafka — event streaming experience is essential. The platform is evaluating Kafka as the strategic backbone for ordered event logs, consumer group replay, and guaranteed partition ordering as the microservices fleet scales • ETL/ELT pipeline design — building and maintaining pipelines ...

quality and reliable deliverables. Optionally contribute to on-call platform support where required. What We’re Looking For Technical Expertise Strong experience with Databricks, Apache Spark, and cloud-based data architectures (preferably Azure). Excellent programming capability in Python and SQL. Experience orchestrating pipelines with Azure Data Factory ...

making. Collaborating with engineering teams and stakeholders to solve complex technical challenges. Technology exposure Typical environments include: Python, SQL and modern data engineering tooling Spark, Databricks and distributed data platforms AWS, Azure and Google Cloud Modern data lake and data warehouse architectures Who we’re keen to speak with ...

applications. Knowledge of MLOps, DevOps, CI/CD, and automated model pipelines within Foundry and cloud-based environments. Familiarity with big data ecosystems (Foundry, Spark, Hadoop) and commercial data source integration. ...