1 to 25 of 85 Apache Jobs in England

similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced SQL and distributed ...

highly regulated enterprise environments. Strong hands-on expertise in Python Proven experience with Databricks, including notebooks, workflows, jobs and Delta Lake Strong knowledge of Apache Spark/PySpark Experience designing and delivering large-scale ETL/ELT pipelines Advanced SQL skills and experience with relational databases Experience working with ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

understanding of Quantexa components such as: Entity Resolution (ER), Scoring Framework, Contextual Network Generation, UI Configuration. Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch. Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

/CD pipelines and deployment automation. • Familiarity with monitoring, logging, and observability tools. • Awareness of application platforms and middleware such as Java, Tomcat, Apache, and NodeJS. • Experience using Git and collaborative development workflows. • Basic scripting skills using Shell or Python. • Understanding of containerisation concepts using Docker. • Strong problem-solving ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software Engineer … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend services ...

grade data pipelines. Familiarity with Git, CI/CD and Agile delivery practices. Excellent communication and stakeholder engagement skills. Nice to Have Experience with Apache Airflow for workflow orchestration. Experience with Databricks . #dataengineer #aws #snowflake #dbt #airflow #databricks #contractrole #outsideir35 Reasonable Adjustments: Respect and equality are core values ...

developing others. Desirable to have skills or experience in: - SAS Viya for data analytics. - Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). - Denodo for data virtualisation. - Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). - ALM Tooling (Jira, Confluence). ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...

alerting. Minimum requirements: 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of Python, SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code ...

reviews and continuous improvement Skills & Experience * Commercial experience in Data Engineering * Strong Python or Java development skills * Experience with AWS cloud services * Knowledge of Apache Spark, Hadoop, NiFi or similar big data technologies * Experience building ETL/ELT pipelines and processing large datasets * Understanding of software engineering best practices ...

production, providing subject matter expertise on the .Net stack and contributing to technical design discussions. You'll use a range of technology including Apache Flink with Java for large scale data processing and will be able to assess and recommend new and emerging technologies, using the best tool ...

lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines hands-on technical leadership with test strategy ownership . … reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis for production data issues and platform defects Risk Management & Reporting Identify, assess, and communicate quality risks across releases ...

with data scientists, analysts, and business stakeholders. Strong understanding of data platform reliability, performance, scalability, and cost optimisation. Experience with technologies such as ClickHouse, Apache Iceberg, BigQuery, dbt, Elasticsearch, Druid, GCP, or AWS is highly desirable. If you're an experienced Senior Data Engineer looking to lead complex data ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. EEO Statement At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs ...

Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial domains ...

data models and ETL processes, to support the construction of scalable and efficient data systems. Familiarity with big data tools like DuckDB, Polars or Apache Spark for processing out-of-memory datasets" . Proven experience collaborating with first-team stakeholders within a professional sports environment, demonstrating strong interpersonal ...

planning, and UAT support. Experience with cloud‐based data platforms, particularly Snowflake. Understanding of ETL/ELT frameworks and tools such as dbt and Apache Airflow. Knowledge of data governance, metadata management, and data lineage concepts. Experience supporting Business Intelligence and reporting solutions. Familiarity with modern analytics engineering practices. ...

audit trails are essential. Desirable Skills & Experience Databricks Certified Data Engineer Professional or equivalent Databricks certification. Experience with adjacent data tooling such as dbt, Apache Spark, or AWS Glue. Familiarity with data product management tooling and data marketplace patterns. Previous experience in a technology modernisation or platform migration programme ...

environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high … commercial experience developing data engineering solutions using Python . Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake. Strong experience with Apache Spark/PySpark . Proven experience building scalable ETL/ELT pipelines. Strong SQL and data modelling skills. Experience designing modern Lakehouse architectures. Experience ...