1 to 25 of 158 Apache Jobs in England

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands-on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud-native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal experience ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding experience ...

value. Excellent analytical, problem-solving, and communication skills. Proven experience leading data modernisation initiatives across multiple sectors. Technical Expertise Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog. Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation). Hands-on expertise ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

while collaborating with the Data & Analytics Lead, technical colleagues and business stakeholders. Experience in the following areas would also be useful: Materialised Lake Views Apache Spark or Python. Power BI semantic modelling. APIs and cloud business system integration. Data governance, lineage and quality frameworks. Professional services experience. What ...

Mentor engineers and promote best practices in Databricks and data engineering.What it takesMust-Have Skills & Experience· Expert-level experience with Databricks, PySpark, SQL, and Apache Spark· Strong experience designing and delivering cloud-based data platforms and lakehouse architectures· Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows ...

highly desirable but strong Python/PySpark engineers who can learn Foundry quickly are encouraged to apply. Extensive experience with Python and PySpark/Apache Spark. Proven track record designing and delivering scalable data pipelines and data products. Strong SQL and data modelling skills. Experience with modern data architectures ...

with DataStrong SQL expertise, particularly in complex analytical and warehouse queriesProficiency in Python and/or JavaExperience with Data Warehouses, such as Snowflake or Apache IcebergPractical knowledge deploying and operating containerised workloads using Docker and KubernetesKnowledge of streaming tooling (Kafka, Kafka Connect, Flink,...)Experience with ETL/ ...

understand where traditional patterns remain appropriate – and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness of DevOps ...

experience in Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

Terraform and Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery.Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines.Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

architecture standards and best practices Working closely with business and technical stakeholders Leading data transformation and cloud migration initiatives Desirable Databricks, Snowflake, dbt Kafka, Apache Iceberg, Delta Lake Data Mesh/Data Fabric AI & ML-enabled data platforms ...

Required Strong hands-on experience with Python, PySpark , and SQL/Spark SQL for building production-grade ETL/ELT pipelines Solid knowledge of Apache Spark , including DataFrames, joins, aggregations, partitioning, caching, and performance optimisation Experience working within Azure Databricks , including clusters, job clusters, notebooks, repositories, jobs, and Databricks ...

with monitoring/logging tools (ELK stack, Prometheus, Splunk) Familiarity with service mesh and modern networking (Istio, Envoy) Exposure to data engineering and integration (Apache ecosystem, SQL, Python) Recognised technical certifications (AWS/Azure Architect, Kubernetes CKA/CKAD, Terraform) Experience in influencing technical standards, best practices, or contributing ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data Meshes ...

cataloguing tools to track data and metadata through its journey such as: Google Cloud DataCatalog, AWS Glue DataCatalog, Azure DataCatalog, Informatica Enterprise DataCatalog, Alation, Apache Atlas etc. Ensuring compliance with business standards. Lead the use of Agile practices in the team toleveragedata engineering tools like version control, automation, monitoring ...

audiences and senior customers.Using tools such as industry standard ETL tools, network databases and scheduling and orchestration tools g. SQL, Linux, bash scripts, PostgreSQL, Apache Derby, AWS Glue & Step Functions.Familiarity with data visualisation tools (e.g. Power BI), and emerging tools such as LLMs.Data engineering in the context ...