1 to 25 of 103 Apache Jobs in England

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling (e.g. ...

Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability \n Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations \n Model and transform data for analytics using SQL and dbt to support business intelligence ...

Java or Scala, and SQL. \n Hands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). \n Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. \n Experience working with cloud‐native ...

Java or Scala, and SQL. \n Hands‐on experience building and operating large‐scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). \n Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. \n Experience working with cloud‐native ...

with distributed event stores and stream-processing platforms such as Kafka of Kinesis. ● Experience working with batch processing frameworks, such as DBT, Argo Workflows, Apache Airflow, etc. ● Familiarity with Docker, Kubernetes, Amazon EKS ● Familiarity with Continuous Integration with GitHub Actions ● Familiarity with Test-Driven Development and XP Our commitment ...

Django, Docker \n Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo \n Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. \n Data acquisition and development of data sets and improving data quality \n Preparing data for predictive and prescriptive modelling ...

Engineer, Principal Data Engineer or Technical Lead within a large-scale, data-driven organisation Strong hands-on software engineering expertise with Python, PySpark and Apache Airflow Deep experience designing and building distributed data processing systems and large-scale data platforms on AWS Strong knowledge of modern AWS data technologies ...

years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

Django, Docker \n Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo \n Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. \n Data acquisition and development of data sets and improving data quality \n Preparing data for predictive and prescriptive modelling ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

based solutions. Experience working with Cosmos DB SDKs in technologies such as .NET, Java, Python, or Node.js. Strong understanding of distributed computing concepts and Apache Spark architecture. Hands-on experience developing data pipelines using PySpark. Experience with Pytest and automated testing frameworks. Strong understanding of partitioning, indexing strategies ...

/CD pipelines and deployment automation. • Familiarity with monitoring, logging, and observability tools. • Awareness of application platforms and middleware such as Java, Tomcat, Apache, and NodeJS. • Experience using Git and collaborative development workflows. • Basic scripting skills using Shell or Python. • Understanding of containerisation concepts using Docker. • Strong problem-solving ...

SDLC with an integrated development team using CI/CD tooling including Git \n\n Desirable Skills \n \n Hands-on experience with Apache NiFi for data pipeline development and orchestration \n AWS experience including familiarity with EMR, S3 and SQS \n Awareness of domain-driven design approaches ...

principles Experience working within an SDLC with an integrated development team using CI/CD tooling including Git Desirable Skills Hands-on experience with Apache NiFi for data pipeline development and orchestration AWS experience including familiarity with EMR, S3 and SQS Awareness of domain-driven design approaches and architecture ...

skills and attention to detail ✅ A proactive approach with the ability to own and deliver tasks Desirable: Experience with Azure Data Factory, Databricks or Apache Spark Familiarity with tools such as Fivetran Understanding of modern data architectures (e.g. medallion architecture) Exposure to CI/CD, Azure DevOps or Jira ...

engineers and support code reviews, design reviews, and engineering best practices. Your Skills: Strong hands-on experience with Databricks on AWS. Strong experience with Apache Spark/PySpark. Excellent programming skills in Python. Strong SQL skills including complex queries, joins, CTEs, window functions, and query optimization. Experience with Spark ...

Owning the cloud infrastructure underpinning data systems through a DevOps approach \n Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop \n Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software Engineer … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend services ...

developing others. Desirable to have skills or experience in: - SAS Viya for data analytics. - Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). - Denodo for data virtualisation. - Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). - ALM Tooling (Jira, Confluence). ...

Core Java Spring Boot frameworks Proven experience building distributed backend systems Hands-on experience with Kafka (or similar streaming platforms) Experience with Scala and Apache Spark for data processing Integration withbackend services such as: PostgreSQL Redis S3 (or equivalent object storage) Expertise in modern engineering practices, including: Continuous delivery ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

alerting. Minimum requirements: 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of Python, SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code ...