1 to 25 of 82 Apache Jobs in England

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling (e.g. ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop, maintain ...

skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure as Code. Git/GitLab and modern software development practices. API integration ...

Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ETL/ELT development and orchestration tools (e.g., Apache Airflow, dbt, Prefect). Knowledge of data modelling, data warehousing, and lakehouse architectures. Familiarity with DevOps practices, CI/CD pipelines, and infrastructure ...

Collaborate with senior developers and quantitative teams on new analytics capabilities Participate in debugging, testing, and production support activities Technology Environment Python Databricks/Apache Spark SQL Pandas/NumPy Git REST APIs Cloud technologies (Azure or AWS) Numerix (or equivalent pricing/risk platform) Jupyter/notebook-based ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration and continuous ...

Responsibilities Pipeline Development: Design, develop, and maintain robust end-to-end data pipelines and orchestration workflows using Python , Apache Airflow , and Cloud Composer on GCP. Data Modernization: Bridge the gap between legacy/on-premise database environments and modern cloud architectures, ensuring seamless data ingestion and migration. Optimization & Performance ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

analytics and reporting, and contributing to the adoption of AI-powered workflows and automation. Key Responsibilities & Skills: Build and maintain scalable data pipelines using Apache Airflow or similar orchestration tools. Develop integrations with internal and third-party APIs, ensuring reliable and efficient data flows. Design and optimise SQL-based ...

processin g.Cloud Pub/Su b: For real-time messaging and event ingestio n.Cloud Storag e: For scalable and durable object storag e.Cloud Composer (Apache Airflow ): For workflow orchestration and pipeline managemen t.Cloud Datapro c: For managed Hadoop and Spark clusters (good to have ).Cloud SQL/Cloud Spanner ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness of DevOps ...

platform role Strong hands-on experience with AWS (EC2, S3, RDS, Lambda, Redshift, Glue, Athena) Experience with real-time and batch data processing - Apache Flink, Kafka, Spark, or similar Proficiency in one or more of: Python, Java, Scala Experience with data modelling , ETL processes, and distributed systems Comfortable working ...

systems. • Ability to obtain UK Security Clearance (SC). We’d love it if you also have: • Experience with data visualisation tools such as Apache Superset, Jupytr or Power BI • Familiarity with visualisation libraries like D3.js, Chart.js or Plotly for building interactive dashboards. • Exposure to DevOps practices including ...

HTML5, CSS3, SASS Cloud, DevOps & Infrastructure Kubernetes (GKE/IKP/AWS EKS) Docker, Terraform CI/CD: GitHub Actions, Jenkins Messaging, Streaming & ETL Apache Kafka Apache Flink, Apache Beam Redis Databases PostgreSQL Oracle MongoDB Security & Compliance OAuth 2.0, Okta Enterprise security scanning: Checkmarx, Aqua Security Strong ...

Core Java Spring Boot frameworks Proven experience building distributed backend systems Hands-on experience with Kafka (or similar streaming platforms) Experience with Scala and Apache Spark for data processing Integration withbackend services such as: PostgreSQL Redis S3 (or equivalent object storage) Expertise in modern engineering practices, including: Continuous delivery ...

Strong hands-on experience wi th AWS (EC2, S3, RDS, Lambda, Redshift, Glue, Athena) Experience wi th real-time and batch data process ing — Apache Flink, Kafka, Spark, or similar Proficiency in one or more o f: Python, Java, Scala Experience wi th data modell ing, ETL processes ...

What We're Looking For: Proven experience as a Data Engineer within a cloud-based data environment. Strong hands-on experience with Databricks and Apache Spark. Experience building or improving medallion architecture, including Bronze, Silver and Gold data layers. Strong SQL skills for transformation, validation and data analysis. Python ...

strong focus on scalability, resilience, and performance. Develop and optimise scalable data pipelines , supporting both batch and streaming workloads, using technologies such as Apache Spark . Work extensively with data technologies , leveraging Python and SQL to deliver high-quality analytical and data-driven solutions. Lead the design and delivery ...

What We're Looking For: Experience working as a Data Engineer within a modern cloud data environment. Strong hands-on experience with Databricks and Apache Spark. Experience building structured data layers, ideally within a Bronze, Silver and Gold architecture. Strong SQL skills for transformation, validation and analysis. Python ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

promote DevOps. Support internal and external stakeholders, including customers and regulatory authorities. Tools, Systems & Continuous Improvement Manage and optimise the use of tools (e.g. Apache Spark, AWS Glue, Databricks or equivalent). Support system enhancements of data processing and transformation, migrations, and process improvements. Contribute to the development ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will play ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will play ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will play ...

Experience with data governance frameworks, metadata management, and regulatory compliance It would be a bonus if you have: Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, or data observability tooling Experience leading architecture workshops with both technical and non-technical stakeholders A background providing technical mentorship ...