176 to 200 of 238 Apache Jobs in London

working with cloud‐native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such … Glue, EMR Serverless, Lambda, and S3. Contribute to the evolution of lakehouse architecture leveraging Apache Iceberg. Apply Software Engineering Best Practices Use version control, CI/CD pipelines, automated testing, and modular code principles. Participate in pair programming, code reviews, and architectural design sessions. Data Quality & Observability Build monitoring ...

Role As a Senior Data Engineer, you'll take ownership of the design, development and optimisation of production-grade data solutions using Databricks, Apache Spark and Delta Lake. Your responsibilities will include: Designing, building and optimising scalable ETL/ELT data pipelines using Databricks, PySpark and Spark SQL Implementing … engineering skills with the ability to take technical ownership of complex data solutions. You'll ideally have: Strong commercial experience with Databricks Excellent PySpark, Apache Spark and Spark SQL skills Experience building production-grade ETL/ELT pipelines Strong knowledge of Delta Lake and Lakehouse architecture Experience implementing Medallion ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team where … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. ...

Lead the development of transformation workflows using dbt, with a strong emphasis on testing, documentation, and data quality. Architect and manage pipeline orchestration via Apache Airflow, ensuring reliability and observability at every stage. Monitor and optimise pipeline performance for low-latency, high-throughput use cases. Collaborate closely with Data … customer and marketing data at scale. Expert-level SQL and proven experience building robust transformation pipelines with dbt. Solid experience orchestrating workflows with Apache Airflow in production environments. Comfortable working across cloud platforms (AWS, GCP, or Azure) for data ingestion, storage, and processing. Proficiency in Python or a similar ...

mindset focused on automation, reliability, and maintainable solutions. Preferred Qualifications Experience with Argo CD, Helm, or other GitOps tooling. Familiarity with Airflow, Snowflake, or Apache Iceberg. Experience with Apache Kafka or Amazon MSK. Experience with hybrid cloud, enterprise networking, or multi-account AWS environments. Experience with Ansible ...

working as a Data Architect within the Defence space. Active DV Clearance and sole UK nationality Strong AWS data lake, platform and security expertise Apache NiFi architecture, governance and pipeline delivery Secure data integration experience across multiple security domains API, file-based, legacy and bespoke integration design and delivery ...

working as a Data Architect within the Defence space. Active DV Clearance and sole UK nationality Strong AWS data lake, platform and security expertise Apache NiFi architecture, governance and pipeline delivery Secure data integration experience across multiple security domains API, file-based, legacy and bespoke integration design and delivery ...

architectures (such as load balancing, LAMP, CDN’s), even if you haven’t worked on one Configuration and maintenance of common applications such as Apache, memcached, Squid, MySQL, NFS, DHCP, NTP, SSH, DNS, and SNMP A healthy respect for our mottos “Move Fast” and “Be Bold” Good communications skills ...

WIndows (Note: You will be required to install, configure, and troubleshoot Linux systems.) Puppet/Ansible Satellite/Katello Network Monitoring (Nagios/Other) Apache/OpenSSL Bash/Python OpenLDAP Virtualisation Citrix VMware: ESX, Virtual Center Network technologies TCP/IP stack, OSI Model VPNs Cisco ...

Data, Streaming & Analytics Oversee large-scale telemetry ingestion via highly available MQTT brokers handling 6+ billion messages per day . Guide streaming pipelines using Apache Pulsar and data storage across DynamoDB and Google BigQuery . Enable high-fidelity time-series analytics, machine learning, and forecasting at scale. Ensure robust ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. Benefits We strive to provide comprehensive benefits and perks that meet the needs of all our employees. Our Commitment ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits ...

semantic data fabric, converting multi-stream clinical execution datasets into an object-centric model. Design and execute a modern Data Lake strategy centred on Apache Iceberg as the core storage format, ensuring high-performance querying and seamless interoperability with Snowflake and heterogeneous compute engines. AI-Accelerated Schema & Pipeline Engineering … resilient, self-healing pipeline architectures. Key Business Value & Strategic Impact Your leadership will directly advance life sciences technology. By pairing modern lakehouse architecture (Snowflake, Apache Iceberg, Kafka) with AI-augmented workflows, you will empower our platform to process critical safety signals in hours rather than weeks. This enables adaptive ...

Data Engineer within GCP environments Hands-on experience with BigQuery, Dataflow and Spanner Strong programming capability in Python, Java or Scala Experience with Apache Beam/Dataflow and distributed processing frameworks Experience designing and managing ETL/ELT pipelines Solid understanding of data modelling and database design Experience with … workflow orchestration tools such as Apache Airflow Strong understanding of data governance, security and scalability Desirable Skills for the GCP Data Engineer: Experience with Pub/Sub, Cloud Composer or Cloud Data Fusion Exposure to CI/CD, DevOps and Infrastructure as Code Experience with real-time streaming architectures ...

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python scripting. … communication skills with an ability to lead a team and keep them motivated. • Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow #J-18808-Ljbffr ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders ...

find to improve model and system performance. Build reliable ML infrastructure Develop and maintain data and machine learning workflows using technologies such as Apache Airflow. Help improve CI/CD, automated testing, monitoring and reproducibility across our ML stack. Engineer on AWS Build and improve cloud infrastructure using services … expect you to know everything: PyTorch, TensorFlow or JAX Hugging Face LLM APIs and open-source models RAG and vector databases Apache Airflow AWS S3 ECS/EKS Redshift Docker Kubernetes GitHub Actions Pulumi or Terraform SQL Model monitoring and observability Distributed processing Financial markets, economics or commodities Financial ...

DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible. Experience building and supporting reactive, event-sourced systems. Hands-on experience with Apache Spark and Apache Flink. Strong understanding of Apache HTTP, Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data formats. ...

engineer with strong experience in streaming technologies, event-driven architecture, and distributed data systems. The Role Design and build real-time data pipelines using Apache Flink , Apache Kafka , and AWS Kinesis Develop scalable event-driven and streaming solutions Integrate data from source systems including Oracle, MySQL, and Sybase … enrichment, and quality capabilities Deliver low-latency data into enterprise data platforms Create and manage infrastructure using Terraform Skills & Experience Strong commercial experience with Apache Flink Strong experience with Apache Kafka Experience with AWS Kinesis Advanced SQL skills and strong relational database knowledge Experience working with Oracle, MySQL ...

work independently or as part of a team and operate to tight deadlinesSense of humour pleaseAdvantageous But Not a Must: Great knowledge of Apache, specifically Mod RewriteGood working knowledge of Linux, including command lineServer administration experienceEcommerce experienceSmartyCompetitive SalaryCareer Growth and Financial StabilityExcellent Benefits: Opportunity to work ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits ...

Options trading desk to translate business needs into scalable software solutions. You will contribute hands-on development using Core Java, Kafka, CouchbaseDB, and Apache Storm, while mentoring engineers and shaping architectural direction for the bank's risk platform. #J-18808-Ljbffr ...