201 to 225 of 254 Apache Spark Jobs in London

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python scripting. … building native python ETL pipelines for scraping/processing. • Good to have experience in processing big data. • Proficient in SQL programming, Postgres SQL. Spark knowledge is a good to have • Knowledge on DevOps CI/CD like Azure Pipelines, Jenkins, Git. • Experience working with AWS services(S3, lambda ...

remediation, or automation programs (preferably in financial services). Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet. Advanced SQL skills; experience with Oracle databases. Experience designing and supporting …/ELT pipelines and data quality frameworks (rules, thresholds, exception handling). Working knowledge of Autosys and Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift or equivalent). Familiarity with ...

building and managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real-time data ingestion and storage. Programming skills … similar language. Proficiency in database technologies (SQL, NoSQL, time-series databases) and data modelling. Strong understanding of data pipeline orchestration tools (e.g., Apache Airflow, Kubernetes). You thrive when working as part of a team Comfortable in a fast-paced environment Have excellent written and verbal English skills Last ...

pipelines and data quality frameworks (rules, thresholds, exception handling). Working knowledge of scheduling/orchestration tools such as Autosys and/or Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift or equivalent … reporting outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Exposure to data governance concepts: critical data elements (CDEs), lineage, data quality dimensions, audit frameworks. Experience ...

experience designing and implementing data solutions on at least one public cloud platform (AWS, Azure, or GCP) Working knowledge of distributed data systems: Apache SparkTM, Delta Lake, or equivalent (Hadoop, Kafka, Flink) Experience leading technical customer conversations - discovery, whiteboarding, architecture reviews Familiarity with one or more: data engineering … Engineering, or a quantitative discipline (or equivalent experience) Nice to Have: Databricks certification or experience with the Databricks Platform Experience with Unity Catalog, Lakeflow Spark Declarative Pipelines, or MLflow Background at a data/AI company, cloud provider, or technical consulting firm Interview Process: Recruiter Screen → Hiring Manager Screen ...

Help shape the technical roadmap for data engineering across Elliptic. Leverage and deploy AI and agentic systems to operate more effectively. Tech environment Scala | Spark | Databricks | AWS | Airflow | Kubernetes | Terraform | Functional Programming You will thrive here if you Enjoy end‐to‐end ownership, from writing the code to architecture … programming. Ability to design, build, and maintain distributed data systems in a cloud‐based environment. Hands‐on experience with big data frameworks such as Spark or Databricks. Knowledge of cloud infrastructure (AWS, GCP, or Azure). Judgement to balance scalability, performance, and maintainability. Experience with data modelling and workflow ...

enable analytics, machine learning and AI across the business. We are looking for an experience Lead Data Engineer with expectational hands on knowledge ofDatabricks, Spark, SQL and Python. Main Responsibilities Lead the development and improvement of the data platform and engineering standards. Mentor and support a team of data … built into solutions. Skills and experience you need Strong experience in Data Engineering and cloud-based data platforms. Hands-on knowledge of Databricks, Spark, SQL and Python. Understanding of Airflow, Terraform and modern engineering practices. Experience building large-scale batch and real-time data pipelines. Strong problem-solving ...

power business critical dashboards, enable self-service analytics, and fuel advanced machine learning models and real-time data products. Working with tools like DBT, Spark, and Airflow, you'll transform high volume raw event data into user-friendly, high impact datasets that support machine learning use cases across … training workflows. Comfort with data modelling and building efficient data marts and warehouses in the cloud. Experience building data pipelines using tools such as Spark and Airflow, or similar technologies, within a cloud environment such as AWS. Familiarity with both real-time and batch data workloads, along with modern ...

domain data (bets, offers, rewards, digital data) into reusable, well‐defined datasets for downstream consumption. Implement data transformation logic using modern tooling (e.g. dbt, Spark, SQL‐based frameworks), ensuring consistency and testability. Build and optimise streaming data pipelines (Kafka/Flink or equivalent) to enable near real‐time data … maintaining scalable data platforms. Strong SQL and data modelling skills, with experience designing analytical datasets. Experience with modern data stack tools (e.g. dbt, Spark, Airflow, or similar orchestration and transformation frameworks). Experience with cloud‐based data platforms, ideally AWS. Understanding of medallion architecture or similar data layering approaches. ...

enable analytics, machine learning and AI across the business. We are looking for an experience Lead Data Engineer with expectational hands on knowledge ofDatabricks, Spark, SQL and Python. Main Responsibilities Lead the development and improvement of the data platform and engineering standards. Mentor and support a team of data … built into solutions. Skills and experience you need Strong experience in Data Engineering and cloud-based data platforms. Hands-on knowledge of Databricks, Spark, SQL and Python. Understanding of Airflow, Terraform and modern engineering practices. Experience building large-scale batch and real-time data pipelines. Strong problem-solving ...

shared platform capabilities for data ingestion, transformation, orchestration, and processing across batch and real-time workloads. Own the integration strategy across tools such asdbt, Spark and Beam Lead the implementation and adoption of data quality, observability, security, testing and deployment including creating and sharing standards Partner closely with Analytics … excellence, and supporting others Hands on experience transitioning to an AI-native Data Engineering landscape A strong Python and SQL engineer with experience usingdbt, Spark and Terraform Strong experience working with GCP (or a comparable cloud platform), Infrastructure as Code, and CI/CD practices. Deep understanding of data ...

As a Junior Software Engineer (Data Products) at Yapily, you will help build and scale our modern, next-generation data platform. This is a junior level role perfect for someone with a strong computer science ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache SparkTM and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/AI models ...

optimisation. You will design and deploy production models with PyTorch, build AI agents and MCP integrations, and architect end-to-end ML pipelines using Spark and Airflow. #J-18808-Ljbffr ...

Proven experience in consultative technical sales or professional services with a proven record of success Commercial solution architecture experience in Big Data technologies e.g. Spark, Hadoop, EMR, MapR etc. Strong verbal and written communications skills are a must, as well as the ability to lead effectively across internal ...

society's experience with software. Come join one of the fastest‐growing startups, supported by best‐in‐class institutions like Battery Ventures, Salesforce Ventures, Spark Capital and Meritech. Pendo Core Values: Bias to Act, Hone Your Craft, The Team is Pendo, and Maniacal Focus. Location: Pendo is a hybrid ...

society's experience with software. Come join one of the fastest-growing startups, supported by best-in-class institutions like Battery Ventures, Salesforce Ventures, Spark Capital and Meritech. Pendo Core Values: Bias to Act, Hone Your Craft, The Team is Pendo, and Maniacal Focus. Location: Pendo is a hybrid ...

person attendance expected at least 3 days per week. At Robinhood, we believe in the power of in-person work to accelerate progress, spark innovation, and strengthen community. Our office experience is intentional, energizing, and designed to fully support high-performing teams. Requires participation in an on-call rotation ...

relevant in data governance, compliance, or automation programs. Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet . Advanced SQL skills; experience with Oracle databases . Experience designing and supporting …/ELT pipelines and data quality frameworks . Working knowledge of Autosys& Apache Airflow Experience with CI/CD tools (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift) Familiarity with AWS S3 for large-scale data storage. Experience supporting Tableau dashboards. Nice-to-Have Experience in regulated or enterprise ...

knowledge across a team. Nice to have: A PhD or equivalent research training in a quantitative field, and/or familiarity with Databricks, dbt, Spark/PySpark, or similar large-scale data platforms. Experience modifying infrastructure-as-code (e.g. Terraform) or contributing to production data pipelines. Experience building shared … with UTXO, EVM, or other blockchain data, graph computation, and/or the cryptocurrency ecosystem. Technologies we use: Python SQL Databricks Dbt PySpark/Spark Terraform AWS Postgres AI at Chainalysis AI is not a feature at Chainalysis - it is a new way of working. One that turns instructions ...

bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F, led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products. THE ROLE We’re building a founding team of Strategic ...

home affairs is highly desirable. Technical expertise in data storage, processing, and containerization technologies such as Cloud or On‐prem Distributed File Systems, Spark, Kubernetes, and Elasticsearch. Our perks and quirks. What makes you Q will help you to realize your full potential, flourish and enjoy what ...

validate transformations and trace data across platforms. Tooling/Delivery Methods Working knowledge of scheduling/orchestration tools such as Autosys and/or Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift … reporting outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Hands-on exposure to DCRM tooling and operational exception management processes. Experience with governance/catalog ...

technology estate Our Data Technology Stack Cloud: Microsoft Azure Data platforms: Azure Databricks, SQL Server, Cosmos DB and Blob Storage Data processing: Python, Spark and SQL Messaging: Azure Service Bus and Event Hubs DevOps and infrastructure: Azure DevOps, GitHub Actions and Terraform You don’t need to have worked … production data pipelines Strong understanding of ETL and ELT architectures and patterns Strong programming and data transformation skills, particularly using technologies such as Python, Spark and SQL Experience working with cloud-based data platforms Experience designing systems for reliability, observability, performance and data quality Strong understanding of data modelling ...

customers. The role requires 5+ years of production-grade software experience, proficiency in Kotlin/Java/Go, and hands-on work with Kafka, Spark/Beam, and cloud warehouses. #J-18808-Ljbffr ...