1 to 25 of 156 Apache Spark Jobs in London

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities.Team Leadership & Delivery ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline … lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

ensuring outcomes align to resiliency, security, and auditability expectations.Strong proficiency in hands-on development using industry-standard tools and languages (e.g., SQL, Python, Spark, cloud platforms).Preferred Qualifications, Capabilities, and Skills:Hands-on experience with Databricks and proficiency in managing large datasets using Apache Iceberg.Experience architecting and implementing … intelligence.Ability to automate data pipeline deployment and monitoring using CI/CD tools and infrastructure-as-code practices.Expertise with distributed data processing frameworks (e.g., Apache Spark, Hadoop) and real-time data streaming technologies (e.g., Kafka, Kinesis, Spark).Experience collaborating with cross-functional teams to deliver impactful data ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

semi-structured, and unstructured dataHands-on expertise in containerization and orchestration using Docker and Kubernetes for scalable production systemsFamiliarity with distributed computing frameworks (e.g., Spark, Dask), cloud platforms (e.g., AWS, Azure, GCP), and analytics libraries (e.g., pandas, numpy, matplotlib)Exposure to DevOps, DataOps, and MLOps concepts and best practicesExperience ...

building, and operating ETL/ELT pipelines and large-scale data architectures.Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential).Ability ...

high-growth environments.Technical Skills:Experience building and operating ETL/ELT pipelines.Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

plusAbility to write clean, maintainable, scalable, and robust code in PythonFamiliarity with analytics libraries (e.g., pandas, numpy, matplotlib), distributed computing frameworks (e.g., Spark, Dask), and cloud platforms (e.g., AWS, Azure, GCP)Basic understanding or exposure to containerization technologies such as Docker and Kubernetes would be beneficialExposure to software engineering ...

strategic planningRequired Skills & ExperienceTechnical SkillsStrong proficiency in SQL and data modellingExcellent programming skills in Python, Scala, or JavaHands-on experience with distributed data frameworks (Spark, Hadoop)Experience with data pipelines and ETL toolsExpertise in at least one cloud platform:Azure (Synapse, Data Factory, Databricks)AWS (S3, Glue, Redshift, Lambda ...

continuous learning to understand new data management systems Strong proficiency in hands-on development using industry-standard tools and languages (e.g., SQL, Python, Spark, cloud platforms), with a commitment to code quality and delivery excellence Preferred qualifications, capabilities, and skills Hands-on experience with Databricks for scalable data engineering … analytics, and collaborative development, as well as proficiency in managing large datasets using Apache Iceberg for efficient, reliable table management and data lake optimization Experience architecting and implementing enterprise-scale data warehouses on cloud platforms (e.g., AWS, Azure, GCP), including migration from legacy systems Advanced proficiency in designing ...

engineers should be adept at Extract, Transform, Load (ETL) processes. This includes using frameworks such as the Medallion Architecture, as well as tools like Apache Airflow, Fivetran, Azure Data Factory, and Amazon Glue to automate data workflows and ensure data quality. Familiarity with cloud computing services, such … Cloud Platform, is increasingly important. Data engineers often leverage these platforms for scalable data storage and processing capabilities. Knowledge of big data frameworks like Apache Spark, Beam is crucial for handling large datasets and real-time data processing. Understanding data modelling techniques helps data engineers design efficient data ...

practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high-performing migration solutions, leveraging technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML-based configurations.You will thrive in a collaborative, client-facing environment, with a passion for solving complex data … target cloud architecturesData Pipeline Testing & Validation (Core Focus)Test end-to-end ETL data solutions running on AWS services, including AWS Glue and Apache IcebergValidate data feed pipeline migrations transitioning to new AWS Data Lakehouse architecturesTest pipelines underpinned by:AI ETL acceleratorsPython/PySpark transformationsSQL-based logicYAML configuration-driven ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

data quality and consistency across platforms while complying with governance policies.What You Bring:Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs.Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB, Cosmos ...

Azure, GCP).Knowledge of data modelling methodologies (star schemas, Data Vault, Kimball, Inmon).Familiarity with data lake architectures and distributed processing frameworks (e.g., Spark, Hadoop).Experience with version control tools (GitHub, Bitbucket) and CI/CD pipelines.Understanding of data governance, security, and compliance frameworks.Exposure to AI/ML workloads ...

machine learning or data science.Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials.Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives).Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow).Strong experience ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

e.g., AWS, Azure, GCP) and their AI/ML services.Data Engineering: Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL, data pipelines, and automated workflows for moving and processing data.Data Governance: Adherence to data governance, security, and privacy standards ...

Python, or similar languages relevant to data manipulation and scripting.Data Engineering: Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL/ELT, data warehousing, and automated workflows for moving and processing data.Data Governance: Strong understanding and practical experience with ...