1 to 25 of 80 Apache Spark Jobs in London

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

DescriptionRole – Senior ConsultantTechnology – Scala, Apache Spark, SQL, PythonNice to Have - AWS, UNIX shell scriptingLocation – London, UKContext and Job Description: Infosys provides Integration/middleware services for global clients covering a multitude of Integration technologies and products. The services are rendered typically following a global delivery model using … fundamental Data Structures and their usage At least 5+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. Having expertise in Spark Core, Spark SQL and Spark Streaming. Experience with Python, Hadoop, HDFS, Hive ...

modern AI systems. Strong programming experience in Python. Data & Infrastructure Experience with some of the following technologies is highly desirable: Frameworks/Infrastructure Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Libraries Pandas Polars Engineering ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

hands-on Senior Databricks Solutions Architect to support a complex, high-performance data platform within a global organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements … Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding ...

solutions that have created measurable business value. Strong experience using Python or Scala. Hands-on knowledge of Big Data and streaming technologies such as Spark and Kafka. Experience designing data models, ETL and ELT pipelines, data warehouses and data lakes. Practical experience with Google Cloud tools such as BigQuery ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache IcebergProficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)Hands-on experience with CI/… governance, security practices and enterprise integration controlsBackground in financial trading data or related domainsOfficial Databricks certifications such as Certified Data Engineer Professional or Apache Spark DeveloperWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness cover Private medical insurance and dental ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … experience in Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

while collaborating with the Data & Analytics Lead, technical colleagues and business stakeholders. Experience in the following areas would also be useful: Materialised Lake Views Apache Spark or Python. Power BI semantic modelling. APIs and cloud business system integration. Data governance, lineage and quality frameworks. Professional services experience. What ...

Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version control, CI/CD, testing ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI).Data Science: Familiarity or experience with classical NLP techniques (BERT, topic modelling ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic ...

GCP. MSc or equivalent experience in Data Science, Statistics, Mathematics, Computer Science, Artificial Intelligence, Physics, Engineering or a related quantitative discipline. Experience with Spark or distributed data processing frameworks. Experience preparing data pipelines or analytical datasets to support machine learning workloads. Awareness of data engineering, data architecture, data governance ...

data analytics, or data science, with the ability to work effectively with various data types and sources. Experience using big data technologies (e.g. Hadoop, Spark, Hive) and database management systems (e.g. SQL and NoSQL).Graph Database Expertise: Deep understanding of graph database concepts, data modeling, and query languages (e.g. ...

Skills & Experience Strong commercial experience in data engineering within cloud-based environments. Advanced Python and SQL development skills. Hands-on experience with Azure Databricks, Apache Spark and PySpark. Good understanding of Delta Lake, lakehouse architecture and modern data modelling. Experience building and managing production data pipelines, including batch ...

requiredDo not rule yourself out if you do not have these. They are a bonus, not a checklist. Experience with Delta Lake, Iceberg, Spark or Databricks, PalantirExperience optimising LLM serving with tools such as vLLM, TGI or TensorRT LLMSearch and ranking experience. For example Elasticsearch or rerankersBackground in time ...

migration strategies as well as with machine learning, LLMs, and agentic AI workflows. Proficient in programming, debugging, and problem-solving using SQL, Python, Apache Spark, cloud platforms, and Lakehouse architectures and with AI tools. Experience collaborating with Global System Integrators (GSIs) and third-party consulting organizations to drive ...