1 to 25 of 62 Apache Spark Jobs in London

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

Lead, Data Engineering Manager etc. Proven success with modern data infrastructure: distributed systems, batch and streaming pipelines Hands-on knowledge of tools such as Apache Spark, Kafka, Databricks, DBT or similar Experience building, defining, and owning data models, data lakes, and data warehouses Programming proficiency in the likes ...

Lead, Data Engineering Manager etc. Proven success with modern data infrastructure: distributed systems, batch and streaming pipelines Hands-on knowledge of tools such as Apache Spark, Kafka, Databricks, DBT or similar Experience building, defining, and owning data models, data lakes, and data warehouses Programming proficiency in the likes ...

Excellent communication and documentation skills. · Ability to work independently and lead cross-functional teams. · Adaptability to rapidly evolving technological environments PREFERRED SKILLS · Airflow, dbt, Spark, Kafka, RabbitMQ, Redis. · Git, CI/CD pipelines. · Experience with data warehousing solutions such as Snowflake, Redshift, BigQuery, or Synapse. · Exposure to AI/ ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

queries, data modeling, and transformation logic Strong experience with data pipelines and orchestration tools Hands-on experience with big data ecosystems (e.g., Snowflake, Databricks, Spark, AWS/GCP/Azure data services) Advanced skills with Tableau, Power BI, or Looker, including data source optimization and dashboard automation Proficiency ...

cloud environments (AWS, Azure, GCP) including services like S3, BigQuery, Azure ML, etc. Solid understanding of data engineering principles , SQL, and distributed computing frameworks (Spark, Hadoop). Experience in consulting or client-facing roles , with ability to manage stakeholders and deliver strategic insights. Strong knowledge of machine learning algorithms ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

proficiency in programming languages such as Python, Java, or Scala. Experience with cloud platforms like Azure. Knowledge of big data technologies such as Hadoop, Spark, or Kafka. Proficiency in SQL and database management systems. Familiarity with data warehousing concepts and tools. Ability to work collaboratively with cross-functional teams. ...

Quantexa delivery experience Strong knowledge of Entity Resolution, Scoring Framework, Network Generation, and UI configuration 5 -10 years' experience with Big Data technologies (Scala, Spark, Hadoop, Elasticsearch) Solid data modelling and analytics platform design experience Excellent stakeholder communication skills Experience with cloud platforms (AWS, Azure, or GCP) Desirable Experience ...

evaluation techniques. Cloud & MLOps: Experience with AWS, GCP, or Azure; Docker/Kubernetes; MLflow or similar tools. Data Handling: Familiarity with big data frameworks (Spark, Hadoop) and data versioning tools (DVC). Soft Skills: Strong problem-solving, communication, and ability to work in agile teams. Preferred Qualifications Experience with ...

Solid understanding of data warehousing, lakehouse and datalake architectures Familiarity with modern data engineering patterns (ETL/ELT, medallion architecture, observability) Experience working with Apache Spark and large-scale data processing For a full consultation on this exciting new role, please get in touch with ARC IT Recruitment ...

lakehouse and datalake architectures Familiarity with modern data engineering patterns (ETL/ELT, medallion architecture, observability) Excellent Python and SQL skills Experience working with Apache Spark and large-scale data processing For a full consultation on this exciting new role, please get in touch with ARC IT Recruitment ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

technology (AWS or Azure). Strong experience with creation of data ingestion pipeline and ET process. Strong knowledge of big data tool such as Spark, Databricks and Python. Strong understanding of common machine learning techniques and frameworks (e.g. mlflow). Strong knowledge of Natural language processing (NPL) concepts. Strong ...

network analysis for relationship mapping Familiarity with cloud platforms (AWS, Azure, or GCP) for ML model deployment Experience with big data processing frameworks like Apache Spark Knowledge of NoSQL databases (MongoDB, Elasticsearch) for handling unstructured data Experience with data versioning and feature stores for machine learning Proficiency ...

years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks . Good proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure ...

plus. Preferred Qualifications Certifications in Pentaho, Big Data, or Cloud Platforms (AWS/GCP/Azure). Experience with Big Data technologies (Hadoop, Spark) and cloud data services. ...

Lake Azure Synapse Azure SQL and Azure Storage · Azure Data Factory ADF Expertise in building and managing complex data pipelines · Databricks Handson experience with Spark based data processing notebooks and ML workflows · Data Modeling Proficiency in conceptual logical and physical data modeling · SQL Python Advanced skills for data manipulation ...

Lake Azure Synapse Azure SQL and Azure Storage Azure Data Factory ADF Expertise in building and managing complex data pipelines Databricks Handson experience with Spark based data processing notebooks and ML workflows Data Modeling Proficiency in conceptual logical and physical data modeling SQL Python Advanced skills for data manipulation ...

improve data engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance … data pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ...

Optimise data workflows for performance, cost efficiency, and maintainability using modern data-engineering tools and platforms (e.g., Azure Data Factory, AWS Data Pipeline, Databricks, Apache Spark). Support the integration of data into visualisation platforms and analytical environments (e.g., Power BI, ServiceNow). Ensure adherence to data governance ...

outcomes Your Profile Experience: From 1 year in data science or analytics to Principal-level leadership Proficiency in Python, R, SQL, and familiarity with Spark, TensorFlow, PyTorch Strong knowledge of machine learning, statistical modeling, and data engineering principles Experience with cloud platforms (AWS, Azure, GCP) Excellent communication and client ...

data quality. Strong proficiency in one the programming languages, such as Python, Java, Scala, C++, etc. Experience with distributed computing frameworks such as Ray, Spark, and MapReduce. Attention to detail and a strong commitment to data integrity. Excellent problem-solving skills and ability to work in a fast-paced ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need to program using the languages such as SQL, Python, R, YAML ...