1 to 25 of 26 Apache Spark Jobs in Central London

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Team Leadership ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

skills. Ability to think creatively and propose innovative solutions. To qualify for this role, you must have: Proven experience and knowledge with PySpark and Apache Spark including the fundamentals of how it works. Experience with cloud platforms (e.g., AWS, Azure, GCP) and data lake architectures. Strong experience ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

Skills MUST HAVE: Strong hands‐on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

role 7Dxperts is looking for a Senior Data Engineer to join our UK team. Key Responsibilities Build scalable pipelines using Databricks, Spark and PySpark. Develop ETL/ELT and data transformation workflows using Python and SQL. Create and maintain Databricks notebooks, jobs, workflows and clusters. Work with Delta Lake ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

drift monitoring; versioning and release management. Cloud architecture (Azure/AWS/GCP), Kubernetes/Docker, serverless, IAM and network security. Data engineering depth (Spark/Databricks; ETL/ELT); cloud-native data + AI architectures. Enterprise integration and SRE principles (SLIs/SLOs, runbooks, rollback). Consulting leadership ...

classification techniques, and algorithms Fluency in a programming language (Python, C,C++, Java, SQL) Familiarity with Big Data frameworks and visualization tools (Cassandra, Hadoop, Spark, Tableau) Desirable Skills & Experience Familiarity with MLOps, AI governance, and responsible AI practices. Experience with modern AI frameworks and tooling such as LangChain, Hugging ...

Feature Store Containerisation and orchestration using Docker, Kubernetes or ECS Infrastructure as Code using Terraform or CloudFormation Large-scale data processing frameworks such as Spark or Dask Knowledge of data governance, compliance and regulated environments Experience delivering solutions within highly regulated industries such as government, finance or healthcare Security ...

optimisation and administration Strong programming experience with Python, Go and/or SQL Experience using dbt for data modelling and transformation Experience working with Apache Airflow or similar orchestration tooling Proven experience designing and building production systems on GCP and/or AWS Strong understanding of event-driven architectures … distributed systems Experience deploying and supporting batch and real-time machine learning workloads Experience working with distributed processing technologies such as Spark or Apache Beam Strong understanding of data quality principles and engineering best practices Experience optimising database performance, troubleshooting issues and maintaining production data systems Ability ...

data science and machine learning libraries such as scikit-learn, pandas, PyTorch and/or TensorFlow . Practical experience with Databricks , including MLflow and Spark-based data processing. Hands-on knowledge of Generative AI and LLMs , including prompt engineering, RAG architectures and vector search. Strong understanding of MLOps , including ...

machine learning and data science frameworks. Hands-on experience with large language models, generative AI, prompt engineering, and RAG architectures. Experience with Databricks, MLflow, Spark, and cloud-based AI platforms (Azure preferred). Strong understanding of MLOps, software engineering best practices, and production deployment pipelines. Experience leading technical teams ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

Hands-on experience with ETL/ELT pipelines, data engineering, and large-scale data processing Experience with containerisation (Docker) and scalable data systems (e.g. Spark, Kafka) Nice-to-Have Experience with AWS SageMaker or similar managed ML platforms Background in safety-critical or regulated industries (aerospace, aviation, or similar ...

such as Government, Public Sector, Security, Justice, Defence, Healthcare or other similarly challenging environments . Experience with large-scale data technologies such as SQL, Spark or Hadoop would also be beneficial. About You You will be a technically credible leader who is still comfortable getting into the detail. ...

Responsibilities Azure Platform Support & Monitoring Support and maintain Azure-based data solutions, including Azure Data Factory pipelines, datasets, linked services, triggers; Azure Databricks (Spark jobs, notebooks, clusters); Azure Machine Learning models and endpoints; Power BI dashboards and dataset refreshes. Monitor and troubleshoot failures in pipelines, jobs, and ML workflows … detailed SOPs, runbooks, knowledge‐base articles, and client documentation. Required Skills and Qualifications Azure Services – Azure Data Factory (pipelines, triggers, parameterization, monitoring); Azure Databricks (Spark, notebooks, job orchestration); Azure Machine Learning (pipelines, model deployment, monitoring); Power BI Service (dataset refreshes, access control, report diagnostics). DevOps & Automation – CI/ ...

statistics, machine learning, and mathematical modelling. Experience designing evaluation frameworks for AI systems, including model performance and monitoring. Hands-on experience with Databricks, MLflow, Spark, or similar cloud-based data platforms. Knowledge of responsible AI, model governance, and explainability. Strong communication skills with the ability to explain complex technical ...

startup, or equivalent experience navigating fast-paced startup lifecycles. Strong understanding of modern data warehouse architectures, real-time analytics, and data governance (e.g., BigQuery, Spark, Kafka). Deep technical expertise in Generative AI development (e.g., prompt engineering, model tuning, vector databases like Vertex AI Vector Search, and AI frameworks ...

their techniques * Experience with data science, big data analytics technology stack, analytic development for endpoint and network security, and streaming technologies (e.g., Kafka, Spark Streaming, and Kinesis). Amazon is an equal opportunities employer. We believe passionately that employing a diverse workforce is central to our success. We make ...

leading Data Management consultancy in London seeks a skilled professional experienced in PySpark and Apache Spark for a hands-on role. You will work closely with clients and play a crucial part in developing innovative solutions using cloud technologies like AWS, Azure, and GCP. Excellent communication and ability ...

Desirable Experience with cloud-based risk platforms and pricing engines Azure Kubernetes Service (AKS) Azure Data Lake Microservices and event-driven architectures Kafka, Redis, Spark, HBase, and HDFS Market Risk platforms covering VaR, Stress Testing, and FRTB Risk exposure reconciliation and reporting platforms Enterprise scheduling tools such as Control ...

programming, hardware-aware optimization, or FPGA-adjacent systems. Experience with ML/data tooling such as PyTorch, JAX, CUDA, GPU clusters, Ray, Kafka, Kubernetes, Spark, or comparable technologies. Understanding of market microstructure, backtesting pitfalls, transaction costs, and the operational needs of live quantitative strategies. Experience in environments operating real ...

Step into a role where your ideas spark innovation and your impact is felt every day. As a Services Account Executive in our Services Sales team, you put the customer at the centre of every decision, shaping how services create measurable value for UK Public Sector customers, with ...

Azure PaaS components, including Azure DevOps, Vault, and other security and configuration services (AKS experience is beneficial) Integrate and optimize data processing frameworks with Spark, Databricks, or Cloudera platform (Hue, Hive, HDFS) Support ETL pipelines built for master data management, customer profiles, and consent management workflows Ensure environments … release cycles Exposure to Azure Cloud and PaaS services, including Azure DevOps and Vault Familiarity with data engineering and big data ecosystems, including Spark and Cloudera stacks (hue, hive, HDFS) Excellent problem‐solving skills and ability to communicate technical solutions effectively Previous experience working on data‐driven projects ...