1 to 25 of 63 Apache Spark Jobs in London

systems are developed and used responsibly and ethically. Tooling the Future: Get hands-on with cutting-edge technologies like Hugging Face, PyTorch, TensorFlow, Apache Spark, Apache Airflow, and other modern data and ML frameworks. Collaborate and Lead: Partner closely with ML Engineers, Data Scientists, and Researchers … services. Programming Prowess: Strong programming skills in Python and SQL are essential. Big Data Ecosystem Expertise: Hands-on experience with big data technologies like Apache Spark, Kafka, and data orchestration tools such as Apache Airflow or Prefect. ML Data Acumen: Solid understanding of data requirements for machine ...

capabilities across the organization. Key Responsibilities Design and implement data architectures on the Databricks Lakehouse Platform Build and optimize ETL/ELT pipelines using Apache Spark and Databricks Implement Delta Lake and medallion architecture (Bronze, Silver, Gold) Ensure data governance, security, and performance optimization Integrate Databricks with cloud … stakeholders to deliver scalable data solutions Required Skills & Qualifications Databricks Certification (e.g., Databricks Certified Data Engineer Associate/Professional) Strong experience with Databricks, Apache Spark, and Delta Lake Proficiency in Python, SQL, and data pipeline development Experience with cloud platforms (Azure, AWS, or GCP) Solid understanding of modern ...

flexibility to attend key meetings onsite. Required: Good knowledge and experience using programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful: R, Apache Spark Very Useful: Python, SQL, JSON, LakeFS ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark Very useful - Python, SQL, JSON, LakeFS. ...

Skills & Experience Strong experience with Python and/or SQL in a production environment. Hands-on experience with modern data engineering frameworks such as: Apache Spark, Databricks Airflow/Prefect/Dagster Kafka/EventHub/Kinesis Experience building data solutions on at least one major cloud platform ...

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

data platform and engineering capability as the business continues to scale its trading operations. They are looking for a Data Engineer with strong Apache Airflow experience to help design and build the pipelines that power trading, analytics and commercial decision-making. This is a great opportunity to join … platform and the opportunity to work with modern open-source technologies . What You’ll Be Doing Designing and building data pipelines orchestrated with Apache Airflow Developing ETL/ELT workflows in Python and SQL to process trading and operational data Building reliable, scalable data infrastructure to support analytics ...

downstream data services across the organisation. Key Responsibilities Design, build, and maintain data pipelines using Databricks Develop ETL/ELT processes with PySpark and Spark SQL Transform and model structured and semi-structured datasets Improve performance, reliability, and cost efficiency of data workloads Ensure compliance with data governance, security … practice routes Required Skills & Experience Active SC Clearance - mandatory and non-negotiable Strong hands-on experience with Databricks in production environments Expert knowledge of Apache Spark , including PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure ...

logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across Cloud (Azure, AWS, GCP) and Big Data (Spark, Kafka) stacks. Collaboration: Work closely with business teams to define data flows and ensure security protocols (encryption/authentication) are met. Technical Toolkit Languages ...

analytics environments. Knowledge of additional AWS services such as S3, Lambda, Athena, Kinesis, EMR, Fargate, or API Gateway. Experience with big data technologies including Spark, Databricks, Hadoop, or Dask. Familiarity with data visualisation tooling such as Plotly. Experience mentoring junior engineers or leading technical initiatives. What You’ll Receive ...

Production Hands-On Experience with Deep Learning Frameworks (TensorFlow, PyTorch, XGBoost) Deep Knowledge of Data Warehousing, ETL/ELT Architectures & Heavy Analytic Environments (Apache Beam, Hadoop, Spark, Hive) Responsibilities: Leading Customer-Facing AI Deployments across Europe, from the Initial Consulting Phase right through to the Live Implementation … Production ML Systems Architecting & Optimising Scalable Data Pipelines, ETL/ELT Solutions & Distributed ML Environments (using Hadoop, Spark & Apache Beam). Deploying Recommendation Engines & Deep Learning Models (TensorFlow, PyTorch) to solve specific Client Problems. Running Technical Design Sessions, Tailoring ML Ops Architectures using Vertex AI & Gen AI. Writing ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

quality and reliable deliverables. Optionally contribute to on-call platform support where required. What We’re Looking For Technical Expertise Strong experience with Databricks, Apache Spark, and cloud-based data architectures (preferably Azure). Excellent programming capability in Python and SQL. Experience orchestrating pipelines with Azure Data Factory ...

making. Collaborating with engineering teams and stakeholders to solve complex technical challenges. Technology exposure Typical environments include: Python, SQL and modern data engineering tooling Spark, Databricks and distributed data platforms AWS, Azure and Google Cloud Modern data lake and data warehouse architectures Who we’re keen to speak with ...

Vault . Hands-on experience exposing Lakehouse solutions to BI tools (e.g. Power BI , Tableau ). Proficiency in Databricks Workflows , Delta Live Tables , and Spark for batch and streaming pipelines. Strong programming capability in Python , Scala , and/or SQL . Practical experience with at least one major cloud ...

Skills & Experience * Proven experience delivering enterprise-scale Databricks solutions end-to-end. * Strong background in Lakehouse Architecture, including structured and unstructured data. * Expertise in Spark, PySpark, Delta Lake, and Databricks workflows. * Experience building scalable ETL/ELT pipelines, including Delta Live Tables. * Strong programming skills in Python, Scala ...

efficiency. Excellent communication skills, able to work across departments, engage with senior leadership, and work directly with our customers. Experience with the AWS stack, Spark, Databricks, and/or Snowflake is desirable. Working for you: We know that your wellbeing and happiness are key to a long and successful ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Improve engineering standards, processes, and tooling ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

uplift practices and delivery maturity. Required Technical Skills & Expertise AWS Data & Analytics (Hands-On Expertise) AWS Glue (ETL/ELT development, PySpark, Glue Catalog) Apache Spark AWS Redshift (modelling, optimisation) AWS Lambda AWS RDS AWS Step Functions Experience with Open Table Formats: Delta Lake Apache Hudi Strong ...

explain technical concepts to varied audiences. Nice to Have Cloud experience: AWS/Azure/GCP . Containerisation: Docker, Kubernetes . Distributed processing: Apache Spark/Flink . Please apply within for further details - Matthew Holmes, Harvey Nash To From Record Yes No Always use these settings ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

validation experience across large datasets. Experience testing integrations with other Salesforce products (Sales Cloud, Service Cloud, Marketing Cloud). Exposure to big data technologies (Spark, Databricks, Snowflake, etc.). Excellent problem-solving skills and ability to work in fast-paced delivery environments. Strong communication and documentation skills. Desirable Experience ...