76 to 100 of 254 Apache Spark Jobs in London

into actionable insights, and power high-impact trading and AI decisions. Ideal Candidate: 5+ years of experience in data engineering with MPP databases or Apache Spark Strong skills in Python and SQL, with solid data modeling experience Extensive knowledge of AWS/cloud services (ECS, EKS, Lambda, Kinesis ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

hands-on engineering experience delivering enterprise data platforms or analytics solutions. Strong commercial experience building and delivering Databricks solutions in production. Deep expertise in Apache Spark, including performance tuning and optimisation at scale. Expert-level Python and SQL, with a track record of writing production-quality, well-tested ...

mesh concepts and their practical implementation within large organisations Experience integrating Databricks with AWS services including S3, IAM, VPC, and AWS Glue Familiarity with Apache Spark, dbt, data pipelines, and modern analytics architectures Understanding of access management, audit frameworks, and enterprise governance controls Delivery Experience Experience leading data ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

years of strong experience in Python development Experience with system design and leading multi-person development efforts Experience with distributed data processing frameworks like Spark Experience with building scalable APIs, and software applications w/focus on backend infrastructure Experience with cloud platforms and services for data storage ...

with experience in frameworks such as PyTorch, TensorFlow and Hugging Face Experience using data platforms and big data frameworks like Databricks and Spark Excellent communication skills for engaging C-level stakeholders and aligning technical outcomes with business value Nice to have Experience applying AI in drug discovery, safety monitoring ...

security Practical cloud-native experience Preferred Qualifications, Capabilities, and Skills: Experience with modern data technologies such as Databricks or Snowflake Hands-on experience with Spark/PySpark and other big data processing technologies Demonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial ...

platforms. One or more Oracle certifications across OCI, Data Management, AI Services or Autonomous Database. Experience working with complementary technologies such as Python, Spark, Databricks, Azure, AWS, OpenShift, Kubernetes or MLOps platforms. Demonstrated ability to structure client problems, shape solutions and identify opportunities for further value creation. Ability ...

including data quality, data lineage, model testing, prompt testing, access controls, monitoring, explainability, human oversight and governance. Strong data analytics capability, including Python, SQL, Spark, Power BI, Tableau or equivalent tools. Ability to translate complex technical concepts into practical business risks, control implications and clear audit conclusions. Internal Audit ...

platforms. One or more Oracle certifications across OCI, Data Management, AI Services or Autonomous Database. Experience working with complementary technologies such as Python, Spark, Databricks, Azure, AWS, OpenShift, Kubernetes or MLOps platforms. Demonstrated ability to structure client problems, shape solutions and identify opportunities for further value creation. Ability ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … distributed data processing and performance tuning Hands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … distributed data processing and performance tuning Hands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … distributed data processing and performance tuning Hands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake and engines such as Spark, Flink, or Trino and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

Engineering Strong experience working with AWS in data engineering environments. Hands‐on experience with AWS Glue for data integration and processing. Strong experience using Apache Airflow for workflow orchestration and pipeline management. Advanced Python development skills. Experience with Apache Spark or similar distributed data processing technologies would ...

Engineering Strong experience working with AWS in data engineering environments. Hands-on experience with AWS Glue for data integration and processing. Strong experience using Apache Airflow for workflow orchestration and pipeline management. Advanced Python development skills. Experience with Apache Spark or similar distributed data processing technologies would ...

Engineering Strong experience working with AWS in data engineering environments. Hands-on experience with AWS Glue for data integration and processing. Strong experience using Apache Airflow for workflow orchestration and pipeline management. Advanced Python development skills. Experience with Apache Spark or similar distributed data processing technologies would ...

e.g., Java, TypeScript, SQL) Experience building and consuming APIs and event‐driven services in a cloud environment Experience with data processing frameworks such as Apache Spark and working with large structured datasets Working knowledge of LLM‐based application development, including prompting, tool calling, retrieval, and evaluation Experience using … agent frameworks (e.g., Google ADK, LangGraph) and agent protocols such as MCP, A2A, or AG-UI Experience with Databricks, MLflow, and Delta Lake or Apache Iceberg Exposure to model serving on Kubernetes and to model monitoring (drift, latency, accuracy) Familiarity with CRM platforms such as Salesforce and their APIs ...

Warehouses Experience with enterprise data source integration — MS SQL Server (on-premises and Azure SQL) and Oracle Database ingestion into Azure landing zones Strong Apache Spark skills — PySpark or Scala, optimisation, partitioning, and job scheduling in Databricks Experience assessing and remediating partially built or vendor-abandoned data platform ...

frontier AI models. Ideal candidates are hands-on Data Engineers with strong SQL and Python, ETL/ELT, orchestration, streaming, and warehousing experience across Spark, Kafka, Airflow, dbt, Snowflake, BigQuery or Databricks on AWS, GCP or Azure. #J-18808-Ljbffr ...

mentor the team Translate business and regulatory requirements into technical solutions Essential Proven Data Architect experience designing production Databricks platforms Strong Delta Lake, Spark (PySpark/SQL) and Unity Catalog skills Financial services background, ideally with regulatory data (BCBS 239, GDPR, FCA/PRA) Data security expertise, including RBAC ...

tools, and emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated ...

mentor the team Translate business and regulatory requirements into technical solutions Essential Proven Data Architect experience designing production Databricks platforms Strong Delta Lake, Spark (PySpark/SQL) and Unity Catalog skills Financial services background, ideally with regulatory data (BCBS 239, xehkeey GDPR, FCA/PRA) Data security expertise, including ...

/ML offerings. Strong programming skills (Python, Java, C++) and experience with DevOps practices (CI/CD). Knowledge of big data technologies (Hadoop, Spark, Flink). Knowledge of NoSQL databases (MongoDB, Cassandra, DynamoDB), message queueing systems (Kafka, RabbitMQ), and version control systems (Git). Preferred Skills: Hugging Face ...

auction systems Excellent communication skills, able to explain complex technical systems to commercial, operations and senior leadership teams. Experience with Google Cloud, Docker, Kafka, Spark, Airflow, ElasticSearch, ClickHouse or similar production infrastructure Experience owning product roadmaps, release governance or cross-functional delivery across distributed teams and time zones. What ...