101 to 125 of 366 Apache Spark Jobs in London

experience designing and implementing data solutions on at least one public cloud platform (AWS, Azure, or GCP)Working knowledge of distributed data systems: Apache Spark, Delta Lake, or equivalent (Hadoop, Kafka, Flink)Experience leading technical customer conversations — discovery, whiteboarding, architecture reviewsFamiliarity with one or more: data engineering … Science, Engineering, or a quantitative discipline (or equivalent experience)Nice to Have:Databricks certification or experience with the Databricks PlatformExperience with Unity Catalog, Lakeflow Spark Declarative Pipelines, or MLflowBackground at a data/AI company, cloud provider, or technical consulting firmInterview Process: Recruiter Screen Hiring Manager Screen Design ...

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

stakeholders, and business leaders.Strong presentation, demo, and storytelling skills, with the ability to explain complex technical concepts clearly and credibly.Familiarity with SQL, Python, R, Apache Spark, or similar technologies is a plus.Experience with Alteryx or similar analytics platforms is highly desirable.Ability to work independently in a fast-paced ...

Experience working for a software vendor, SaaS provider, or professional services organisation. Familiarity with Ataccama or similar enterprise data platforms. Exposure to Hadoop/Spark, NoSQL, XSLT, Hive, Kafka, Snowflake, or Databricks. Python or other programming languages. Familiarity with SDLC, code management and upgrades, and Unix/Linux. Perks ...

aggregations, and data transformations.Basic to intermediate proficiency in Python with experience using data libraries (e.g., pandas, NumPy) through coursework, projects, or internships.Exposure to Databricks, Apache Spark, or cloud-based data environments.Experience working with real-world datasets and performing data validation and troubleshooting.Strong curiosity and willingness to learn.Strong analytical ...

Collaborate with cross-functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial ...

Life Sciences value chain.Core Architecture: Expertise in Data Modernization, Microservices, Cloud Architecture, and Lakehouse patterns.Data & AI: Deep hands-on proficiency in Databricks (Spark, PySpark, DLT), GenAI (Agent Core, Copilot Studio, Amazon Bedrock), and Graph Databases (Neo4j).Cloud & Tools: Strong command of AWS (Lambda, Glue, EMR, MWAA, S3), Azure, Airflow ...

complex joins, window functions, aggregation, and query optimization; Experience with a major data warehouse or big data technology such as BigQuery, Snowflake, Redshift, Hive, Spark, Vertica, or StarRocks; Experience with ETL/ELT tools or orchestration frameworks such as Airflow, dbt, or internal frameworks; Proficiency in at least ...

projects from concept to production.Strong knowledge of Python and core ML libraries (e.g., spaCy, NumPy, SciPy, Transformers).Understanding of big data and cloud technologies (Spark, Hadoop, Azure, AWS, or GCP).Deep expertise in MLOps principles, scalable model deployment, model governance, and the practical application of responsible AI within enterprise ...

complex Data & AI programmes, including planning, mobilisation, resourcing, financial management, delivery governance, risk and dependency management.Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data & AI Intelligence capabilities.Strong experience defining enterprise ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL.Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution design through development, testing … building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologiesProficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability.Designing and building solutions on a cloud data warehouse or lakehouse platform (e.g., Databricks ...

Data Engineer (Python SQL Spark Azure Databricks) London to 160kAre you a tech savvy Data Engineer with a first class education? You could be progressing your career working on complex and challenging systems at a Hedge Fund with over $17 billion under management. What … Python or C# and SQL, experience with version control and contributing to a shared codebasesYou have experience with modern data tools and technologies including Apache Spark and Azure Databricks preferredYou have a strong knowledge of data management principles and best practicesYou have experience with data analysis, visualisation tools ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

hands-on engineering experience delivering enterprise data platforms or analytics solutions. Strong commercial experience building and delivering Databricks solutions in production. Deep expertise in Apache Spark, including performance tuning and optimisation at scale. Expert-level Python and SQL, with a track record of writing production-quality, well-tested ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive ...

complex pipelines and data models in a cloud environment Strong proficiency in SQL; solid Python and/or PySpark capabilityDemonstrable experience with Databricks, Apache Spark, and Delta Lake architectureProven ability to design and own enterprise-grade canonical data models across multiple upstream source systemsExperience integrating data from APIs ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

model deployment using MLOps frameworks; Strong Python and SQL programming skills, with experience in PyTorch, TensorFlow, and Hugging Face; Experience with Databricks, Spark, and other data platforms and big data frameworks; Excellent communication skills for engaging C-level stakeholders and aligning technical outcomes with business value; Nice to have ...

security Practical cloud-native experience Preferred Qualifications, Capabilities, and Skills: Experience with modern data technologies such as Databricks or Snowflake Hands-on experience with Spark/PySpark and other big data processing technologies Demonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial ...

with experience in frameworks such as PyTorch, TensorFlow and Hugging Face Experience using data platforms and big data frameworks like Databricks and Spark Excellent communication skills for engaging C-level stakeholders and aligning technical outcomes with business value Nice to have Experience applying AI in drug discovery, safety monitoring ...

application resiliency, and securityPractical cloud-native experiencePreferred Qualifications, Capabilities, and Skills:Experience with modern data technologies such as Databricks or SnowflakeHands-on experience with Spark/PySpark and other big data processing technologiesDemonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial intelligence ...

machine learning, model development, and data engineering, with an appreciation of responsible AI principles.Experience with cloud environments (Azure, AWS, GCP) and tools such as Spark, Hive, and Redshift.Demonstrated ability to lead cross-functional teams and mentor junior practitioners.Ability to communicate complex technical concepts clearly to non-technical audiences ...

learning, model development, and data engineering, with an appreciation of responsible AI principles. Experience with cloud environments (Azure, AWS, GCP) and tools such as Spark, Hive, and Redshift. Demonstrated ability to lead cross-functional teams and mentor junior practitioners. Ability to communicate complex technical concepts clearly to non-technical ...