51 to 75 of 194 Remote/Hybrid Apache Spark Jobs

performance support, and creating space for innovation and career growth WHAT YOU NEED TO KNOW (or TECHNICAL SKILLS) Strong hands‐on experience with Databricks, Spark/PySpark, Delta Lake, cloud data platforms, and enterprise‐scale data pipelines. Hands‐on experience designing and operating Databricks lakehouse architectures, including governed data ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS. The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled. As platform landscape evolved towards cloud-native, AI enabled ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

critical data ecosystem while supporting the delivery of high-value data products. Key Responsibilities Design, build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimise batch and streaming data workflows, ensuring reliability, scalability and performance. Contribute to the design and implementation of modern Lakehouse data ...

alignment with business priorities and best practice Implementing robust ETL/ELT pipelines using Azure Data Factory, Synapse Pipelines and Azure Databricks (PySpark/Spark SQL) to support batch and streaming data processing Designing and delivering real-time and batch data ingestion solutions using Azure Event Hubs, Kafka, Databricks ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.)Knowledge of programming languages (e.g. Java, Python, SQL)Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

Experience working for a software vendor, SaaS provider, or professional services organisation. Familiarity with Ataccama or similar enterprise data platforms. Exposure to Hadoop/Spark, NoSQL, XSLT, Hive, Kafka, Snowflake, or Databricks. Python or other programming languages. Familiarity with SDLC, code management and upgrades, and Unix/Linux. Perks ...

Cosmos DB, including SQL API and query optimisation C#, ASP.NET and .NET Core Cosmos DB SDKs and NoSQL data modelling Change Data Feed and Spark integration PySpark and Apache Spark Microsoft Fabric data pipelines Partitioning, indexing and consistency levels Pytest and automated testing Git and GitLab DevOps ...

cloud platforms (GCP or Azure) and Databricks. Familiarity with deploying ML workflows using MLflow, Vertex AI, or Azure ML. Nice‐to‐have: Experience with Spark, CI/CD pipelines, and orchestration tools. Nice‐to‐have: Knowledge of Elasticsearch or digital/web analytics platforms. Nice‐to‐have: Understanding ...

concept to production. Strong knowledge of Python and core ML libraries (e.g., spaCy, NumPy, SciPy, Transformers). Understanding of big data and cloud technologies (Spark, Hadoop, Azure, AWS, or GCP). Deep expertise in MLOps principles, scalable model deployment, and model governance. Excellent communication and stakeholder management skills with ...

including migration from legacy data platforms, lakehouse modernisation, performance optimisation, governance, and production‐grade delivery. Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data & AI Intelligence capabilities. Strong experience defining ...

into business processes and decision-making. Technical Skills: Hands-on experience in one or more of the following areas: Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analytics Cloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCP AI Technologies – GenAI frameworks, Agentic AI, vector databases ...

Data & AI programmes, including planning, mobilisation, resourcing, financial management, delivery governance, risk and dependency management. Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data & AI Intelligence capabilities. Strong experience defining ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL. Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution design through development … scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologies Proficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability. Designing and building solutions on a cloud data warehouse or lakehouse platform (e.g. ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

hands-on engineering experience delivering enterprise data platforms or analytics solutions. Strong commercial experience building and delivering Databricks solutions in production. Deep expertise in Apache Spark, including performance tuning and optimisation at scale. Expert-level Python and SQL, with a track record of writing production-quality, well-tested ...

/Azure, Terraform, Kubernetes, CI/CD, Linux, automation, DevSecOps and SRE Data Engineering - Python, SQL, ETL/ELT, modern cloud data platforms, Spark/Databricks and data architecture AI & Machine Learning - ML Engineering, MLOps, Python, model deployment, evaluation, LLM/GenAI and production AI Technical Leadership - Senior/ ...

/Azure, Terraform, Kubernetes, CI/CD, Linux, automation, DevSecOps and SRE Data Engineering - Python, SQL, ETL/ELT, modern cloud data platforms, Spark/Databricks and data architecture AI & Machine Learning - ML Engineering, MLOps, Python, model deployment, evaluation, LLM/GenAI and production AI Technical Leadership - Senior/ ...

efficient use cases. Profil Experience: 5+ years in data engineering and workflow automation Databricks & Data Engineering: Strong hands-on expertise with Databricks (Delta Lake, Spark, Unity Catalog, workflows, medallion architecture, lakehouse design); Genie experience a plus. Workflow Automation: Proven experience automating business processes with orchestration/integration platforms (like ...

with experience in frameworks such as PyTorch, TensorFlow and Hugging Face Experience using data platforms and big data frameworks like Databricks and Spark Excellent communication skills for engaging C-level stakeholders and aligning technical outcomes with business value Nice to have Experience applying AI in drug discovery, safety monitoring ...

structured data. Experience with batch and near real-time processing across multiple sources and targets. Practical capability with Azure Data Factory and Databricks or Spark, plus Azure Blob Storage or Azure Data Lake. Understanding of data modelling, data marts, metadata-driven development, data dictionaries and lineage. Experience delivering through ...

this role leads the engineering of cloud-native data services for the firm's strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS.The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled. As platform landscape evolved towards cloud-native, AI enabled ...

networking CI/CD The Atlassian Suite (JIRA, Confluence, Bitbucket, Bamboo) Azure Services Machine learning algorithms Big Data (Hadoop, Azure Data Lake, S3, Apache Spark) Additional programming languages – Python, Scala, Typescript, CSS 3/SASS (or other pre-processors) Other Information The successful candidate will be rewarded with ...