76 to 100 of 253 Remote Apache Spark Jobs

performance support, and creating space for innovation and career growth WHAT YOU NEED TO KNOW (or TECHNICAL SKILLS) Strong hands‐on experience with Databricks, Spark/PySpark, Delta Lake, cloud data platforms, and enterprise‐scale data pipelines. Hands‐on experience designing and operating Databricks lakehouse architectures, including governed data ...

large request volumes, large data volumes, or both.Deep expertise in distributed data systems: streaming (Kafka or similar), batch and ELT/ETL frameworks (Spark, Flink, dbt, Airflow or Argo Workflows), and modern lakehouse or warehouse technologies.Strong production Python experience and awareness of other relevant languages (Java/Kotlin) sufficient ...

large request volumes, large data volumes, or both. Deep expertise in distributed data systems: streaming (Kafka or similar), batch and ELT/ETL frameworks (Spark, Flink, dbt, Airflow or Argo Workflows), and modern lakehouse or warehouse technologies. Strong production Python experience and awareness of other relevant languages (Java/ ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS.The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled.* As platform landscape evolved towards cloud-native, AI enabled ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

alignment with business priorities and best practice Implementing robust ETL/ELT pipelines using Azure Data Factory, Synapse Pipelines and Azure Databricks (PySpark/Spark SQL) to support batch and streaming data processing Designing and delivering real-time and batch data ingestion solutions using Azure Event Hubs, Kafka, Databricks ...

/continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration.Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.)Knowledge of programming languages (e.g. Java, Python, SQL)Hands-on experience with SQL/NoSQL database design.Degree in STEM, or similar ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.)Knowledge of programming languages (e.g. Java, Python, SQL)Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM ...

Experience working for a software vendor, SaaS provider, or professional services organisation. Familiarity with Ataccama or similar enterprise data platforms. Exposure to Hadoop/Spark, NoSQL, XSLT, Hive, Kafka, Snowflake, or Databricks. Python or other programming languages. Familiarity with SDLC, code management and upgrades, and Unix/Linux. Perks ...

Cosmos DB, including SQL API and query optimisation C#, ASP.NET and .NET Core Cosmos DB SDKs and NoSQL data modelling Change Data Feed and Spark integration PySpark and Apache Spark Microsoft Fabric data pipelines Partitioning, indexing and consistency levels Pytest and automated testing Git and GitLab DevOps ...

cloud platforms (GCP or Azure) and Databricks. Familiarity with deploying ML workflows using MLflow, Vertex AI, or Azure ML. Nice‐to‐have: Experience with Spark, CI/CD pipelines, and orchestration tools. Nice‐to‐have: Knowledge of Elasticsearch or digital/web analytics platforms. Nice‐to‐have: Understanding ...

concept to production. Strong knowledge of Python and core ML libraries (e.g., spaCy, NumPy, SciPy, Transformers). Understanding of big data and cloud technologies (Spark, Hadoop, Azure, AWS, or GCP). Deep expertise in MLOps principles, scalable model deployment, and model governance. Excellent communication and stakeholder management skills with ...

Product, Engineering, and Data teams.Extra awesomeExperience working in a B2B SaaS product company.Experience with orchestration tools such as Airflow or Argo Workflows.Familiarity with SQL, Spark, Snowflake, or other data processing technologies.Experience building systems that combine structured data, unstructured data, and generative AI.Experience with AI security, privacy, responsible AI, prompt ...

including migration from legacy data platforms, lakehouse modernisation, performance optimisation, governance, and production‐grade delivery. Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data & AI Intelligence capabilities. Strong experience defining ...

Data & AI programmes, including planning, mobilisation, resourcing, financial management, delivery governance, risk and dependency management. Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data & AI Intelligence capabilities. Strong experience defining ...

complex Data & AI programmes, including planning, mobilisation, resourcing, financial management, delivery governance, risk and dependency management.Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data & AI Intelligence capabilities.Strong experience defining enterprise ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL. Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution design through development … scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologies Proficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability. Designing and building solutions on a cloud data warehouse or lakehouse platform (e.g. ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL.Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution design through development, testing … building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologiesProficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability.Designing and building solutions on a cloud data warehouse or lakehouse platform (e.g., Databricks ...

Data Engineer (Python SQL Spark Azure Databricks) London to 160kAre you a tech savvy Data Engineer with a first class education? You could be progressing your career working on complex and challenging systems at a Hedge Fund with over $17 billion under management. What … Python or C# and SQL, experience with version control and contributing to a shared codebasesYou have experience with modern data tools and technologies including Apache Spark and Azure Databricks preferredYou have a strong knowledge of data management principles and best practicesYou have experience with data analysis, visualisation tools ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

hands-on engineering experience delivering enterprise data platforms or analytics solutions. Strong commercial experience building and delivering Databricks solutions in production. Deep expertise in Apache Spark, including performance tuning and optimisation at scale. Expert-level Python and SQL, with a track record of writing production-quality, well-tested ...

stakeholder engagement, and a passion for national security. Ideal candidates will have knowledge in some of the following:• ETL processing languages such as Python• Apache Spark/NiFi/Kafka• Relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra)• AWS and data-related services• Palantir FoundryWhy ...

Analytics, Artificial Intelligence or a related field, or equivalent experience, would be effective, although practical experience is equally valuable.“Nice to Have”Azure, Databricks, Apache Spark or comparable cloud and data platforms.Data pipelines, ETL and workflow-orchestration technologies.Machine-learning methods such as forecasting, prediction or classification.Experience with enterprise ...