1 to 25 of 91 Remote Apache Spark Jobs

scalable application components, microservices, and data integration layers to support high-throughput regulatory reporting.Big Data Processing: Develop and optimize distributed data processing jobs using Apache Spark, Hadoop, and Hive to aggregate and transform large-scale financial datasets.Real-Time Streaming: Build and maintain real-time data ingestion and streaming … pipelines using Apache Kafka.Database Management: Design and optimize data models in MongoDB and relational databases, ensuring high performance for complex queries and data storage.Agile Execution: Drive sprint planning, task estimation, and daily progress within an Agile/Scrum framework to ensure timely delivery of project milestones.Requirement Translation: Partner closely ...

business. Key Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platformDevelop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta LakeContribute to platform architecture, engineering standards and best practices across the data estateWork with Product, Analytics and Data Science teams … contribute to continuous improvement across the engineering teamJob Requirements: Significant experience building and supporting scalable, production-grade data platformsStrong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQLGood understanding of cloud-based data engineering, ideally within Microsoft Azure environmentsExperience with Infrastructure as Code tools such ...

designs, and support decommission planning for legacy components Core Technical Skills Azure Synapse & Data Platform Mandatory hands-on expertise with: Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool) Azure Data Lake Storage Gen2 (ADLS Gen2) Delta Lake on Azure (Synapse Lakehouse patterns) Oracle Golden Gate Replication for real … patterns (YAML, PySpark, SQL-driven generation from mapping documents) Databricks & Lakehouse Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred) Strong Apache Spark skills (PySpark/Spark SQL) Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse Ability ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache IcebergProficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)Hands-on experience with CI/… governance, security practices and enterprise integration controlsBackground in financial trading data or related domainsOfficial Databricks certifications such as Certified Data Engineer Professional or Apache Spark DeveloperWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness cover Private medical insurance and dental ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

cloud platform (AWS, Azure or GCP) Experience with data warehousing (Snowflake, Redshift, BigQuery or Synapse) Understanding of data modelling and CI/CD Desirable Spark, Databricks or Kafka Orchestration tools such as Airflow or dbt Infrastructure as Code (Terraform) Experience supporting ML workloads What's On Offer Competitive salary ...

while collaborating with the Data & Analytics Lead, technical colleagues and business stakeholders. Experience in the following areas would also be useful: Materialised Lake Views Apache Spark or Python. Power BI semantic modelling. APIs and cloud business system integration. Data governance, lineage and quality frameworks. Professional services experience. What ...

programming skills, with experience developing and maintaining data pipelines Familiarity with ETL/ELT technologies and data streaming tools such as Airflow, Azure, SSIS, Spark or Kafka Experience working with databases and data platforms such as PostgreSQL, SQL Server, Oracle, Snowflake or similar technologies Knowledge of data visualisation tools ...

Glue and Lambda.· Develop and manage Data Lake solutions using Bronze, Silver, and Gold data architectures.· Implement batch and real-time data processing using Spark Structured Streaming and Auto Loader.· Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.· Establish data quality, governance, and security controls using Unity … Mentor engineers and promote best practices in Databricks and data engineering.What it takesMust-Have Skills & Experience· Expert-level experience with Databricks, PySpark, SQL, and Apache Spark· Strong experience designing and delivering cloud-based data platforms and lakehouse architectures· Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

ideal candidate will have: * Experience in data engineering, ideally aerospace/defence or industrial sectors. * ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka). * Strong Python and SQL; C++, C#, or MATLAB a plus. * Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB. * Data Platform Capabilities ...

hybrid/on-prem environments.Hard SkillsExperience in data engineering, ideally aerospace/defence or industrial sectors.ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka).Strong Python and SQL; C++, C#, or MATLAB a plus.Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB.Data Platform Capabilities: Low-Code No-Code ...

hybrid/on-prem environments.Hard SkillsExperience in data engineering, ideally aerospace/defence or industrial sectors.ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka).Strong Python and SQL; C++, C#, or MATLAB a plus.Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB.Data Platform Capabilities: Low-Code No-Code ...

prem environments. Hard SkillsExperience in data engineering, ideally aerospace/defence or industrial sectors. ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka).Strong Python and SQL; C++, C#, or MATLAB a plus. Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB.Data Platform Capabilities: Low-Code No-Code ...

trusted, reusable data foundations. Essential Experience developing data engineering solutions on cloud platforms Strong SQL and Python skills. Experience with Databricks, Delta Lake and Spark (PySpark or Scala). Knowledge of modern data warehousing or lakehouse architectures. Hands-on experience developing production-grade data pipelines. Understanding of data modelling ...

Cloud Composer and IAM You have strong Python, SQL and PySpark skills You have experience with real-time data streaming using Kafka or Spark You have a good knowledge of Data Lakes, Data Warehousing, Data Modelling You're familiar with DevOps principles, containerisation and CI/CD tools such ...

skills Experience building data pipelines and integrating data from multiple sources Exposure to technologies such as Power BI, Snowflake, SQL Server, PostgreSQL, Spark, Kafka, Airflow, Docker, and Kubernetes Experience within engineering, manufacturing, industrial, or other complex regulated environments would be advantageous Excellent stakeholder management and problem-solving abilities What ...

Product, Engineering, and Data teams.Extra awesomeExperience working in a B2B SaaS product company.Experience with orchestration tools such as Airflow or Argo Workflows.Familiarity with SQL, Spark, Snowflake, or other data processing technologies.Experience building systems that combine structured data, unstructured data, and generative AI.Experience with AI security, privacy, responsible AI, prompt ...

Data teams. Extra awesomeExperience working in a B2B SaaS product company. Experience with orchestration tools such as Airflow or Argo Workflows. Familiarity with SQL, Spark, Snowflake, or other data processing technologies. Experience building systems that combine structured data, unstructured data, and generative AI.Experience with AI security, privacy, responsible ...