26 to 50 of 194 Remote/Hybrid Apache Spark Jobs

structured streaming, Delta Lake and Delta Live Tables Proven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache Iceberg Proficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP) Hands-on experience with CI/… security practices and enterprise integration controls Background in financial trading data or related domains Official Databricks certifications such as Certified Data Engineer Professional or Apache Spark Developer We offer EPAM Employee Stock Purchase Plan (ESPP) Protection benefits including life assurance, income protection and critical illness cover Private medical ...

platformsAdvanced Python and SQL skills, with experience in modern engineering practices such as CI/CD and TDDExperience with big data frameworks such as Apache Spark or similar technologiesSolid understanding of data modelling, data warehousing, OLTP and data lake architecturesExperience working with cloud ecosystems such as AWS, Azure ...

doing Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline ...

Factory, Databricks and Azure Data Lake. Advanced SQL and Python (PySpark) skills for data transformation, automation and optimisation. Hands-on experience with Delta Lake, Spark, Databricks Workflows and Structured Streaming. Knowledge of data warehousing, lakehouse architectures and dimensional modelling. Experience delivering scalable, high-quality data solutions to support analytics ...

Factory, Databricks and Azure Data Lake. Advanced SQL and Python (PySpark) skills for data transformation, automation and optimisation. Hands-on experience with Delta Lake, Spark, Databricks Workflows and Structured Streaming. Knowledge of data warehousing, lakehouse architectures and dimensional modelling. Experience delivering scalable, high-quality data solutions to support analytics ...

modern data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python. Experience with data engineering and pipeline technologies such as Apache Airflow and Spark. Experience working with cloud platforms, particularly AWS and Azure. Knowledge of big data technologies and distributed data processing. Understanding of data ...

maintain scalable Databricks pipelines (ETL, streaming, ML workflows). Perform cluster and job performance tuning: optimize cluster sizing, caching, partitioning, and shuffle management. Monitor Spark job metrics, analyze logs, and identify bottlenecks in data throughput or latency. Implement cost-optimization strategies for Databricks jobs and clusters using autoscaling … Databricks REST APIs, Prometheus, or Azure Monitor. Maintain best practices for data engineering performance, testing, and documentation. Technical Skills Required Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake. Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies. Experience tuning ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

Strong experience in IaC and guidance on deploying infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc.) with Apache Spark, Databricks or Hadoop. Good understanding of modern data system design architectures (Data Warehouse, Data Lakes, Data Meshes). Ability to create data ...

with data warehousing concepts and technologies (e.g., Azure Synapse Analytics, Azure Data Lake Storage). Familiarity with big data technologies and frameworks (e.g., Hadoop, Spark) is a plus. Knowledge of data governance and data security best practices. Strong problem-solving skills and attention to detail. Excellent communication and collaboration ...

Nice to Have Hands‐on experience with cloud platforms such as Azure, AWS or GCP. Exposure to modern data platforms such as Snowflake, Databricks, Spark or similar. Experience with MLOps practices such as model deployment, monitoring, versioning or model registries. Exposure to workflow orchestration or deployment tools such ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

Company Description Company DescriptionPublicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom and Zenith , as well as global accelerator PMX ; immersive experience group PMCI and access to integrated platform-based technologies and offerings from … marketing platforms. Build and optimize schemas, transformations, and ETL logic for high-performance, reusable data workflows. Contribute directly to the DFP codebase (Python, SQL, Spark) — developing pipelines and infrastructure alongside the engineering team. Establish data versioning, testing, and governance frameworks ensuring reliability, lineage, and compliance. Data Engineering Leadership Drive ...

Skills MUST HAVE: Strong hands-on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

Skills MUST HAVE: Strong hands‐on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

data pipeline orchestration tools* Containerisation (Docker, EKS, AKS, Kubernetes)* Proficiency in CI/CD principles and tools* Familiarity with open-source data tools (e.g., Spark, Kafka, PostgreSQL)* Infrastructure as Code (Terraform, Ansible)* Air-gapped development* Automation* Client/stakeholder management* Public sector domain understanding* Technical architecture**Experience:*** You will ...

decisions, documentation and continuous improvement initiatives. Key Skills & Experience Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

comfort working through ambiguity. Desirable MLOps excellence: reproducible pipelines, model versioning, CI/CD, observability, and automated evaluation. Data engineering: proficiency with Databricks, Apache Spark, Delta Lake, MLflow, and SQL; experience integrating datasets and maintaining data quality. Model training and optimisation: experience with pre‐training, fine‐tuning, distributed ...

grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience is a strong plus Databases & Data Warehousing Comfortable working with: RDBMS data warehousing solutions Strong understanding of data modelling aligned with business ...

twin platforms.* Experience with vector databases, embeddings, semantic search, or RAG.* Experience with streaming data platforms such as Kafka, Pub/Sub, Flink, or Spark Streaming.* Experience supporting closed-loop automation, predictive assurance, or self-healing network use cases.* Experience working with OSS systems, inventory platforms, assurance systems ...

Server, Oracle, Snowflake, Databricks).Experience with data integration (ETL/ELT, APIs).Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) a plus. Domain Knowledge: Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations. In-depth knowledge of regulatory ...