1 to 25 of 139 Remote Apache Spark Jobs

detect threats, analyze patterns, and respond to incidents in real-time.You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines that … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teamsBuild and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructureDesign event-driven systems leveraging Kafka for high-throughput, low-latency message streamingTake ...

develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>IT, Express>IT).Build scalable batch and streaming workloads using Apache Spark (PySpark/Scala).Develop and orchestrate data ingestion flows using Apache NiFi for real-time and near-real-time use cases.Optimize …/ELT jobs for scalability, throughput, and resource efficiency.Perform bottleneck analysis and tuning for Spark jobs, NiFi processors, and Ab Initio graphs.Demos and knowledge transfers to stakeholders around product developments.Spread your experience and knowledge to junior engineers.Improve automated testing and deployments using CI/CD pipelines.Expectation to line manage ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

Testing Guild with common standards such as metrics and use of test tools Here's a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT We'd love to hear from ...

platformsAdvanced Python and SQL skills, with experience in modern engineering practices such as CI/CD and TDDExperience with big data frameworks such as Apache Spark or similar technologiesSolid understanding of data modelling, data warehousing, OLTP and data lake architecturesExperience working with cloud ecosystems such as AWS, Azure ...

lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines hands-on technical leadership with test strategy ownership . … reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis for production data issues and platform defects Risk Management & Reporting Identify, assess, and communicate quality risks across releases ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

e.g., AWS, Azure, GCP) and their AI/ML services.Data Engineering: Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL, data pipelines, and automated workflows for moving and processing data.Data Governance: Adherence to data governance, security, and privacy standards ...

Python, or similar languages relevant to data manipulation and scripting.Data Engineering: Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL/ELT, data warehousing, and automated workflows for moving and processing data.Data Governance: Strong understanding and practical experience with ...

strategy. What We're Looking For Essential 5+ years' experience in Data Engineering or a similar role. Strong experience with Python, SQL, Apache Spark, and Cloud platforms (AWS, Azure or GCP) Experience designing and operating production-scale data platforms. Desirable Experience working with AWS. Experience supporting machine learning ...

strategy. What We're Looking For Essential 5+ years' experience in Data Engineering or a similar role. Strong experience with Python, SQL, Apache Spark, and Cloud platforms (AWS, Azure or GCP) Experience designing and operating production-scale data platforms. Desirable Experience working with AWS. Experience supporting machine learning ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

Have: Certifications: Google Cloud or AWS Certified Solutions Architect certification. ETL and Data Pipeline Expertise: Experience with ETL tools and Hadoop-based technologies (e.g., Spark) and/or batch/streaming data pipelines such as Beam and Flink. Data Architecture: Proven ability in designing and building data lake ...

Have:Certifications: Google Cloud or AWS Certified Solutions Architect certification.ETL and Data Pipeline Expertise: Experience with ETL tools and Hadoop-based technologies (e.g., Spark) and/or batch/streaming data pipelines such as Beam and Flink.Data Architecture: Proven ability in designing and building data lake and data warehouse ...

e.g., AWS, Azure, GCP) and their AI/ML services.Data Engineering: Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL, data pipelines, and automated workflows for moving and processing data.Data Governance: Adherence to data governance, security, and privacy standards ...

Python, or similar languages relevant to data manipulation and scripting.Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL/ELT, data warehousing, and automated workflows for moving and processing data.Familiarity with data modelling techniques (e.g., dimensional, relational) and data ...

Company descriptionCompany DescriptionPublicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom and Zenith, as well as global accelerator PMX; immersive experience group PMCI and access to integrated platform-based technologies and offerings from CJ, Epsilon … across 20+ marketing platforms.Build and optimize schemas, transformations, and ETL logic for high-performance, reusable data workflows.Contribute directly to the DFP codebase (Python, SQL, Spark) — developing pipelines and infrastructure alongside the engineering team.Establish data versioning, testing, and governance frameworks ensuring reliability, lineage, and compliance.Data Engineering LeadershipDrive best practices ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

large request volumes, large data volumes, or both.Deep expertise in distributed data systems: streaming (Kafka or similar), batch and ELT/ETL frameworks (Spark, Flink, dbt, Airflow or Argo Workflows), and modern lakehouse or warehouse technologies.Strong production Python experience and awareness of other relevant languages (Java/Kotlin) sufficient ...

twin platforms.* Experience with vector databases, embeddings, semantic search, or RAG.* Experience with streaming data platforms such as Kafka, Pub/Sub, Flink, or Spark Streaming.* Experience supporting closed-loop automation, predictive assurance, or self-healing network use cases.* Experience working with OSS systems, inventory platforms, assurance systems ...

Experience with data warehousing concepts and technologies.Knowledge of other BI tools (e.g., Tableau, Looker) and data modelling techniques.Familiarity with big data technologies (e.g., Hadoop, Spark).Familiarity with modern data architecture (e.g., Data Mesh, Fabric).An understanding of infrastructure as code (e.g., Terraform)Experience with CI/CD pipelines ...

Server, Oracle, Snowflake, Databricks).Experience with data integration (ETL/ELT, APIs).Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) a plus.Domain Knowledge:Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations.In-depth knowledge of regulatory reporting ...