1 to 25 of 117 Remote Apache Spark Jobs

detect threats, analyze patterns, and respond to incidents in real-time.You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines that … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teamsBuild and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructureDesign event-driven systems leveraging Kafka for high-throughput, low-latency message streamingTake ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.Design … build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Design and oversee the delivery of secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure reliable ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Build, lead ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

Testing Guild with common standards such as metrics and use of test tools Here's a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT We'd love to hear from ...

platformsAdvanced Python and SQL skills, with experience in modern engineering practices such as CI/CD and TDDExperience with big data frameworks such as Apache Spark or similar technologiesSolid understanding of data modelling, data warehousing, OLTP and data lake architecturesExperience working with cloud ecosystems such as AWS, Azure ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

strategy. What We're Looking For Essential 5+ years' experience in Data Engineering or a similar role. Strong experience with Python, SQL, Apache Spark, and Cloud platforms (AWS, Azure or GCP) Experience designing and operating production-scale data platforms. Desirable Experience working with AWS. Experience supporting machine learning ...

strategy. What We're Looking For Essential 5+ years' experience in Data Engineering or a similar role. Strong experience with Python, SQL, Apache Spark, and Cloud platforms (AWS, Azure or GCP) Experience designing and operating production-scale data platforms. Desirable Experience working with AWS. Experience supporting machine learning ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

Have: Certifications: Google Cloud or AWS Certified Solutions Architect certification. ETL and Data Pipeline Expertise: Experience with ETL tools and Hadoop-based technologies (e.g., Spark) and/or batch/streaming data pipelines such as Beam and Flink. Data Architecture: Proven ability in designing and building data lake ...

Have:Certifications: Google Cloud or AWS Certified Solutions Architect certification.ETL and Data Pipeline Expertise: Experience with ETL tools and Hadoop-based technologies (e.g., Spark) and/or batch/streaming data pipelines such as Beam and Flink.Data Architecture: Proven ability in designing and building data lake and data warehouse ...

Company descriptionCompany DescriptionPublicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom and Zenith, as well as global accelerator PMX; immersive experience group PMCI and access to integrated platform-based technologies and offerings from CJ, Epsilon … across 20+ marketing platforms.Build and optimize schemas, transformations, and ETL logic for high-performance, reusable data workflows.Contribute directly to the DFP codebase (Python, SQL, Spark) — developing pipelines and infrastructure alongside the engineering team.Establish data versioning, testing, and governance frameworks ensuring reliability, lineage, and compliance.Data Engineering LeadershipDrive best practices ...

optimising ETL/ELT pipelines. Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures. Experience with Azure Data Factory, Azure Synapse, and Apache Spark. Knowledge of data modelling, semantic models, and enterprise reporting solutions. Experience implementing data governance, security, and quality frameworks. Strong understanding of modern data engineering ...

large request volumes, large data volumes, or both.Deep expertise in distributed data systems: streaming (Kafka or similar), batch and ELT/ETL frameworks (Spark, Flink, dbt, Airflow or Argo Workflows), and modern lakehouse or warehouse technologies.Strong production Python experience and awareness of other relevant languages (Java/Kotlin) sufficient ...

twin platforms.* Experience with vector databases, embeddings, semantic search, or RAG.* Experience with streaming data platforms such as Kafka, Pub/Sub, Flink, or Spark Streaming.* Experience supporting closed-loop automation, predictive assurance, or self-healing network use cases.* Experience working with OSS systems, inventory platforms, assurance systems ...

Server, Oracle, Snowflake, Databricks).Experience with data integration (ETL/ELT, APIs).Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) a plus.Domain Knowledge:Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations.In-depth knowledge of regulatory reporting ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS.The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled.* As platform landscape evolved towards cloud-native, AI enabled ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience is a strong plus Databases & Data Warehousing Comfortable working with: RDBMS data warehousing solutions Strong understanding of data modelling aligned with business ...

Experience working for a software vendor, SaaS provider, or professional services organisation. Familiarity with Ataccama or similar enterprise data platforms. Exposure to Hadoop/Spark, NoSQL, XSLT, Hive, Kafka, Snowflake, or Databricks. Python or other programming languages. Familiarity with SDLC, code management and upgrades, and Unix/Linux. Perks ...

control, and Responsible AI/MRM compliance Enterprise AI Integration & Data API- and event‐driven integration of AI into enterprise workflows Data platforms: Databricks, Spark, Snowflake; streaming via Kafka Backend & Distributed Systems Languages: Python (expert), Java/Spring Boot (enterprise standard), Go (plus) Architecture: Microservices, domain‐driven design, event ...