1 to 25 of 42 Remote Apache Spark Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

you. You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise …/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high levels of data quality, governance and pipeline reliability. Optimise performance across large-scale distributed data processing platforms. Collaborate with ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.What … lake & Fabric. Understand which Azure data focused components to use for which purpose.Microsoft certified architectDatabricksDesign and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Ability to Design, Build and deploy AI/ ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

design and deliver scalable, enterprise-grade data solutions. This role will focus on building and optimising modern data platforms using Python, Databricks and Spark, enabling advanced analytics, reporting and data-driven innovation across the business. Client Details Our client is a leading international financial institution with a long-established … complex global environment. Description Design, build and maintain scalable data pipelines using Python and Databricks Develop and optimise ETL/ELT processes leveraging Spark and Delta Lake Create robust data models and data architectures within modern Lakehouse environments Implement and manage data workflows across the Databricks ecosystem Ensure data ...

seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives. This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement … similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced ...

lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines hands-on technical leadership with test strategy ownership . … reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis for production data issues and platform defects Risk Management & Reporting Identify, assess, and communicate quality risks across releases ...

Rust Data Platforms: BigQuery, Snowflake, Redshift, Databricks, PostgreSQL, MySQL, MongoDB, or other SQL/NoSQL technologies Data Tooling: Airflow, dbt, AWS Glue, Step Functions, Spark, Hadoop, or similar tooling Infrastructure: AWS, Terraform, Docker, Kubernetes, or other cloud/platform technologies CI/CD: GitHub Actions, Jenkins, ArgoCD, or similar ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

strategic thinking. Essential Skills & Experience Strong Microsoft Fabric expertise (Lakehouse, Warehouse, Pipelines, Notebooks) OneLake architecture and management Advanced SQL/T-SQL PySpark and Spark optimisation Data warehousing, modelling and ETL/ELT development Power BI semantic modelling Experience delivering enterprise cloud data platforms Experience integrating multiple business systems ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

Architecture.- Strong programming skills in Python and SQL.- Familiarity with cloud platforms such as AWS, Azure, or GCP.- Knowledge of Snowflake and experience with Spark/PySpark.- Experience in designing and implementing modern data platform engineering solutions.If you are ready to make an impact within a dynamic team ...

multidisciplinary team to support and enhance over 70 active BI products. What You'll Bring Essential Skills & Experience: Core Technologies: Azure Databricks (or similar Spark-based platforms) and Azure Data Factory (ADF). Languages: SQL and Python. Data Platforms: Azure Data Lake and SQL Server RDBMS. ETL Tools ...

driven decision making. What you'll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

scale datasets Preferred Qualifications Knowledge or interestin investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka Goldman Sachs Highlights At Goldman Sachs, we commit our people, capital ...

datasets Preferred Qualifications Knowledge or interest in investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka ABOUT GOLDMAN SACHS At Goldman Sachs, we commit our people, capital ...

systems, ideally in an AWS cloud environment or with Docker and Kubernetes Practical experience with DataLake technologies, such as Databricks or AWS Athena and Apache Spark Practical experience in professional software development (e.g., dependency injection, mocking, TDD, CI/CD pipelines) Confident handling of relational databases (PostgreSQL, Oracle ...

auction systems Excellent communication skills, able to explain complex technical systems to commercial, operations and senior leadership teams. Experience with Google Cloud, Docker, Kafka, Spark, Airflow, ElasticSearch, ClickHouse or similar production infrastructure Experience owning product roadmaps, release governance or cross-functional delivery across distributed teams and time zones. What ...

Company Description Publicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom, and Zenith, as well as global accelerator PMX; immersive experience group PMCI and access to integrated platform‐based technologies and offerings from CJ, Epsilon ...

ownership.Experience designing and enforcing data contracts and quality SLAs in distributed data ecosystems.Proficiency in Python, Java, or Scala, and experience building pipelines with Databricks, Spark, or Kafka.Strong understanding of data governance principles, privacy regulations (GDPR, CCPA), and secure data handling practices.Familiarity with metadata management and data catalog tools (e.g.DataHub ...

platforms that enable real‐time and batch data processing at scale. Responsibilities Drive platform capability improvements – continuously enhance distributed data platforms using knowledge of Spark, Databricks, Kafka, Pulsar, Cassandra, and object stores to maintain best‐in‐class capabilities. Enable application data consumption – understand how applications produce and consume data … structures, partitioning strategies, consistency models, and performance optimisation, as well as OLTP databases such as PostgreSQL and MySQL. Distributed compute frameworks – strong experience with Apache Spark and distributed data processing, including optimisation, performance tuning, and large‐scale batch/streaming workloads. Cloud‐native data platforms – demonstrated experience with ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

both OnPrem and Cloud* Experience with processing large datasets as well as streaming data and unstructured data sources including Cassandra HBase Kafka and Spark* Strong ETL and data pipeline experience particularly with modern big data toolsets such as Airflow and Nifi or equivalent* Strong programming background in either Java ...

partner management and ability to influence across teams and organisations.* Continuous learning and improvement mentality.## **Desired*** Experience with big data technologies (e.g. Hadoop, Hive, Spark).* Experience with generative AI, LLMs, or retrieval-augmented generation (RAG).* Exposure to Agentic AI concepts, including autonomous agents, tool use, and orchestration ...