1 to 25 of 59 Remote Apache Spark Jobs

role We are seeking a high‐skilled Senior Software Engineer with deep expertise in API Development using Spring Boot while also possessing knowledge of Apache Spark‐based Big Data pipelines using Airflow. This hybrid position bridges software engineering and data engineering, requiring the ability to design, build … teams to architect and manage robust, production‐grade data products that power critical analytics and measurement capabilities. You will work with technologies such as Apache Spark, Apache Airflow, Trino, Druid, Spring Boot, StarRocks, and Looker to deliver reliable, high‐performance solutions. The ideal candidate is a proactive ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

tech skills required include • Great SQL • Experience in complex query optimization • Strong Python programming skills with experience in data processing libraries (pandas, NumPy, Apache Spark) • Hands-on experience building and maintaining data ingestion pipelines • Proven track record of optimising queries, code, and system performance • Experience with open-source … data processing frameworks (Apache Spark, Apache Kafka, Apache Airflow) If you are interested in this Data engineering role, please send your latest CV to ross@virtuetech.io Data Engineers x 2 | Open-source data processing, SQL, Python | Energy trading | Up to £85,000 + Bonus + Benefits ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … code generation in regulated environments Strong hands-on experience with AWS ecosystem: AWS Glue, S3, and related data platforms (e.g. Snowflake) Strong experience with Apache Airflow and DAG-based orchestration Experience designing configuration-driven frameworks: Using YAML or JSON Experience building automated testing for data pipelines: Functional equivalence testing ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

streaming (e.g., Kafka), and cloud services (e.g., AWS, GCP, Azure).* Programming Skills: Programming skills in languages like Python (NumPy, pandas, PySpark), Java (Core Spark with Java, functional interface, collections), or Scala with experience in automation and scripting.* Infrastructure Tools: Experience with containerization and orchestration tools like Docker ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

strong programming skills for data processing, automation, and scripting. Azure data stack: Synapse, Databricks, ADLS Gen2, Key Vault — including Linked Services, Managed Identity, and Spark Pool configuration. Azure platform fundamentals: compute, storage, networking (VNets, NSGs, Private Endpoints), identity and RBAC. CI/CD: GitHub Actions (with OIDC federation ...

strong programming skills for data processing, automation, and scripting. Azure data stack: Synapse, Databricks, ADLS Gen2, Key Vault — including Linked Services, Managed Identity, and Spark Pool configuration. Azure platform fundamentals: compute, storage, networking (VNets, NSGs, Private Endpoints), identity and RBAC. CI/CD: GitHub Actions (with OIDC federation ...

engineering, data science, or a hybrid role Strong programming skills in Python and SQL Experience building data pipelines and working with large datasets (e.g., Spark, Airflow, or similar tools) Hands-on experience with cloud platforms (AWS, Azure, or GCP) Familiarity with machine learning frameworks (e.g., scikit-learn, TensorFlow, PyTorch ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

enterprise data strategies, optimize pipelines (ETL/ELT), and incorporate technologies such as cloud platforms (e.g., AWS GovCloud, Azure Government), Big Data tools (e.g., Spark, Hadoop), and relational/NoSQL databases for secure, performant solutions. Designs approaches for automated data labeling, lifecycle management, monitoring, real-time ingestion, and integration ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark or other big-data tools Hybrid working – 3 days per week in a central London office (Tues - Thurs) If you’re excited ...

skills Experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or similar Hands-on experience with transformation frameworks such as dbt or Spark Experience with cloud data warehouses such as Snowflake, BigQuery, Redshift, or Databricks Strong understanding of data modeling principles across analytical and operational systems Experience with ...

Scikit-learn AI/GenAI: prompt engineering (preferred), Claude CLI/Code(preferred), LLMs(preferred) Hugging Face, OpenAI APIs Data Tools (Optional): Pandas, NumPy, Spark APIs & Microservices development Version control (Git) Cloud & DevOps: Experience with AWS, Containers: Docker, ECS/EKS, CI/CD pipelines (GitHub Actions Or Jenkins ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

scale datasets Preferred Qualifications Knowledge or interestin investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka Goldman Sachs Highlights At Goldman Sachs, we commit our people, capital ...