1 to 25 of 54 Remote Apache Spark Jobs

role We are seeking a high‐skilled Senior Software Engineer with deep expertise in API Development using Spring Boot while also possessing knowledge of Apache Spark‐based Big Data pipelines using Airflow. This hybrid position bridges software engineering and data engineering, requiring the ability to design, build … teams to architect and manage robust, production‐grade data products that power critical analytics and measurement capabilities. You will work with technologies such as Apache Spark, Apache Airflow, Trino, Druid, Spring Boot, StarRocks, and Looker to deliver reliable, high‐performance solutions. The ideal candidate is a proactive ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Data Mesh and Data Fabric approaches, distributed data layer deployments, and data product metadata standards. Big Data & NoSQL Stack: Practical experience implementing and configuring Apache Spark, Apache NiFi, Apache Atlas, MinIO, Solr, and other Non-SQL databases. Security & Gateways: Proven experience implementing API Gateways featuring Zero ...

Highly competitive Status: Inside IR35 Working pattern: Full time We are seeking an experienced Technical Delivery Lead with strong expertise in Python, Databricks, and Apache Spark to lead the successful delivery of enterprise-scale data engineering initiatives.. This is a hands-on leadership role combining technical delivery, solution … Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern data architecture and Lakehouse principles. ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. Key Responsibilities Design and architect Databricks-based data platforms and solutions. Build scalable data lakehouse and data warehouse architectures. Design … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. You'll also play a key role in pre-sales activities, solution design, and helping to drive strategic Databricks initiatives … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … code generation in regulated environments Strong hands-on experience with AWS ecosystem: AWS Glue, S3, and related data platforms (e.g. Snowflake) Strong experience with Apache Airflow and DAG-based orchestration Experience designing configuration-driven frameworks: Using YAML or JSON Experience building automated testing for data pipelines: Functional equivalence testing ...

data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/ ...

data solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/ ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

streaming (e.g., Kafka), and cloud services (e.g., AWS, GCP, Azure).* Programming Skills: Programming skills in languages like Python (NumPy, pandas, PySpark), Java (Core Spark with Java, functional interface, collections), or Scala with experience in automation and scripting.* Infrastructure Tools: Experience with containerization and orchestration tools like Docker ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

About the Job you are considering: Results-driven Hadoop Spark Data Engineer with strong expertise in designing and implementing scalable big data solutions using Scala and Apache Spark. Experienced in working with the Hadoop ecosystem, including HDFS, Hive, and YARN, to process and analyze large datasets efficiently. Skilled … Azure. Your Skills: Proficient in Scala programming with strong expertise in functional programming concepts for building scalable data applications. Extensive experience in Apache Spark (Core, SQL, and Streaming) for processing large-scale distributed data efficiently Strong knowledge of Hadoop ecosystem components including HDFS, YARN, Hive, and HBase. Skilled ...

optimising ETL/ELT pipelines. Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures. Experience with Azure Data Factory, Azure Synapse, and Apache Spark. Knowledge of data modelling, semantic models, and enterprise reporting solutions. Experience implementing data governance, security, and quality frameworks. Strong understanding of modern data engineering ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

data models and ETL processes, to support the construction of scalable and efficient data systems. Familiarity with big data tools like DuckDB, Polars or Apache Spark for processing out-of-memory datasets" . Proven experience collaborating with first-team stakeholders within a professional sports environment, demonstrating strong interpersonal ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

Financial Services, multi-market experience Data Integration & System Engineers Build scalable data pipelines supporting decisioning systems. Key Skills: GCP (BigQuery) & Hadoop/Cloudera (mandatory) Spark, Scala, Python, SQL Data pipelines (batch & near real-time) Data modelling, governance & CI/CD Pega CDH integration experience QA/Test Specialists Ensure ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

scale datasets Preferred Qualifications Knowledge or interestin investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka Goldman Sachs Highlights At Goldman Sachs, we commit our people, capital ...