1 to 25 of 40 Remote Apache Spark Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

evolution of data architecture and help shape the organisations data engineering capabilities. Key Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark and Python. Develop and maintain robust ELT processes to ingest, transform and integrate data from multiple sources. Contribute to the design and evolution … platform performance. Essential Skills & Experience Strong hands-on experience with Databricks . Proven experience designing, implementing and optimising data pipelines. Strong knowledge of Apache Spark and Python . Understanding of Lakehouse architecture and modern data engineering principles. At least five years experience across ELT, data engineering and data ...

Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platform Develop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta Lake Contribute to platform architecture, engineering standards and best practices across the data estate Work with Product, Analytics and Data Science … continuous improvement across the engineering team Job Requirements: Significant experience building and supporting scalable, production-grade data platforms Strong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQL Good understanding of cloud-based data engineering, ideally within Microsoft Azure environments Experience with Infrastructure as Code ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

enterprise cloud strategies. Evaluate and recommend technologies for data storage, processing, streaming, governance, and observability. Provide technical leadership and mentorship to engineering teams implementing Spark, Kafka, Databricks, and cloud-native services. Drive performance optimization, cost management, and operational excellence across the data platform. Support executive stakeholders with migration roadmaps … distributed systems. Demonstrated experience leading enterprise-scale cloud migration initiatives involving hundreds of terabytes or petabytes of data. Strong experience with: o Databricks o Apache Spark o Apache Kafka o AMPS, MQ, Hadoop, or experience with other enterprise messaging systems o AWS cloud services o Large-scale ...

Senior Data Engineer Role (Python/Spark/Databricks) - Hybrid - Contract - Banking Role - Senior Data Engineer Rate - £850 p/d (Inside IR35) Duration - 6 months with very likely extension Location - Hybrid/Liverpool Street (London) - 3 days per week in a Liverpool Street office About the Role Global … delivering critical data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working ...

solutions while ensuring compliance with healthcare data governance and regulatory requirements. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, Delta Lake, Apache Spark, SQL, and Python. Build and optimize ETL/ELT processes to support enterprise reporting, analytics, and operational data needs. Develop efficient data … datasets. Implement data validation, monitoring, and observability processes to ensure data reliability and operational excellence. Design and support workflow orchestration solutions using Databricks Workflows, Apache Airflow, or similar tools. Contribute to cloud-based data modernization initiatives and enterprise analytics platforms. Develop and maintain CI/CD pipelines and deployment ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

understanding of data modeling. Familiarity with data governance, lineage, and quality frameworks. Fluency with AI tools and ADLC Working knowledge of Databricks Experience with Apache Spark Preferred: Hands-on experience with AWS data services (e.g., S3, Glue, Lambda, MSK). Proficient in CI/CD pipelines, automated testing ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

Designed and delivered modern data platforms end to end on AWS and/or Databricks Still writes production-grade code: strong Python and SQL, Spark, dbt or equivalent, infrastructure-as-code (Terraform preferred) Track record in a consultancy, systems integrator or other client-facing setting comfortable being accountable ...

into business processes and decision-making. Technical Skills: Hands-on experience in one or more of the following areas: Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analytics Cloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCP AI Technologies – GenAI frameworks, Agentic AI, vector databases ...

efficient use cases. Profil Experience: 5+ years in data engineering and workflow automation Databricks & Data Engineering: Strong hands-on expertise with Databricks (Delta Lake, Spark, Unity Catalog, workflows, medallion architecture, lakehouse design); Genie experience a plus. Workflow Automation: Proven experience automating business processes with orchestration/integration platforms (like ...

years at principal or staff scope Proven expertise in cloud-native data platforms (Azure, AWS, or GCP) and distributed systems Deep knowledge of Databricks, Spark, Kafka, Flink, and Lakehouse architectures Strong background in data lineage, governance, metadata management, and streaming design patterns Experience leading large-scale data streaming ...

communication and technical writing skills. Ability to explain technical concepts to diverse audiences. Preferred Skills: Experience with large-scale data platforms such as Databricks, Spark, Hadoop, or Hive. Experience developing, evaluating, or deploying machine learning and AI solutions, including work with large language models, and cloud environments (e.g. ...

Engineering Strong experience working with AWS in data engineering environments. Hands-on experience with AWS Glue for data integration and processing. Strong experience using Apache Airflow for workflow orchestration and pipeline management. Advanced Python development skills. Experience with Apache Spark or similar distributed data processing technologies would ...

Stack Data Engineer should have most of the following key skills: Strong commercial experience designing and implementing data pipelines at an enterprise scale – Airflow, Apache spark etc Experience developing API’s and microservices to expose data – Java, Python etc Experience building data lakes, data warehouses and/… engineering within a modern enterprise technology environment, then please apply now to be considered. £90,000 – £100,000 + remote working + excellent benefits Spark, Airflow, dbt, Kubernetes, OpenShift, AWS, Azure, PostgreSQL, GitHub, ArgoCD ...

Redshift, Glue, EMR) or GCP (BigQuery, Dataflow, Pub/Sub) Strong coding abilities in Python/Java Advanced SQL skills Hands-on experience with Spark and Kafka Excellent written and spoken skills in Dutch & English Bachelor's degree Proactive and adaptable mindset in dynamic environments Proactive and adaptable Strong ...

about you: Strong Python knowledge. Ability to read through code. Demonstrable experience collaborating with engineers. Experience with big data technologies such as Hadoop, Spark, or similar. Familiarity with cloud platforms (e.g., AWS, GCP, Azure) and data warehousing solutions. Strong knowledge in at least a few of the following areas ...

effectively with technical teams including Data Scientists and MLOps Engineers. Desirable Skills Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/ ...

partner management and ability to influence across teams and organisations. Continuous learning and improvement mentality. Desired Experience with big data technologies (e.g. Hadoop, Hive, Spark). Experience with generative AI, LLMs, or retrieval-augmented generation (RAG). Exposure to Agentic AI concepts, including autonomous agents, tool use, and orchestration ...