1 to 25 of 172 Remote Apache Spark Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Pandas, numpy, scipy, scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place … scale. Candidate must have experience: Building end-to-end data pipelines leveraging Python Using orchestration tools to deploy data pipelines, including configuring and updating Spark Jobs Containerizing and deploying applications in cloud environments like AWS. Working with MySQL and PostgreSQL including performance tuning, schema design, and query optimization ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … code generation in regulated environments Strong hands-on experience with AWS ecosystem: AWS Glue, S3, and related data platforms (e.g. Snowflake) Strong experience with Apache Airflow and DAG-based orchestration Experience designing configuration-driven frameworks: Using YAML or JSON Experience building automated testing for data pipelines: Functional equivalence testing ...

About the Job you are considering: Results-driven Hadoop Spark Data Engineer with strong expertise in designing and implementing scalable big data solutions using Scala and Apache Spark. Experienced in working with the Hadoop ecosystem, including HDFS, Hive, and YARN, to process and analyze large datasets efficiently. Skilled … Azure. Your Skills: Proficient in Scala programming with strong expertise in functional programming concepts for building scalable data applications. Extensive experience in Apache Spark (Core, SQL, and Streaming) for processing large-scale distributed data efficiently Strong knowledge of Hadoop ecosystem components including HDFS, YARN, Hive, and HBase. Skilled ...

engineering, data science, or a hybrid role Strong programming skills in Python and SQL Experience building data pipelines and working with large datasets (e.g., Spark, Airflow, or similar tools) Hands-on experience with cloud platforms (AWS, Azure, or GCP) Familiarity with machine learning frameworks (e.g., scikit-learn, TensorFlow, PyTorch ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

C#/C++/C Database: SQL Server, PostgreSQL, Snowflake, Redshift, Aurora, Presto, BigQuery, Oracle DevOps: git, docker, subversion, Kubernetes, Jenkins Additional Technologies: Spark, Databricks, Kafka, Kinesis, Hadoop, Lambda, EMR Popular Certifications: AWS Cloud Practitioner, Microsoft Azure Data Fundamentals, Google Associate Cloud Engineer Additional Information We want everyone ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark or other big-data tools Hybrid working – 3 days per week in a central London office (Tues - Thurs) If you’re excited ...

skills Experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or similar Hands-on experience with transformation frameworks such as dbt or Spark Experience with cloud data warehouses such as Snowflake, BigQuery, Redshift, or Databricks Strong understanding of data modeling principles across analytical and operational systems Experience with ...

/or Extract/Load/Transform (ELT) tools, including both batch and real-time data transmission applications such as SSIS, Informatica, Kafka, Spark, MuleSoft, or equivalent softwareAdvanced knowledge of database technologies such as Microsoft SQL Server, Oracle, or NoSQL databasesAdvanced knowledge of multiple programming languages, such as Python ...

/or Extract/Load/Transform (ELT) tools, including both batch and real-time data transmission applications such as SSIS, Informatica, Kafka, Spark, MuleSoft, or equivalent softwareAdvanced knowledge of database technologies such as Microsoft SQL Server, Oracle, or NoSQL databasesAdvanced knowledge of multiple programming languages, such as Python ...

proficiency in SQL and database technologies Experience with programming languages such as Python, Java, or Scala Hands-on experience with data pipeline tools (e.g., Apache Airflow, Kafka) Familiarity with cloud platforms (AWS, Azure, Google Cloud) Understanding of data warehousing solutions (e.g., Snowflake, Redshift, BigQuery) Strong problem-solving and analytical … skills Preferred Skills Experience with big data technologies (e.g., Hadoop, Spark) Knowledge of data modeling techniques and schema design Familiarity with containerization tools (Docker, Kubernetes) Experience with CI/CD pipelines and DevOps practices Understanding of data security and compliance standards ...

Masters degree in Computer Science, Data Engineering, or related field. 7+ years of professional experience in data engineering . Expert-level proficiency in Databricks, Apache Spark, and SQL . Experience with Kafka, Airflow, or AWS Glue for ETL. Proficiency in CI/CD pipelines and GitHub/GitLab ...

Masters degree in Computer Science, Data Engineering, or related field. 7+ years of professional experience in data engineering . Expert-level proficiency in Databricks, Apache Spark, and SQL . Experience with Kafka, Airflow, or AWS Glue for ETL. Proficiency in CI/CD pipelines and GitHub/GitLab ...

Scikit-learn AI/GenAI: prompt engineering (preferred), Claude CLI/Code(preferred), LLMs(preferred) Hugging Face, OpenAI APIs Data Tools (Optional): Pandas, NumPy, Spark APIs & Microservices development Version control (Git) Cloud & DevOps: Experience with AWS, Containers: Docker, ECS/EKS, CI/CD pipelines (GitHub Actions Or Jenkins ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

through influence with strong collaborator engagement skills. Preferred Experience Digital Twin, asset virtualization, or industrial data platforms Graph technologies (e.g., Neo4j), big data ecosystems (Spark, Hive, Hadoop) Databricks, Python, and modern analytics platforms Delivery of data products at enterprise scale Key Competencies Strategic product thinking with strong execution bias ...

Senior Data Engineer (Scala/Spark) London | Hybrid Working | Initial 12 month contract | £450-550pd PAYE + holiday pay We are partnering with a world-leading technology organisation seeking an experienced Senior Data Engineer to join a high-performing engineering team responsible for building and operating large-scale … machine learning workflows at scale. As a Senior Data Engineer, you will design, build and operate large-scale batch data pipelines using Scala and Apache Spark. Working alongside Machine Learning Engineers, Researchers and Platform Engineers, you will help develop the infrastructure and tooling required to process vast datasets ...

. Prior experience or familiarity with DISA's Big Data Platform or other Big Data systems (e.g. Cloudera CDH Hadoop, MapR Hadoop, Databricks, Apache Spark, etc ) is a plus. We have many more additional great benefits/perks that you can find on our website at . ...

Gitlab, etc.) Prior experience or familiarity with DISA's Big Data Platform or other Big Data systems (e.g. Cloudera CDH Hadoop, MapR Hadoop, Databricks, Apache Spark, etc ) is a plus. DoD 8140/8570 IAT Level II compliance certifications may be required in this position as directed ...