26 to 50 of 51 Remote/Hybrid Apache Spark Jobs

Ensure high standards of data quality, governance, security and accessibility across data platforms Develop scalable data transformation logic using Python, SQL and Spark-based frameworks Monitor pipeline performance and troubleshoot data workflows to ensure reliability and efficiency Work closely with data scientists, analysts and engineering teams to support advanced … data services (such as S3, Glue, Redshift, EMR, Athena, Lambda or similar) Strong programming experience with Python and SQL Hands-on experience with Spark or distributed data processing frameworks Proven experience building and maintaining data pipelines, data lakes and data warehouses Strong understanding of data modelling, metadata management ...

Define and enforce data architecture across mesh and domain-driven data products. Implement and govern real-time/batch processing (Kafka, Spark, Glue). Ensure strong metadata, cataloguing, and lineage practices across the enterprise. Lead teams of engineers across global hubs, mentoring and supporting high standards. Knowledge on data ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing … Engineer or Solutions Architect certification (desirable). Databricks Certified Data Engineer or Machine Learning Associate (desirable). Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable). Knowledge of machine learning and AI on Databricks (desirable). ...

Kubernetes Strong Git-based development workflows Experience delivering features end-to-end in distributed teams Confident troubleshooting production issues in real time Desirable Skills Apache Spark (EMR) & Delta Lake PostgreSQL (Aurora) CI/CD using GitHub Actions IaC (Helm, CloudFormation) Monitoring tools (Sentry) Role Responsibilities … React frontend Build and maintain Dockerised microservices deployed to AWS (EKS) Troubleshoot and optimise performance across the stack Support data ingestion and transformation pipelines (Spark-based) Work cross-functionally with product and data specialists Help shape new features and influence engineering standards This role is 100% hands-on engineering ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Python Experience with Kubernetes and Docker Understanding of CI/CD pipelines (eg GitLab) Exposure to data platforms, MLOps or machine learning environments Desirable: Spark or Scala AWS services (eg S3) Elasticsearch or graph databases Vector databases/modern data tooling OIDC/OAuth Node.js or React To apply ...

e.g. Pandas, NumPy, Matplotlib) Data visualisation using Power BI or Tableau ideal Familiarity with Agile delivery methodologies Microsoft Fabric Power Apps, Power Automate, Dataverse Apache Spark/PySpark ...

will be working across different client setups. Typically, that includes: AWS, Azure and GCP depending on the project Tools like Databricks, Snowflake and Synapse Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kafka or Kinesis You will use your expertise to choose the right ...

/ELT) pulling from core platforms, cloud storage, databases and APIs Turn raw, messy, disparate data into curated, analysis-ready datasets Work with Python, Spark and SQL every day Develop models and schemas using a medallion architecture approach Help shape our data architecture with the Data Architect Implement governance … solve a problem you'll fit right in. What you'll bring You'll likely have: Hands-on expertise with Databricks Confidence in Python, Spark and SQL Experience building pipelines in Azure A solid understanding of data modelling and scalable architectures But just as important: You're inclusive ...

team engineering organisations Strong background in distributed systems, large-scale data pipelines and ML model deployment Deep understanding of modern data stacks such as Spark, Databricks or cloud-native architectures Solid understanding of modern frontend engineering, DevOps knowledge and experience delivering machine learning systems Strong communication, organisation and stakeholder ...

with EKS) Proficiency in Git-based development workflows Experience working independently and communicating effectively in distributed teams Desirable : Familiarity with (or happy to learn): Apache Spark (EMR) and Delta Lake PostgreSQL (Aurora) CI/CD (Github Actions) and IaC (Helm, CloudFormation) Monitoring (Sentry) Knowledge/interest in research … improve the TypeScript/React-based frontend application. Build and manage containerised services deployed on AWS, utilising Kubernetes orchestration. Support data and transformation workflows (Spark) Diagnose, troubleshoot, and improve overall application performance. Handle and resolve live production issues efficiently and under time pressure. Collaborate closely with product and data ...

/ELT pipelines integrating large, complex datasets Work with structured, semi-structured, and unstructured data across SQL and NoSQL technologies Develop solutions using Hadoop, Spark, and Splunk in large-scale environments Write maintainable Python code, applying object-oriented and functional programming principles Implement and maintain CI/CD pipelines … sustainability, and recoverability Essential Skills & Experience Proven experience as a Lead or Senior Data Engineer in enterprise-scale environments Hands-on expertise with Hadoop, Spark, and Splunk Advanced Python development skills Experience designing and optimising high-performance data pipelines Strong understanding of CI/CD, source control, and automated ...

data architectures Build scalable ETL workflows and data lake solutions using AWS-native services Engineer high-performance data processing solutions using Python, Java, Scala, Spark and SQL Integrate structured and unstructured data from multiple sources using APIs, scripting and cloud-native tooling Design and build robust AWS cloud data … expertise, particularly across data and analytics services Hands-on experience building ETL processes and data lake/warehouse architectures Proficiency in Python, Java, Scala, Spark and SQL Experience processing large-scale structured and unstructured datasets Experience extracting and integrating data via APIs and custom scripting Strong analytical thinking ...

data architectures Build scalable ETL workflows and data lake solutions using AWS-native services Engineer high-performance data processing solutions using Python, Java, Scala, Spark and SQL Integrate structured and unstructured data from multiple sources using APIs, scripting and cloud-native tooling Design and build robust AWS cloud data … expertise, particularly across data and analytics services Hands-on experience building ETL processes and data lake/warehouse architectures Proficiency in Python, Java, Scala, Spark and SQL Experience processing large-scale structured and unstructured datasets Experience extracting and integrating data via APIs and custom scripting Strong analytical thinking ...

data architectures Build scalable ETL workflows and data lake solutions using AWS-native services Engineer high-performance data processing solutions using Python, Java, Scala, Spark and SQL Integrate structured and unstructured data from multiple sources using APIs, scripting and cloud-native tooling Design and build robust AWS cloud data … expertise, particularly across data and analytics services Hands-on experience building ETL processes and data lake/warehouse architectures Proficiency in Python, Java, Scala, Spark and SQL Experience processing large-scale structured and unstructured datasets Experience extracting and integrating data via APIs and custom scripting Strong analytical thinking ...

Data Engineer Role: Data EngineerLocation: Newcastle Upon TyneSalary: TBC – Depending on experienceLevels: Senior Analyst, Specialist Hybrid Working: 3 days per week in our Newcastle, Cobalt business park office Please Note: Any offer of employment is ...

production-ready PySpark code. Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDs Spark Performance & Optimisation Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning. Experience troubleshooting and improving distributed data processing pipelines. Python ...

data engineering projects. Responsibilities Design, build and deliver data solutions using Databricks Develop and optimise ETL/ELT data pipelines using Spark, Python and SQL Build and maintain data lakes and data warehousing solutions Collaborate with stakeholders to deliver scalable and insight-driven data platforms Contribute to projects involving … Snowflake, Azure Data Factory, Synapse or Microsoft Fabric Skills and Experience Strong experience with Databricks with either Azure or AWS Strong SQL, Python and Spark expertise Proven track record delivering data engineering solutions across full project lifecycles Strong understanding of data modelling and modern data architectures Ability to engage ...

with Azure Cosmos DB including query optimisation and throughput management Thorough understanding of Cosmos DB Change Data Feed (CDF) and integration and debugging of Spark with Cosmos CDF. High technical proficiency with Cosmos DB SDKs (e.g., .NET, Java, Python, Node.js) Thorough understanding of Spark distributed computing concepts. High … technical proficiency in PySpark and Spark Concepts. Experience with concurrency patterns, CLR, and scalable application design. Deep understanding of Azure services including Azure Functions, App Services, AKS, and Logic Apps. Experience with SQL API and familiarity with other APIs. Strong understanding of partitioning, indexing, and consistency levels. Experience with ...

Developing and maintaining scalable data pipelines using Python - Automating manual Excel/VBA workflows into reliable, production-grade systems - Working with large datasets using Spark and Azure (Databricks/Synapse) - Supporting data modelling and analytics use cases (Crossing over to more Data Science driven work) - Driving best practices including … code reviews - Collaborating with stakeholders to deliver actionable data solutions What they re looking for: - Strong Python and SQL skills - Experience or exposure to Spark, big data tools, or cloud platforms (ideally Azure) What s on offer: - The chance to join a business at an exciting stage of growth ...

Developing and maintaining scalable data pipelines using Python - Automating manual Excel/VBA workflows into reliable, production-grade systems - Working with large datasets using Spark and Azure (Databricks/Synapse) - Supporting data modelling and analytics use cases (Crossing over to more Data Science driven work) - Driving best practices including … code reviews - Collaborating with stakeholders to deliver actionable data solutions What they're looking for: - Strong Python and SQL skills - Experience or exposure to Spark, big data tools, or cloud platforms (ideally Azure) What's on offer: - The chance to join a business at an exciting stage of growth ...

experience with Cloudera Data Platform (CDP) on AWS or recent Hadoop administration on similar platforms. AWS administration and platform management experience. Strong experience in Spark, Airflow, Apache Ranger, Impala (AWS CDK is a plus). Agile experience with Scrum, and tools including Jira, Slack. Familiarity with PowerBI ...

Head of Engineering - Remote - UK - £90,000-£130,000 GenAI, ML, Spark Why Join This Company? This is a leadership role for someone who wants to work in a truly AI-first engineering environment, not a business experimenting at the edges. The company is already well ahead … solving rather than legacy ways of working. You'll Work With Cross-functional teams spanning: Web Engineering (React/TypeScript) Data Engineering (distributed systems, Spark-based platforms) Data Science & Machine Learning A highly technical CTO who is deeply invested in AI adoption and modern engineering Product leadership following significant ...

DataEngineer-RemoteWorking LynxarecurrentlyworkingwithalargeITconsultancytohelpthemsourceaDataEngineer.OurclientislargeglobalConsultancywhoworkwithenterpriseclients,solvingtheirbusinessandtechnologyproblemsusingcuttingedgesolutions. Asa DataEngineer ,youwillplayakeyroleindesigning,building,andmaintainingrobustdataplatformsandpipelines.Youllworkalongsidedatascientists,analysts,andengineerstoensurereliable,scalable,andhigh-qualitydatasolutions. Whatyoullbedoing: Designingandbuildingdatapipelines,models,andarchitectures DevelopingandmaintainingETL/ELTprocessesusingmoderntools Implementingdatastoragesolutionsacrossrelational,non-relational,andcloudplatforms Ensuringdataquality,reliability,andoperationalexcellence Leadingthedesignanddeliveryofnewdataproductsandpipelines Collaboratingcloselywithstakeholdersandtechnicalteams Providingtechnicalleadershipandmentoringwithindeliveryteams Whattherelookingfor: Strongexperienceindataengineeringroles Provenleadershiporlinemanagementexperience Hands ...