1 to 25 of 105 Remote Apache Spark Jobs

data processing. In this role, you will architect and maintain production-grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high-performance solutions. You will collaborate with cross-functional partners across Finance, Accounting, and Analytics to translate complex … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries ...

analytics, reporting, AI/ML, and data-driven decision making across the organization. The ideal candidate possesses strong expertise in modern data engineering practices, Apache Spark, cloud-based data platforms, and lakehouse architectures. As a senior individual contributor, the candidate will provide technical leadership, mentor junior engineers, drive … business stakeholders to deliver trusted, scalable, and analytics-ready data products. Key responsibilities Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms. Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Build, lead ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities. Team Leadership ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling. What You'll Do: Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analyticsDesign, develop, and maintain ultra-high-scale data platforms that process petabytes of dataParticipate in technical reviews of our products and help us develop ...

with Red Hat OpenShift and Ceph storage platforms. Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark). Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training ...

data warehouses and data lakes. (Preferred) Experienced in data ingestion, processing, transformation, and delivery using batch, streaming, and real‐time methods. (Preferred) Experience supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting, and AWS CLI. (Preferred) Great background in building and managing AWS platforms for analytics and data lake workloads, including application and framework management. (Preferred ...

data warehouses and data lakes. (Preferred) Experienced in data ingestion, processing, transformation, and delivery using batch, streaming, and real‐time methods. (Preferred) Experience supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting, and AWS CLI. (Preferred) Great background in building and managing AWS platforms for analytics and data lake workloads, including application and framework management. (Preferred ...

common technical standards and ways of working, and presenting these to the community Here's a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT What we're looking for We'd love ...

strategic planningRequired Skills & ExperienceTechnical SkillsStrong proficiency in SQL and data modellingExcellent programming skills in Python, Scala, or JavaHands-on experience with distributed data frameworks (Spark, Hadoop)Experience with data pipelines and ETL toolsExpertise in at least one cloud platform: Azure (Synapse, Data Factory, Databricks)AWS (S3, Glue, Redshift, Lambda ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/ ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

Rust Data Platforms: BigQuery, Snowflake, Redshift, Databricks, PostgreSQL, MySQL, MongoDB, or other SQL/NoSQL technologies Data Tooling: Airflow, dbt, AWS Glue, Step Functions, Spark, Hadoop, or similar tooling Infrastructure: AWS, Terraform, Docker, Kubernetes, or other cloud/platform technologies CI/CD: GitHub Actions, Jenkins, ArgoCD, or similar ...

deliver agreed scope, acting as the key point of escalation for decisions that impact your area of workDemonstrable experience of building data pipelines using Spark, Python or SQLExperience working with AWS and its data and AI/ML services such as SageMakerExperience of big data and lakehouse platforms ...

deliver agreed scope, acting as the key point of escalation for decisions that impact your area of workDemonstrable experience of building data pipelines using Spark, Python or SQLExperience working with AWS and its data and AI/ML services such as SageMakerExperience of big data and lakehouse platforms ...

Know: Salary: Up to £80,000 per annum Team: Work with cross-functional teams and mentor junior engineers Tools: Microsoft Fabric, Azure ML, MLflow, Spark, Python, SQL, CI/CD, and ML frameworks like TensorFlow, PyTorch, Scikit-learn Scope: End-to-end AI/ML solutions for multi-product …/ML Familiarity with Microsoft Fabric (Data Engineering, Data Science, Real-Time Analytics) and Azure ML, including MLflow Proficiency with tools such as Apache Spark, Azure Data Factory, GIT, Python and SQL Skilled in developing, training and deploying machine learning models with frameworks like TensorFlow, PyTorch or Scikit ...

Skills MUST HAVE: Strong hands-on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

large request volumes, large data volumes, or both. Deep expertise in distributed data systems: streaming (Kafka or similar), batch and ELT/ETL frameworks (Spark, Flink, dbt, Airflow or Argo Workflows), and modern lakehouse or warehouse technologies. Strong production Python experience and awareness of other relevant languages (Java/ ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. Experience ...