251 to 275 of 572 Apache Spark Jobs in the UK

platforms capable of processing billions of events per day and managing terabytes scale datasets. Develop modern Data Lakehouse platforms leveraging S3-compatible object storage, Apache Iceberg, Spark, and Trino. Build real-time streaming integration using Apache Kafka and Apache Flink for low-latency event processing, enrichment … OpenShift, Terraform, CI/CD, GitOps, and Infrastructure as Code. Lead metadata, lineage, and data discovery capabilities using technologies such as DataHub, OpenMetadata, or Apache Atlas to improve platform transparency and usability. Ensure platform security and regulatory compliance through fine-grained access controls, encryption, secrets management, auditing, and secure ...

skills: - Strong commercial full stack engineering capabilities - node, Typescript, Python, Java etc - Solid front end understanding - React, Next, Vue.js etc - Experience building scalable pipelines (Apache Spark, Airflow, dbt etc) - Experience building data lakes, data warehouses and/or real-time streaming solutions would be a real plus - Strong … modern enterprise technology environment, then please apply now to be considered. £90,000 - £100,000 + remote working + excellent benefits React, Angular, TypeScript, Spark, Airflow, dbt, Kubernetes, OpenShift, AWS, Azure, PostgreSQL, GitHub, ArgoCD ...

join a growing data platform team. You'll play a key role in designing, building and optimising production-grade data pipelines across Databricks, Spark and Delta Lake, while helping shape the engineering standards and architecture behind the wider data platform. What you'll be doing: Build and optimise scalable … Databricks/Spark data pipelines across batch and streaming workloads. Develop Lakehouse solutions using Delta Lake and medallion architecture. Implement data quality, monitoring, governance and security across the platform. Work closely with Data Scientists, Analysts and Product teams to deliver high-quality data products. Contribute to engineering standards, code ...

engineering experience; Eagerness to mentor and technically guide engineering teams; Experience guiding technical decision making; Extensive, hands‐on experience building distributed data pipelines using Apache Spark; Knowledge of how to build and optimize data pipelines, architectures and data sets with the ability to drive additional learning for knowledge ...

MongoDB. And work with PostgreSQL too. The Senior Data Engineer will be well versed in Kafka. The Senior Data Engineer will know all about Apache Spark, Airflow and Iceberg. The Senior Data Engineer will have solid cloud infrastructure knowledge e.g. Google Cloud (GCP). The Senior Data Engineer ...

years' experience delivering enterprise data platforms or analytics solutions. Strong commercial experience delivering Databricks solutions. Consulting or customer-facing delivery experience. Strong understanding of Apache Spark, modern Lakehouse architecture and Unity Catalog. Experience with Azure, AWS or GCP. Experience implementing CI/CD and deployment automation. Excellent communication ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open‐source frameworks Familiarity with AWS‐native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance ...

Monitor and maintain - Monitor data systems for performance issues and make any necessary updates. What You’ll Bring ETL processing languages such as Python Apache Spark/NiFi/Kafka Relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra) AWS and data-related services Palantir Foundry ...

build out technology roadmapsCodebase ownershipTechnical Application ownershipQualificationsStrong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , HarnessStrong experience of developing on Apache Spark/HadoopStrong experience with AWS cloud services including AWS Glue, S3, StepStrong experience building data pipelines and data lakeExperience of the full ...

data lake/warehouse/hub built in GCP. You are confident using the full suite of Google data products, IAC, CI/CD, Spark and Kafka. Our core toolbox includes Google Cloud Big Data technologies, Scala, Java & Python, Jenkins amongst others. We value first principles reasoning to select ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Building and maintaining data pipelines. Data modelling and warehousing fundamentals. dbt or similar modern transformation frameworks Experience with AWS data services. Exposure to Kafka, Spark, streaming data, or modern BI tools. Interest in AI, analytics engineering, and data products THE BENEFITS You will receive a salary ...

Python or Go. Expertise in constructing highly available and fault-tolerant big data environments in the cloud, utilizing NoSQL and RDMS technologies like Hadoop, Spark, PostgreSQL, and cloud-based management tools such as EMR, Athena, and RDS.Ability to implement systems with a security-first mindset, adhering to firm-wide ...

with cloud‐based data warehouses (e.g., Snowflake, GBQ). Familiarity with cloud‐based data pipeline services (e.g. AWS Glue) or distributed processing frameworks like Apache Spark. Equal Opportunity Employment Goldman Sachs is an equal opportunity employer and does not discriminate on the basis of race, colour, religion, sex, national ...

experience in software engineering, MLOps, and deploying machine learning models at scale. Proficiency in SQL, Python, and familiarity with big data technologies (e.g., Kafka, Spark) and/or cloud platforms (e.g., GCP, AWS, or Azure).Industry knowledge: Experience in gaming or digital entertainment is a strong plus. Compensation PhilosophyWe ...

data science and machine learning libraries such as scikit-learn, pandas, PyTorch and/or TensorFlow . Practical experience with Databricks , including MLflow and Spark-based data processing. Hands-on knowledge of Generative AI and LLMs , including prompt engineering, RAG architectures and vector search. Strong understanding of MLOps , including ...

Java, Scala or Go) Extensive experience working with operational data stores, data warehouse, large‐scale data technologies and data lakes Experience using distributed frameworks (Spark, Flink, Beam, Hadoop) Good knowledge of containers (Docker, Kubernetes etc) and experience with cloud platforms such as GCP, Azure or AWS Strong experience working ...

about you: Strong Python knowledge. Ability to read through code. Demonstrable experience collaborating with engineers. Experience with big data technologies such as Hadoop, Spark, or similar. Familiarity with cloud platforms (e.g., AWS, GCP, Azure) and data warehousing solutions. Strong knowledge in at least a few of the following areas ...

experience in software engineering, MLOps, and deploying machine learning models at scale.* Proficiency in SQL, Python, and familiarity with big data technologies (e.g., Kafka, Spark) and/or cloud platforms (e.g., GCP, AWS, or Azure).* Industry knowledge: Experience in gaming or digital entertainment is a strong plus. ...

ability to architect and demo enterprise-grade solutions liveProficiency in SQL, Python, and data engineering tools commonly used in FSI environments (dbt, Spark, Kafka, or equivalent)Excellent executive communication skills with the ability to engage credibly with CDOs, CTOs, and CIOs at global financial institutionsTrack record of leading complex ...

Experience with Infrastructure as Code (Terraform, CloudFormation, or similar). Strong experience with cloud platforms, ideally AWS. Experience with distributed data processing technologies (e.g. Spark) and streaming platforms (e.g. Kafka). Ability to operate at both a strategic and hands-on level when required. Strong communication and stakeholder management ...