51 to 75 of 255 PySpark Jobs in England

Warehouse structures, notebooks, pipelines and medallion architecture. Strong practical experience of SQL and at least one relevant programming language, such as Python, Postgres, PySpark or Java. Experience designing, developing and supporting ETL or ELT pipelines and data integration processes within a cloud data platform. A strong understanding of data ...

deliver scalable solutions that drive business value. What does this roleinvolve day to day? Data Engineering Design and implement batch/streaming pipelines (PySpark, Spark SQL, DLT, Auto Loader) across bronze, silver & gold layers. Implement CDC patterns, incremental loads, tune partitioning, Z Order, OPTIMIZE/VACUUM. Build robust tests ...

depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience(PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. Deep lakehouse and data platform ...

API Gateway, and CloudWatch. Design, develop, and maintain data pipelines and ETL workflows using technologies such as Glue, Spark/PySpark, Kafka, or similar platforms. Implement and maintain Infrastructure as Code (IaC) using Terraform or equivalent tools. Drive observability, monitoring, logging, and operational readiness across software platforms and services. ...

/reporting outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Exposure to data governance concepts: critical data elements (CDEs), lineage, data quality dimensions, audit frameworks. ...

Generation), Serverless, DWH models, Data Mesh, Data Fabric, Medallion and Data Products. Hands on mindset : Fluent in the language of data, whether SQL, Python, PySpark, R or Scala. Open formats including Parquet, Iceberg and JSON. Agile Ways of Working Run and contribute to agile ceremonies, helping to shape ...

mentor the team Translate business and regulatory requirements into technical solutions Essential Proven Data Architect experience designing production Databricks platforms Strong Delta Lake, Spark (PySpark/SQL) and Unity Catalog skills Financial services background, ideally with regulatory data (BCBS 239, GDPR, FCA/PRA) Data security expertise, including RBAC ...

Data modelling and data warehousing Extract-Transform-Load (ETL) Creating data pipelines Ethics in data and AI Cloud and big data technologies (AWS, Snowflake, PySpark) Working with unstructured data Working with Generative AI – Large language models (LLMs) and image generation (ChatGPT, Gemini) Consulting Skills Effective teamwork Business acumen Time ...

stakeholders to translate business challenges into analytical solutions. YOUR SKILLS AND EXPERIENCE: You will bring strong capability in: Python and SQL skills, with strong PySpark experience Hands-on expertise with Microsoft Fabric and Databricks. Experience building production-grade data pipelines, reusable data assets and machine learning solutions. Experience communicating ...

workflows Defining and enforcing standardised engineering patterns across a shared codebase Operational reliability, including monitoring, structured logging, alerting, and systematic failure recovery Desirable PySpark Experience supporting Airflow in a self‐managed environment Experience in a business building a data platform from the ground up Working Arrangements This ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

London area will be required, with a flexible hybrid approach that supports remote work. Candidates should have hands-on experience with Python, Spark, SQL, PySpark, and PowerBI, and be comfortable working across AWS/Azure/GCP and Snowflake/Data Bricks. SC clearance eligibility is required. #J ...

years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud ...

compliance, controls, remediation, or automation programs (preferably in financial services). Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet. Advanced SQL skills; experience with Oracle databases. Experience designing ...

data engineer and a strong passion to learn. BS/MS in Computer Science or equivalent experience in related fields. Experience in Python, Pandas, PySpark, and Notebooks. SQL knowledge and experience working with relational databases including schema design, access patterns, query performance optimization, etc. Experience with data pipeline technologies ...

Azure Resource Manager/Bicep, or Google Cloud Deployment Manager) to build, maintain, and scale complex cloud environments Strong coding proficiency in Python and PySpark/Spark: you must demonstrate live coding, debugging, and solution‐building skills Solid grounding in distributed data systems architecture: scalable pipelines, streaming architectures, lakehouse ...

security, and governance to support AI agent-driven data products. The ideal candidate has deep SQL skills (SQL Server/Azure SQL), strong Python (PySpark), and experience with Databricks, Synapse or Fabric, plus Power BI semantic modelling. A background in data security and Agile delivery is preferred. #J ...

will design data models, ensure data security, and govern data across the organisation. You'll work with SQL Server/Azure SQL, Python (PySpark), Databricks/Synapse, and Power BI modelling. Experience with Microsoft Fabric is a plus; you will shape data products and guardrails for AI agents. This ...

agents that underpin our data-driven decisions. The role requires deep SQL/T-SQL on SQL Server/Azure SQL, strong Python (PySpark) capabilities, and experience with Databricks, Synapse or Microsoft Fabric, plus Power BI semantic modelling. #J-18808-Ljbffr ...

Data Engineer Key skills: Python SQL Server/SQL Data Warehousing & Data Modelling Data Architecture Data Pipelines & Integration Solutions Microsoft Fabric Azure Cloud Technologies PySpark ETL/ELT Development Data Governance & Security Source Control (Git) Technical Leadership & Team Management You will be a Lead Data Engineer Developer with good ...

record of development in Databricks* Proven track record of working with IaC* Proven track record of working with git* Strong working knowledge of Python (Pyspark) and SQL* Strong working knowledge of data management inside a datalake* Proven ability to explain technical issues and solutions to peers and customers* Proven ...

skills and ensure high-quality deliverables. Data Engineering: Strong proficiency in Python and at least one key programming language commonly used in data engineering (PySpark/SQL strongly preferred, Java/Scala beneficial). Data Modelling: Solid understanding and practical experience in designing and implementing effective data models (e.g. ...

production implementations as lead or architect Deep Unity Catalog experience: you have configured it, not just used it, including RBAC on sensitive data Strong PySpark, Spark SQL and Delta Lake, including performance tuning at scale Production pipeline experience with Lakeflow/DLT, Workflows and DABs Desirable: Databricks Certified Data ...

record of development in Databricks* Proven track record of working with IaC* Proven track record of working with git* Strong working knowledge of Python (Pyspark) and SQL* Strong working knowledge of data management inside a datalake* Proven ability to explain technical issues and solutions to peers and customers* Proven ...

record of development in Databricks Proven track record of working with IaC Proven track record of working with git Strong working knowledge of Python (Pyspark) and SQL Strong working knowledge of data management inside a datalake Proven ability to explain technical issues and solutions to peers and customers Proven ...