1 to 25 of 92 PySpark Jobs in the UK

Azure Data Lake Storage Power BI Microsoft Purview AWS Glue AWS Lake Formation Amazon Redshift Amazon Athena Amazon EMR Amazon S3 SQL, Python, and PySpark CI/CD pipelines and DevOps practices Infrastructure as Code (Terraform preferred) Data Mesh, Master Data Management, and real-time data processing architectures Preferred ...

projects. Hands-on AWS experience. Experience working within Azure environments. Strong SQL skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure ...

ingestion and transformation pipelines from multiple source systems. Strong SQL skills and experience working with relational data models. Experience with Python and/or PySpark for data transformation and automation. Understanding of Power BI, semantic models, DAX and reporting requirements. Experience working with APIs, JSON, CSV, Parquet, Delta tables ...

Duration: 5 months Essential Skills: Informatica Data project experience AWS Azure Desirable Skills: Hands-on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large-scale batch processing. Data engineering at scale in government or similarly complex domains, including performance tuning and data ...

addressing any concerns and ensuring incidents are effectively managed. What you'll need to succeed Knowledge/Qualifications - Expert-level SQL and Python (ideally PySpark). - A strong understanding of DataSecOps; experience with PII handling, data obfuscation, and RBAC (Role-Based Access Control). - Proven ability to design data ...

quality and security best practices Excellent stakeholder management and communication skills Experience operating within Agile delivery environments Desirable Experience Microsoft Fabric Power BI development PySpark REST API integrations Lakehouse architectures AI and Machine Learning platforms Retrieval Augmented Generation (RAG) Vector databases Generative AI solutions Power Platform technologies Data transformation ...

data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/GCP) Experience with modern data warehousing ...

compliance automation Conversational AI/chatbots for banking use cases 2 . Data Engineering & Platform Development Build scalable data pipelines using tools like Spark, PySpark, Databricks, Snowflake, or BigQuery Design data lakes/lakehouse architectures on AWS/Azure/GCP Integrate structured and unstructured data (market feeds, transaction ...

OneLake, Data Factory, Synapse Data Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure ...

Methodology, Master Data Services, Data Quality (DQ), Purview * Azure Data Lake, Azure Data Factory, Logic Apps, Azure Databricks, Azure Functions, Azure Synapse * Python, Scala, PySpark * SharePoint, PowerPivot, Power BI, Data integration layers, Database and storage models, Delta loading, External data loads Desirable Experience * PowerShell scripting, Agile delivery environments ...

Methodology, Master Data Services, Data Quality (DQ), Purview * Azure Data Lake, Azure Data Factory, Logic Apps, Azure Databricks, Azure Functions, Azure Synapse * Python, Scala, PySpark * SharePoint, PowerPivot, Power BI, Data integration layers, Database and storage models, Delta loading, External data loads Desirable Experience * PowerShell scripting, Agile delivery environments ...

deliver practical, value-driven data solutions. Drive continuous improvement across data engineering, reporting, and analytics capabilities. Essential Requirements Strong expertise in SQL and Python (PySpark desirable). Proven experience designing and building cloud-based data platforms from inception. Hands-on experience with Databricks, Spark, modern data lake/lakehouse ...

analytical use cases Ability to work closely with both technical and non-technical stakeholders, translating business requirements into data solutions Experience with Python or PySpark, particularly within Fabric Notebooks Exposure to Azure data services such as Data Factory, Synapse or Data Lake Power BI experience, including supporting reporting ...

continuous improvement, where ideas are shared and supported. Key responsibilities include: • Build & Engineer: Develop and maintain Databricks pipelines, including Delta Live Tables, using PySpark and Python. • Ensure Quality: Maintain data quality, consistency, and lineage across all data sources and destinations. • Orchestrate & Monitor: Implement orchestration, scheduling, and monitoring to ensure ...

working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python, Scala ...

essential - More to React JS and Java script Back-end: Expertise in C# .NET Core, REST APIs, and microservices-based development is essential Python (pyspark): Experience with data pipelines and analytics is a plus. Power BI: Familiarity with developing dashboards and data models. Architecture: Knowledge of microservices, API design ...

+ environment configurations. Analysing current Data Engineering solutions + identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps ...

+ environment configurations. Analysing current Data Engineering solutions + identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps ...

Significant hands-on experience in data engineering, analytics engineering, or related disciplines delivering enterprise data solutions Strong proficiency in Databricks, SQL, and Python/PySpark, including building, optimizing, and troubleshooting ETL pipelines Experience designing scalable, maintainable data architectures and dimensional models for BI, reporting, and analytical use cases Proven ...

initiatives. Review technical designs, data models, ETL/ELT pipelines, and engineering implementation. Provide architectural guidance to data engineers working on Azure Databricks, ADF, PySpark, SQL, and Delta Lake. Collaborate with enterprise architecture, solution architecture, security, infrastructure, DevOps, and business teams. Define CI/CD, DevOps, deployment, monitoring … data modelling, dimensional modelling, canonical data models, and enterprise data models. Strong experience in designing ETL/ELT data pipelines. Good understanding of PySpark, Spark SQL, Python, and SQL. Strong knowledge of data warehouse, data lake, data mart, and semantic layer architecture. Experience in defining data quality, data validation ...

level Performance tuning of complex SQL/T-SQL at scale Confidence taking ownership of workstreams and mentoring junior team members Desirable for both: PySpark/Python in Fabric Notebooks, Azure Data Factory/Synapse/Data Lake, Power BI, AI/ML exposure ...

have: • Proven experience as a Data Engineer working with large, complex datasets. • Hands-on expertise with Apache Spark, and SQL. • Strong proficiency in Python (PySpark preferred). • Experience with AWS cloud services including S3, Glue, Lambda, IAM. • Familiarity with ETL design, data modelling, and data lake/lakehouse concepts. ...

including: CloudWatch, IAM, S3, Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/ ...

operation of Azure-based analytics platforms. Work collaboratively within Agile delivery teams, ensuring adherence to engineering best practice. Required Skills & Experience Python/PySpark and SQL expertise. Proven Databricks experience, including Delta Lake and Unity Catalog. Understanding of Databricks Genie and its application within engineering workflows. Strong knowledge ...

ADLS).· Solid understanding of Scrum practices and a strong Agile mindset.· Advanced proficiency in Python and its key data and ML libraries (NumPy, PySpark, Scikit learn, TensorFlow/PyTorch)· Working knowledge of generative AI models (e.g., ChatGPT, Claude), Databricks GenAI tooling (including embedding models), and modern data structures ...