1 to 25 of 92 PySpark Jobs in England

ingestion and transformation pipelines from multiple source systems. Strong SQL skills and experience working with relational data models. Experience with Python and/or PySpark for data transformation and automation. Understanding of Power BI, semantic models, DAX and reporting requirements. Experience working with APIs, JSON, CSV, Parquet, Delta tables ...

projects. Hands-on AWS experience. Experience working within Azure environments. Strong SQL skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure ...

quality and security best practices Excellent stakeholder management and communication skills Experience operating within Agile delivery environments Desirable Experience Microsoft Fabric Power BI development PySpark REST API integrations Lakehouse architectures AI and Machine Learning platforms Retrieval Augmented Generation (RAG) Vector databases Generative AI solutions Power Platform technologies Data transformation ...

data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/GCP) Experience with modern data warehousing ...

Duration: 5 months Essential Skills: Informatica Data project experience AWS Azure Desirable Skills: Hands-on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large-scale batch processing. Data engineering at scale in government or similarly complex domains, including performance tuning and data ...

Methodology, Master Data Services, Data Quality (DQ), Purview * Azure Data Lake, Azure Data Factory, Logic Apps, Azure Databricks, Azure Functions, Azure Synapse * Python, Scala, PySpark * SharePoint, PowerPivot, Power BI, Data integration layers, Database and storage models, Delta loading, External data loads Desirable Experience * PowerShell scripting, Agile delivery environments ...

Methodology, Master Data Services, Data Quality (DQ), Purview * Azure Data Lake, Azure Data Factory, Logic Apps, Azure Databricks, Azure Functions, Azure Synapse * Python, Scala, PySpark * SharePoint, PowerPivot, Power BI, Data integration layers, Database and storage models, Delta loading, External data loads Desirable Experience * PowerShell scripting, Agile delivery environments ...

OneLake, Data Factory, Synapse Data Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure ...

Exposure to metadata management, data governance, and data quality frameworks Experience with CI/CD (Azure DevOps, Gi t) and ideally scripting (Python/PySpark) Job Offer £700-£800 per day (Inside IR35) Hybrid working (2-3 days in London) Initial 6-month contract with extension potential ...

analytical use cases Ability to work closely with both technical and non-technical stakeholders, translating business requirements into data solutions Experience with Python or PySpark, particularly within Fabric Notebooks Exposure to Azure data services such as Data Factory, Synapse or Data Lake Power BI experience, including supporting reporting ...

working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python, Scala ...

+ environment configurations. Analysing current Data Engineering solutions + identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps ...

+ environment configurations. Analysing current Data Engineering solutions + identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps ...

Experience managing multi-engine compute environments CI/CD experience with Azure DevOps, GitHub, and Databricks Asset Bundles Strong programming skills in PySpark, Databricks SQL, and T-SQL Understanding of Medallion Architecture and modern data modelling approaches (Star Schema, Data Vault 2.0) Experience with performance optimisation in Databricks ...

Azure. Experience with other platforms like AWS, Snowflake, or Databricks is also beneficial. Hands-on expertise with Data Factory (Fabric or Azure), notebooks (Python, PySpark), advanced SQL, and robust data modelling and performance optimisation skills. Proven track record in building scalable data pipelines and implementing modern architectures such ...

engineering tools and storage solutions (e.g., ADLS, Spark, Databricks, SQL/NoSQL databases) Experience with distributed computing and big data processing frameworks such as PySpark Knowledge of infrastructure‐as‐code tools such as Terraform and Helm Advanced/Specialist Expertise Experience building and deploying GenAI solutions using frameworks such ...

level Performance tuning of complex SQL/T-SQL at scale Confidence taking ownership of workstreams and mentoring junior team members Desirable for both: PySpark/Python in Fabric Notebooks, Azure Data Factory/Synapse/Data Lake, Power BI, AI/ML exposure ...

hands-on experience with modern cloud data platforms (e.g. Databricks or Snowflake), ideally within AWS environments Advanced programming and data engineering skills in Python, PySpark and SQL, with a solid understanding of ETL pipelines and data transformation Proven experience in data modelling (end-to-end), including performance optimisation using ...

technical leader and mentor , raising standards across the data engineering community What you’ll bring Essential Deep expertise in Databricks (Spark/PySpark, Delta Lake, Unity Catalog, Workflows) Strong experience building ETL/ELT pipelines , both batch and streaming Advanced Python and SQL skills Hands-on experience with cloud ...

Data Lake and Azure Synapse Analytics Solid understanding of ETL processes, data integration and pipeline development Advanced SQL skills as well as Python/PySpark experience for data processing Exposure to Power BI or other reporting tools is beneficial What's on Offer Salary up to £65,000 Fully ...

Data Lake and Azure Synapse Analytics Solid understanding of ETL processes, data integration and pipeline development Advanced SQL skills as well as Python/PySpark experience for data processing Exposure to Power BI or other reporting tools is beneficial What's on Offer Salary up to £65,000 Fully ...

Data Lake and Azure Synapse Analytics Solid understanding of ETL processes, data integration and pipeline development Advanced SQL skills as well as Python/PySpark experience for data processing Exposure to Power BI or other reporting tools is beneficial What's on Offer Salary up to £65,000 Fully ...

ADLS).· Solid understanding of Scrum practices and a strong Agile mindset.· Advanced proficiency in Python and its key data and ML libraries (NumPy, PySpark, Scikit learn, TensorFlow/PyTorch)· Working knowledge of generative AI models (e.g., ChatGPT, Claude), Databricks GenAI tooling (including embedding models), and modern data structures ...

design decisions. Required Skills & Experience Strong analytical skills with experience working with large-scale and unstructured datasets. Python development experience, including technologies such as PySpark, Pandas and PyArrow. Expertise in distributed data processing frameworks, particularly Apache Spark. Experience building and maintaining ETL pipelines using technologies such as Apache Airflow ...

looking for: Strong background in data engineering and software development Proven experience working with Databricks (essential) Hands-on expertise with tools such as Python, PySpark, Spark, and SQL Experience building and maintaining data pipelines and data platforms Solid understanding of data modelling and data architecture principles Experience working with ...