126 to 150 of 154 Permanent PySpark Jobs

Title: Azure Databricks Data Engineer Primary skills: Advanced SQL, Azure Databricks, Azure Data Factory, Azure Datalake. Secondary skills: Azure SQL, PySpark, Azure Synapse. Experience: 10+ Years of Experience Hybrid - 3 Days/Week onsite is MUST About the job We are looking for an experienced Databricks Data Engineer … ingestion and transformation. • Collaborate with Azure stack modules like Data Lakes and SQL DW to handle large volumes of data. • Write SQL, Python, and PySpark code to meet data processing and transformation needs. • Understand business requirements and create data flow processes that meet them. • Develop mapping documents and transformation ...

Title: Azure Databricks Data Engineer Primary skills: Advanced SQL, Azure Databricks, Azure Data Factory, Azure Datalake. Secondary skills: Azure SQL, PySpark, Azure Synapse. Experience: 10+ Years of Experience Hybrid - 3 Days/Week onsite is MUST About the job We are looking for an experienced Databricks Data Engineer … ingestion and transformation. • Collaborate with Azure stack modules like Data Lakes and SQL DW to handle large volumes of data. • Write SQL, Python, and PySpark code to meet data processing and transformation needs. • Understand business requirements and create data flow processes that meet them. • Develop mapping documents and transformation ...

Title: Azure Databricks Data Engineer Primary skills: Advanced SQL, Azure Databricks, Azure Data Factory, Azure Datalake. Secondary skills: Azure SQL, PySpark, Azure Synapse. Experience: 10+ Years of Experience Hybrid - 3 Days/Week onsite is MUST About the job We are looking for an experienced Databricks Data Engineer … ingestion and transformation. • Collaborate with Azure stack modules like Data Lakes and SQL DW to handle large volumes of data. • Write SQL, Python, and PySpark code to meet data processing and transformation needs. • Understand business requirements and create data flow processes that meet them. • Develop mapping documents and transformation ...

. Maintain/implement data warehousing solutions and manage large-scale data storage systems (e.g. Microsoft Fabric) Build and optimise SQL queries, stored procedures, PySpark notebooks and database objects to ensure data performance and reliability. Migrate and modernise legacy databases to cloud-based architectures. Database Administration Administer, monitor … normalisation, indexing, query optimisation). Strong experience with ETL/ELT tools. E.g. Azure Data Factory, Databricks, Synapse Pipelines, SSIS, etc. Experience with Python, PySpark, or Scala for data processing. Familiarity with CI/CD practices. Experience with Data lake, Data warehouse and Medallion architectures. Understanding of API integrations ...

deliver scalable, high‐quality data solutions. Key Responsibilities: Develop, optimise, and maintain ETL/ELT pipelines within Databricks Build reliable data ingestion frameworks using PySpark and Spark SQL Design well-structured data models across medallion/lakehouse architecture Work with DevOps teams to automate deployments using CI/… Ensure compliance with best practices around data governance, security, and quality Requirements: Strong experience with Databricks (jobs, notebooks, workflows, Delta Live Tables) Proficiency in PySpark and Python Hands-on experience with Azure Data Lake, Azure Data Factory, or similar cloud services Understanding of Delta Lake, streaming pipelines, and lakehouse ...

deliver scalable, high‐quality data solutions. Key Responsibilities: Develop, optimise, and maintain ETL/ELT pipelines within Databricks Build reliable data ingestion frameworks using PySpark and Spark SQL Design well-structured data models across medallion/lakehouse architecture Work with DevOps teams to automate deployments using CI/… Ensure compliance with best practices around data governance, security, and quality Requirements: Strong experience with Databricks (jobs, notebooks, workflows, Delta Live Tables) Proficiency in PySpark and Python Hands-on experience with Azure Data Lake, Azure Data Factory, or similar cloud services Understanding of Delta Lake, streaming pipelines, and lakehouse ...

that form the backbone of our Lakehouse platform, enabling accurate analytics, reporting, and regulatory compliance. You'll work with cutting-edge technologies including Databricks , PySpark , and Azure Data Factory , applying best practices in data engineering and governance to support this critical programme. Lead Databricks Engineer: Key Responsibilities Build … maintain Databricks pipelines (batch and incremental) using PySpark and SQL. Orchestrate end-to-end workflows with Azure Data Factory . Develop and optimise Delta Lake tables (partitioning, schema evolution, vacuuming). Implement Medallion Architecture (Bronze, Silver, Gold) for transforming raw data into business-ready datasets. Apply robust monitoring, logging ...

Senior Data Engineer Salary: Up to £70,000 I am working with a forward-thinking organisation that is modernising its data platform to support scalable analytics and business intelligence across the Group. With a strong ...

Senior Data Engineer Salary: Up to £70,000 I am working with a forward-thinking organisation that is modernising its data platform to support scalable analytics and business intelligence across the Group. With a strong ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

dashboards and reports using BI tools like Oracle Business Intelligence Enterprise Edition (OBIEE) and Oracle Analytics Cloud (OAC).Proficiency in programming languages such as Pyspark and PL/SQL for data processing and automation.Have a strong understanding of Azure cloud platform and its services for deploying and managing cloud … based solutions.Experience with Cloud services such as Azure Data Services, ADLS and AKSExperience with Python and PySpark for distributed data processing, along with proficiency in Numpy, Pandas and other data manipulation libraries.Experience in optimizing big data architectures for high availability and performance.Strong problem-solving skills, analytical mindset, and ability ...

Design end-to-end data architectures using Medallion (Bronze/Silver/Gold) patterns Build metadata-driven ingestion pipelines and transformation frameworks Develop advanced PySpark/Spark SQL notebooks for data cleansing and modelling Create production-ready semantic models and support BI teams Implement governance, security, and CI/… clients to translate business requirements into technical solutions Essential Skills Strong experience with Microsoft Fabric workloads (Lakehouse, Data Factory, Pipelines, Notebooks, Semantic Models) Advanced PySpark/Spark SQL for large-scale transformations Integration with Dynamics 365, Dataverse, and/or Business Central CI/CD implementation using Azure DevOps ...

Design end-to-end data architectures using Medallion (Bronze/Silver/Gold) patterns Build metadata-driven ingestion pipelines and transformation frameworks Develop advanced PySpark/Spark SQL notebooks for data cleansing and modelling Create production-ready semantic models and support BI teams Implement governance, security, and CI/… clients to translate business requirements into technical solutions Essential Skills Strong experience with Microsoft Fabric workloads (Lakehouse, Data Factory, Pipelines, Notebooks, Semantic Models) Advanced PySpark/Spark SQL for large-scale transformations Integration with Dynamics 365, Dataverse, and/or Business Central CI/CD implementation using Azure DevOps ...

Telephonic + Zoom Direct Client Requirement Role: AWS Data Engineer We are seeking a skilled AWS Data Engineer who has experience working with Python, PySpark, lambda, Airflow, and Snowflake.Responsibilities: Design, build, and optimize ETLs using Python, PySpark, lambda, Airflow and other AWS services. Create SQL queries to segment … pipelines to ingest data into Amazon Redshift for analytics and reporting. Requirements: Minimum 5 years of experience as Data Engineer. 3+ years of Python, PySpark, and Lambda. Must have experience with Airflow and Snowflake. Advanced SQL query development proficiency Understanding of data modelling principles and techniques. Knowledge of data ...

Description: Primary Skill : PySpark, DataBricks, Databricks Lakehouse 12+ years of overall experience in data engineering, cloud architecture, or analytics platforms. Strong expertise in AWS big-data stack and Databricks Lakehouse. Hands-on experience building enterprise-grade data platforms and multi-zone Lakehouse architectures. Deep knowledge of: o Spark optimization … Customer 360/MDM o Data quality & observability Strong understanding of zero-trust security, compliance, and enterprise governance. Strong scripting/programming in Python, PySpark, SQL. ...

Doing Designing end-to-end Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. Leading client workshops and guiding technical … technologies. What We're Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. CI/CD expertise with Azure DevOps ...

12 Month Fixed Term Contract The Team: The Data Engineering team is positioned in the Delivery business unit of the IT function within the Data & Automation department sitting alongside the Report Engineering, Data Governance, Data ...

optimise CI/CD pipelines in Azure DevOps to automate deployment of workspaces, Unity Catalog, networking, and security. Work with Databricks (Spark/Scala, PySpark) to support ingestion frameworks, data processing, and platform-level libraries. Implement secure connectivity (VNET injection, Private Link, firewall, DNS, RBAC). Monitor, troubleshoot … engineering adoption. Must Have: Proven expertise with Microsoft Azure (networking, security, storage, compute). Strong proficiency in Databricks with hands-on Scala (Spark) and PySpark . Deep experience with Terraform for Azure resource deployment and governance. Hands-on with Azure DevOps pipelines (YAML, agents, service connections). Understanding ...

optimise CI/CD pipelines in Azure DevOps to automate deployment of workspaces, Unity Catalog, networking, and security. Work with Databricks (Spark/Scala, PySpark) to support ingestion frameworks, data processing, and platform-level libraries. Implement secure connectivity (VNET injection, Private Link, firewall, DNS, RBAC). Monitor, troubleshoot … engineering adoption. Must Have: Proven expertise with Microsoft Azure (networking, security, storage, compute). Strong proficiency in Databricks with hands-on Scala (Spark) and PySpark . Deep experience with Terraform for Azure resource deployment and governance. Hands-on with Azure DevOps pipelines (YAML, agents, service connections). Understanding ...

data for critical client projects. How to develop and enhance your knowledge of agile ways of working and working in open source stack (PySpark/PySql). Quality engineering professionals utilise Accenture delivery assets to plan and implement quality initiatives to ensure solution quality throughout delivery. As a Data … contribute to data analytics insights and visualization concepts, methods and techniques We are looking for experience in the following skills: Palantir (Must Have) Python PySpark/PySQL AWS or GCP Set yourself apart: Palantir Certified Data Engineer Certified cloud data engineering (preferably AWS) What ...

data for critical client projects. How to develop and enhance your knowledge of agile ways of working and working in open source stack (PySpark/PySql). Quality engineering professionals utilise Accenture delivery assets to plan and implement quality initiatives to ensure solution quality throughout delivery. As a Data … contribute to data analytics insights and visualization concepts, methods and techniques We are looking for experience in the following skills: Palantir (Must Have) Python PySpark/PySQL AWS or GCP Set yourself apart: Palantir Certified Data Engineer Certified cloud data engineering (preferably AWS) What ...

Pyspark Engineer- (Data Engineering, AWS GLUE) SC Cleared OR Eligible Stevenage (Hybrid) 2-3 days onsite Up to £80,000 High-impact programme - Revolutionary platform I am looking for a Security-Cleared Pyspark expert to take the reins on a range of highly ambitious Data Migration projects supporting … databases and AWS cloud environments. Strong hands-on experience with ETL processes and tools (Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi) or scripting using Python, PySpark, and SQL. Solid understanding of data warehousing and modelling techniques (Star Schema, Snowflake Schema). Familiarity with security frameworks such as GDPR, HIPAA ...

development and optimisation of scalable data and pipelines in a cloud-based environment, building and managing ETL processes using tools such as Azure, Databricks, PySpark and SQL. The role would suit a strong Data Engineer with 2+ years' commercial experience working with cloud-based data tech (Azure & AWS), Python … relevant languages for ETL), relational and non-relational databases. Key skills required: Proficiency in implementing ETL pipelines Azure/AWS cloud proficiency Python/PySpark Strong SQL skills for database management SSRS/SSIS Ability to develop, optimise & maintain complex SQL queries & stored procedures (T-SQL) The role ...

responsible for: Designing end-to-end data architecture aligned with modern best practices. Building and managing ingestion pipelines using Databricks and related tools. Developing PySpark/Spark SQL notebooks for complex transformations and cleansing. Applying governance, security, and CI/CD best practices across cloud environments. Leading technical discussions … producing professional documentation. To be successful in this role, you will have: Hands-on experience with Databricks including Unity Catalog. Strong PySpark/Spark SQL skills for large-scale transformations. Experience integrating with diverse data sources such as APIs, cloud storage and databases. Experience with the Azure cloud data ...