251 to 275 of 344 PySpark Jobs

will need for success****Technical knowledge and experience:** * Proven experience using Databricks (essential)* Strong programming skills in Python (essential)* Experience working with: + PySpark + Large-scale data processing environments + Complex behavioural datasets* Experience cleaning, preparing and processing: + Large datasets + Real-world data containing inconsistencies ...

will need for success****Technical knowledge and experience:** * Proven experience using Databricks (essential)* Strong programming skills in Python (essential)* Experience working with: + PySpark + Large-scale data processing environments + Complex behavioural datasets* Experience cleaning, preparing and processing: + Large datasets + Real-world data containing inconsistencies ...

will need for success****Technical knowledge and experience:** * Proven experience using Databricks (essential)* Strong programming skills in Python (essential)* Experience working with: + PySpark + Large-scale data processing environments + Complex behavioural datasets* Experience cleaning, preparing and processing: + Large datasets + Real-world data containing inconsistencies ...

role focuses on migrating data pipelines to the cloud, stabilising environments, and implementing robust error handling and production support patterns. Strong Databricks, SQL and PySpark skills are essential, with experience in legacy ETL tools and cloud migrations. The engineer will repoint dependencies, ensure data integrity, and deliver reusable solutions ...

/Computing or equivalent, 5+ years in Data Engineering with 3 years in a technical lead role, and hands-on experience with Azure Databricks, PySpark, #J-18808-Ljbffr ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

driven patterns). Implement secure authentication, authorization, rate limiting, and multi-tenant access controls. Data Engineering & Pipeline Development Design and maintain data pipelines using PySpark and Databricks, following the medallion architecture (Bronze/Silver/Gold) for structured data transformation and quality. Build and optimize ETL/ELT workflows … architectures. Proficiency with SQL and NoSQL databases (PostgreSQL, DynamoDB, Redis, MongoDB) and familiarity with lakehouse platforms. Experience with or strong willingness to work in PySpark and Databricks, including the medallion architecture (Bronze/Silver/Gold). Hands-on experience with Docker, Kubernetes, CloudFormation, and cloud platforms ...

requirements into scalable data solutions. Key Responsibilities Design scalable enterprise data models and lakehouse architectures Build production-grade data pipelines using SQL, Python/PySpark and Databricks Implement data governance, security and access controls through Unity Catalog Work directly with business stakeholders to understand and prioritise requirements Establish data … data engineering and advanced data modelling Hands-on expertise in Databricks, Delta Lake, Unity Catalog and Databricks Workflows Production experience with SQL, Python/PySpark and dbt and/or Spark Declarative Pipelines Strong understanding of data governance, security and access-control design Excellent stakeholder-management and business-communication ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

plus 10% bonus and an extensive benefits package. What you'll be doingDesign, build and maintain scalable data pipelines and data products using Databricks, PySpark, Spark SQL and Delta Lake.Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems.Shape enterprise data … engineer and help mature a growing data function.What you need to bring“Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc)Advanced SQL and Python coding skills.A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

query, validate and optimise data across cloud-based analytical platforms such as Athena, Redshift and Azure Synapse Build scalable data processing solutions using PySpark, Python and cloud-native orchestration services Implement data quality checks, validation processes, monitoring and logging to ensure trusted and reliable data delivery Support the performance … engineering, integration and analytics challenges Collaborate with experienced engineers, architects and data specialists in supportive, multidisciplinary teams Develop expertise across cloud-native data services, PySpark, SQL and modern engineering practices Access mentoring, knowledge sharing and development opportunities to help you grow your career Contribute to solutions that deliver measurable ...

Data Engineer (Azure | Databricks | PySpark | SQL) Contract | Remote | UK Public Sector Project We are seeking a Data Engineer to support delivery of key data components within our evolving data ecosystem. This role will work closely with architects, analysts, engineers, and wider delivery teams to implement pipeline builds, data models … resolution. Ensure documentation meets acceptance standards and is approved centrally. Actively engage in Agile ceremonies and governance forums. Mandatory Requirements Strong experience with Python, PySpark & SQL for data engineering. Hands-on experience with Azure Databricks. Strong knowledge of Azure Data Factory & Fabric. Experience with Microsoft Purview. Familiarity with Event ...

What you'll do Data Engineering Delivery: Design, build and optimise scalable data pipelines, ETL/ELT workflows and data transformation solutions using Databricks, PySpark, SQL and dbt Cloud, delivering trusted, high-quality data products that support reporting, analytics, AI and data science. Data Architecture & Modelling: Develop and maintain … Advanced Data Engineering Experience (5-8 Years): Strong hands-on experience designing, building and optimising enterprise-scale data pipelines and data platforms using Databricks, PySpark and advanced SQL. Cloud Data Platforms & Architecture: Proven experience working with Azure-based data technologies and designing scalable lakehouse architectures, data models and modern ...

ensure reporting is accurate and useful, and improve engineering standards while supporting colleagues. What you'll do ·Build and optimise pipelines using Azure Databricks, PySpark, SQL, ADF and ADLS Gen2. ·Develop governed lakehouse solutions using Delta Lake, medallion architecture and Unity Catalog. ·Own Power BI semantic models, DAX measures … documentation, Git and CI/CD. ·Mentor colleagues and share good practice across the team. What you'll bring ·Strong production experience with Databricks, PySpark, Delta Lake and Unity Catalog. ·Hands-on Power BI expertise, including data modelling, DAX, RLS and performance optimisation. ·Strong SQL skills and experience with ...

Azure Data Engineer - 1/2 days onsite 18 month FTC Summary: Join a team building a modern Azure-based data platform. This hands-on engineering role involves designing and developing scalable, automated data pipelines ...

Azure Data Engineer - 1/2 days onsite 18 month FTC Summary: Join a team building a modern Azure-based data platform. This hands-on engineering role involves designing and developing scalable, automated data pipelines ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to 80,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python, SQL Essential Skills & Experience Degree ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to £80,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python, SQL Essential Skills & Experience Degree ...

will play a key role in data-related activities for datasets used by Quants and Traders. The role requires strong SQL and Python skills (PySpark, Pandas, Jupyter Notebooks), plus experience with Airflow, Linux/Bash, and data-quality practices. You will collaborate with stakeholders and help improve data pipelines. ...

Arbuthnot Latham is seeking a Senior Data Engineer to develop and maintain data systems that support the Bank's operations and reporting needs. The role involves ensuring compliance with standards while mentoring junior team members. ...