76 to 100 of 101 PySpark Jobs

connectivity issues across AWS, Databricks and downstream integrations. Technical Environment: AWS, Databricks, Unity Catalog, Palantir Foundry, Terraform, IaC, Delta, Parquet, CSV, JSON, Python, PySpark, SQL, CI/CD, GitLab, GitHub Actions. ...

engineering is done across the organisation. Skills & Experience Strong hands-on experience with Azure data platforms Advanced SQL skills Commercial experience using Databricks and PySpark Proven background building and maintaining scalable data pipelines If you're looking for a role where you can combine technical depth, ownership, and influence ...

Inside IR35 Initial 6-month engagement Sheffield-based, 3 days per week on-site Key Requirements: Strong hands-on experience with PySpark and Databricks Deep understanding of Spark performance optimisation, including lazy evaluation, column pruning, groupBy operations, and window functions Proven ability to troubleshoot and resolve Spark job performance ...

Sivara GmbH is seeking a Lead AI Technical Architect for a remote 6-month contract in the UK, with high potential for extension. This role emphasizes technical governance and mentoring in AI solutions within the ...

Job Title: Technical Delivery Lead - Data Engineering Lead (Hands-on Python/Databricks) Contract: 6 months - Potentially extension. Location: London Hybrid (5 days in 10 in the office) Rate: Highly competitive Status: Inside IR35 Working ...

Azure platform — including infrastructure-as-code (Pulumi), CI/CD automation, monitoring, security, and Databricks workspace administration. The ideal candidate combines strong Python/PySpark engineering skills with deep Azure platform knowledge and a service-excellence mindset. Duties and Responsibilities Lead solution design activities, collaborating with peers and mentoring … Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation, and scripting. Azure data stack: Synapse, Databricks, ADLS Gen2, Key Vault — including Linked Services, Managed Identity ...

Databricks, APIs, and integration platforms Strong understanding of MLOps and productionizing machine learning Experience leading and scaling engineering teams Tech stack: Azure, Databricks, Python, PySpark, SQL, JavaScript, Node, Bash THE ROLE: You’ll own the technical vision and delivery of our core platforms: Build world‐class platforms Architect cloud … technical leadership where required Define and own the engineering roadmap and align to resource needs Work with: Azure, Databricks, Bash, JavaScript, Node, Python, PySpark, SQL Integration & Ecosystem Enablement Own the strategy and delivery of integration services, including public and private APIs, event‐driven and asynchronous integrations, and authentication/ ...

within Azure cloud environments. Proven hands-on expertise with Azure Databricks in enterprise-scale production environments. Strong programming skills using: Python/SQL/PySpark/Spark SQL. Experience designing and building scalable data pipelines and data processing frameworks. Strong understanding of modern Lakehouse architecture principles. Experience working with … technical teams to understand data requirements and translate them into scalable engineering solutions. Design, develop and maintain high-performance data pipelines using Azure Databricks, PySpark, Spark SQL and Python. Contribute to the design and implementation of Lakehouse architecture using Delta Lake and Databricks best practices. Develop and manage data ...

join a leading organisation on a contract basis. This role will suit someone with strong experience across the Azure data platform , particularly Databricks and PySpark , who can take ownership of data engineering delivery from design through to implementation. This is a hybrid role requiring the candidate … data processing in a cloud environment. Key Responsibilities Design and build scalable data pipelines using Azure technologies Develop and optimise solutions in Databricks using PySpark Work with large and complex datasets, including streaming and batch data Build and maintain data platforms and data warehouses Implement ETL/ELT processes ...

Senior Data Engineer, Lead Data Engineer, Data Engineer, ETL, AWS, AWS Glue, AWS Lambda, AWS S3, AWS Redshift, Amazon EMR, Python, SQL, Pipeline Orchestration, PySpark, data warehousing, data lakehouse, data architecture, Agile, Senior Data Engineer, Lead Data Engineer, Data Engineer) Our client is a global innovator and world leader … with AWS services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR. Proven expertise in Python, SQL, Pipeline Orchestration and PySpark is also expected, as is experience with data lakehouse architectures and data warehousing solutions. Agile experience is also key, as are excellent communication skills ...

contribute to building scalable data pipelines and improving engineering standards across the platform. Responsibilities include: Building and maintaining robust data pipelines using Python and PySpark Supporting CI/CD processes and applying DevOps best practices Contributing to code deployment, testing, and release processes Collaborating with engineers, QA specialists … exposure to data modelling and machine learning workflows over time Your Skills & Experience Hands-on experience with Python and data engineering concepts Exposure to PySpark and large-scale data processing Familiarity with Databricks or similar modern data platforms Understanding of software engineering principles and version control Interest ...

GBMs, GAMs, Random Forests, Neural Networks and clustering techniques Knowledge of statistics and distributions commonly used in insurance Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Proficient at communicating results in a concise manner both verbally and written Behaviours: Motivated by technical excellence Driven to deliver iterative ...

Azure experience to join their team for a contract until the end of the year. Job Responsibilities/Objectives Mandatory Skills: Databricks (PySpark/Spark SQL) , DBT , Azure DevOps (Repos, Pipelines, CI/CD), SQL ,Azure Data Services (Data Lake, Data Factory) Strong experience with Databricks (PySpark/ ...

maintain ETL/ELT pipelines and data models in Fabric (Data Factory, Notebooks, Spark) Write high-performance Spark SQL, T-SQL, Python/PySpark Manage ingestion, transformation, and loading from multiple sources Translate stakeholder requirements into scalable technical solutions Mentor team members and establish engineering standards, security, and governance … assisted development tools like GitHub Copilot, ChatGPT, and Fabric Copilot Essential Experience: Microsoft Fabric is essential Azure Data ecosystem Lakehouse architectures & Data Factory Python, PySpark, Spark SQL Proven hands-on delivery in this stack If you're an experienced Data Engineer with strong Microsoft Fabric and Azure experience, this ...

with Infrastructure as Code (Terraform) Experience working with GitLab and implementing CI/CD pipelines Practical experience building data pipelines in AWS (e.g. EMR, PySpark) Solid understanding of ETL tooling (e.g. Informatica or similar) Strong background in data processing and transformation Proficiency in Python and PySpark Essential Requirements ...

dashboard content Identify and mitigate data quality risks and ensure data integrity Design, build, and maintain reusable data pipelines and datasets using Python and PySpark Develop and test datasets used in BI dashboards and reporting products Work with stakeholders to translate business needs into data solutions and insights Collaborate … DevSecOps Engage in pair programming and adhere to collaborative coding standards Essential Skills & Experience Strong hands-on experience with Python and PySpark Solid background in exploratory data analysis and data modelling Experience with Palantir platforms, ideally including the NHS Federated Data Platform (FDP) Proficiency with BI and data platforms ...

compensation: $170K-$225K base salary + bonus Remote-first environment with occasional customer-site travel Work with cutting-edge cloud technologies including AWS, Snowflake, PySpark, Databricks, Kafka, and Airflow Opportunity to own strategic customer relationships and enterprise architecture initiatives Fast-growing organization with strong career growth opportunities Award-winning … combines deep technical architecture responsibilities with customer-facing consulting, solution design, and strategic account leadership. The ideal candidate will have strong expertise in AWS, PySpark, and Snowflake while also being comfortable operating as a trusted advisor for enterprise clients. What You'll Do: Technical Leadership Architect and design enterprise ...

Technical Skills You'll have experience with some of the following: Cloud & Data Platforms: AWS, Azure, Google Cloud Platform, Databricks Programming Languages: Python, R, PySpark Agentic & GenAI Platforms: Microsoft Copilot Studio, OpenAI GPT-5 Agents, UiPath, Adept AI, Orby AI, Beam AI If you've held any of these … Specialist, Python Developer, AI Transformation Consultant, Databricks Consultant, Azure AI Consultant, AWS Data Scientist, GCP Data Consultant, Agentic AI Consultant, OpenAI GPT Specialist, PySpark Developer. Deerfoot Recruitment Solutions Ltd is a leading independent tech recruitment consultancy in the UK. For every CV sent to clients, we donate ...

Robert Half Technology are assisting UK's market leading legal organisation to recruit a Data Engineer on a 12 month contract basis - Hybrid working - London based. A Data Engineer is required to join a growing ...

development. Key Responsibilities Design and develop end-to-end data pipelines using reusable frameworks Build scalable data solutions using Python, SQL, and PySpark Create modular frameworks separating configuration from orchestration Implement robust automated testing (unit, integration, regression) Establish and enforce data quality controls (schema, nulls, row counts, integrity checks … based workflows Troubleshoot and optimise data pipelines for reliability and performance Required Experience Strong Data Engineering background Advanced Python and SQL skills Experience with PySpark or distributed processing tools Proven track record building reusable data frameworks Strong testing experience (pytest, mock data, data validation techniques) Solid understanding of data ...

digital Screening service. You will build and maintain data pipelines that ingest, transform and prepare data for reporting, developing ETL processes using PySpark and platform tooling. You will work within an Agile multidisciplinary team alongside Data Scientists, Analysts and UCD professionals. This is a largely remote role. Essential skills … experience Strong experience in data pipeline engineering, ETL development and PySpark Experience with Palantir products, including the NHS Federated Data Platform (FDP) Ability to ensure data quality, reliability and performance across reporting pipelines Experience supporting testing and validation of dashboards and underlying datasets Proven ability to work independently ...

insurers. This is a hands-on role focused on building and enhancing a Databricks -based data engineering framework. You will design and develop scalable PySpark pipelines covering data ingestion, validation, transformation, and mapping, while creating reusable libraries and ensuring high engineering standards. You will also provide technical leadership, conducting … code reviews, guiding offshore teams, and ensuring delivery quality aligns with architectural and regulatory requirements. Key Requirements: Strong experience with Databricks and PySpark Solid Azure data platform experience Proven ability to build reusable frameworks and libraries Experience leading code quality and working with offshore teams Desirable: Experience in regulatory ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we're looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

with Microsoft Fabric (Lakehouses, Pipelines, Notebooks) in a live production environment Building, enhancing Data Factory pipelines as part of an ongoing migration into Fabric PySpark for data transformation, engineering within Fabric notebooks SQL queries, transformation logic within Fabric CRM data (Dynamics 365/Dataverse), supporting integration, modelling, optimisation Business … experience Strong SQL skills, including complex queries, transformations Hands-on experience with Microsoft Fabric Experience building, maintaining data pipelines (Data Factory/Fabric Pipelines) PySpark experience for data engineering, transformation Exposure to CRM data, understanding of how CRM supports business processes Ability to work with stakeholders to identify data ...

champion best practice methods; upskill the team through coaching and code reviews. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Communicate pricing proposals and insights clearly and persuasively to stakeholders at all levels. Collaborate with Modelling, Optimisation … insurance pricing. Strong technical expertise in predictive modelling techniques (e.g. GLMs and machine learning models). Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software and optimisation tools. A quantitative degree ...