1 to 25 of 142 PySpark Jobs in London

with Lloyd’s, bordereaux, reinsurance, or regulatory and actuarial reporting. Hands‐on experience designing solutions on Azure Databricks, including Delta Lake, Unity Catalog, Spark (PySpark and Spark SQL) and Databricks SQL. Experience with core Azure data services and their integration with Databricks, such as ADLS Gen2, Azure Data Factory ...

optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow Experience with containerisation ...

Azure Databricks Azure Function Apps& Logic Apps Azure Stream Analytics Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell) Strong PySpark, Delta Lake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software ...

with one or more cloud data ecosystems: Azure (Data Factory, Synapse, Fabric, ADLS, Databricks) AWS (Glue, Redshift, EMR) GCP (BigQuery, Dataflow) Databricks (Spark/PySpark) Snowflake (SQL, Snowpark, performance tuning) Strong SQL and/or Python capability, and familiarity with tools such as dbt, Git, CI/ ...

models in data warehouses, lakehouse environments or analytics tools such as Power BI. Familiarity with modern data engineering frameworks and tools (e.g. Spark/PySpark, Pandas, orchestration tools). Experience working with cloud data platforms on AWS or Azure (e.g. Databricks, Redshift, Microsoft Fabric). Understanding of software engineering ...

Lake Storage, Azure Data Factory, , MS Fabric ETL/ELT processes and designing, building and testing data pipelines Building data transformations using python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

data warehouses or similar analytical platforms. Experience developing reliable ETL or ELT processes in the Medallion architecture. Experience with Microsoft Fabric data engineering, including Pyspark and dbt. Practical knowledge of data quality, testing, monitoring and operational support. Strong problem-solving skills and attention to detail. Clear communication ...

Good understanding of modern software platform architecture and the role of the data platform within the broader platform ecosystem Good Scala skills (or strong PySpark plus a genuine willingness to work primarily in Scala) Strong SQL skills for transformations, data modelling, and debugging data issues Experience building and supporting ...

enterprise-scale environments. Deep expertise in Azure Databricks , Azure Data Factory , Azure Synapse , and Azure Data Lake . Hands-on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. ...

data-science role, including delivery of reliable pipelines, reusable data assets and useful analytical models in Microsoft Fabric and Databricks. Advanced Python, SQL and PySpark, including experience with data transformation, analysis and common machine-learning libraries. Hands-on experience designing and operating scalable data pipelines, lakehouse data models, Delta ...

deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good ...

Data Lake, Lakehouse, Data Warehouse, Data Vault ETL/ELT processes and designing, building and testing data pipelines Building data transformations using Python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

management. Experience integrating education or CRM systems such as an LMS, HubSpot or similar platforms. Knowledge of Power BI, including semantic models. Python or PySpark for data transformation. Microsoft Fabric or Boomi certification. Why join Corndel Work flexibly and stay connected – most of our people work remotely, supported ...

client environments Advanced SQL and strong ETL/ELT pipeline experience Hands-on Azure, Databricks, Microsoft Fabric or comparable cloud platform experience Python or PySpark, plus Git, CI/CD, testing, monitoring and infrastructure as code Strong data modelling and client-facing communication skills Practical experience building AI-powered ...

Engineering across GCP, Azure and/or AWS . Hands-on experience with BigQuery, Databricks, Snowflake or Microsoft Fabric . Strong Python, SQL and PySpark development skills. Experience building scalable ETL/ELT pipelines and data integration solutions. Experience with DBT, Airflow, Azure Data Factory or Dataflow . Knowledge ...

Python, plus hands-on Git Production experience with at least one of Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow or PySpark Cloud-native orchestration and distributed data processing Infrastructure as code (e.g. Terraform) CI/CD and DevOps or DevSecOps practice A sound grasp of data architecture ...

enterprise data source integration — MS SQL Server (on-premises and Azure SQL) and Oracle Database ingestion into Azure landing zones Strong Apache Spark skills — PySpark or Scala, optimisation, partitioning, and job scheduling in Databricks Experience assessing and remediating partially built or vendor-abandoned data platform implementations Proven ability ...

vs. centralized models, capacity (F-SKU) sizing and licensingDesign end-to-end Microsoft Fabric solutions: OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL), Real-Time Intelligence (Eventstreams, Eventhouse/KQL) and Power BI + Power BI CoPilotDesign governance, security and operating frameworks: data governance ...

Strong problem-solving and analytical skills. Desirable Experience with Microsoft Fabric. Experience with Azure Data Services. Working with multi-currency financial information. Knowledge of PySpark or Spark SQL. Understanding of Microsoft Entra ID and security frameworks. Experience with DevOps and automated deployment pipelines. Knowledge of financial, legal, or professional ...

data platform on Microsoft Fabric, Databricks, Snowflake or similar, using a medallion architecture, including managing ingestion performance and forecasting platform costs. Strong Python/PySpark and SQL, with a track record of production ETL/ELT pipelines integrating data from varied business systems, including less well-documented sources. Experience ...

Strong problem-solving and analytical skills. Desirable Experience with Microsoft Fabric. Experience with Azure Data Services. Working with multi-currency financial information Knowledge of PySpark or Spark SQL. Understanding of Microsoft Entra ID and security frameworks. Experience with DevOps and automated deployment pipelines. Knowledge of financial, legal, or professional ...

/CD deployment practices. Strong problem-solving and analytical skills.DesirableExperience with Microsoft Fabric. Experience with Azure Data Services.Working with multi-currency financial informationKnowledge of PySpark or Spark SQL.Understanding of Microsoft Entra ID and security frameworks.Experience with DevOps and automated deployment pipelines.Knowledge of financial, legal, or professional services data environments.Elite ...

diversity of views and opinions. Key Skills & Experience: Proven experience in a Data Engineering role within the insurance industry Advanced SQL and PySpark skills, with hands-on experience building data transformations and data products Demonstrable data modelling experience, including practical experience with medallion architecture, and comfortable contributing technical input ...

orchestrate the ingestion of data from disparate source systems into a centralised data analytics platform. Designing and implementingdata engineering solutions using T-SQL, Python, PySpark and DBT in the Azurecloud environment. Working with Data Analystsin promoting business logic into DBT data models, to support business reportsand dashboards. Maintaining ...

platform such as Databricks, Snowflake or Fabric, and orchestration tooling, with practical experience of ETL/ELT, data testing, Git and CI/CD; PySpark or dbt experience is beneficial. Working knowledge of finance data structures, including chart of accounts, entity hierarchy, sub-ledger to ledger relationships, intercompany ...