1 to 25 of 344 PySpark Jobs

with Lloyd’s, bordereaux, reinsurance, or regulatory and actuarial reporting. Hands‐on experience designing solutions on Azure Databricks, including Delta Lake, Unity Catalog, Spark (PySpark and Spark SQL) and Databricks SQL. Experience with core Azure data services and their integration with Databricks, such as ADLS Gen2, Azure Data Factory ...

optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow Experience with containerisation ...

Azure Databricks Azure Function Apps& Logic Apps Azure Stream Analytics Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell) Strong PySpark, Delta Lake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software ...

with one or more cloud data ecosystems: Azure (Data Factory, Synapse, Fabric, ADLS, Databricks) AWS (Glue, Redshift, EMR) GCP (BigQuery, Dataflow) Databricks (Spark/PySpark) Snowflake (SQL, Snowpark, performance tuning) Strong SQL and/or Python capability, and familiarity with tools such as dbt, Git, CI/ ...

with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark or Spark SQL to manage and process large datasets. Expertise in Building ETL and ELT Pipelines: You are skilled in building robust ...

models in data warehouses, lakehouse environments or analytics tools such as Power BI. Familiarity with modern data engineering frameworks and tools (e.g. Spark/PySpark, Pandas, orchestration tools). Experience working with cloud data platforms on AWS or Azure (e.g. Databricks, Redshift, Microsoft Fabric). Understanding of software engineering ...

Engineer Hands-on experience with Microsoft Fabric (essential) Experience working within Azure data environments Strong SQL development skills Experience with Python and/or PySpark Knowledge of Azure Synapse Analytics and Azure Data Factory Experience building and maintaining data pipelines and data integration solutions Understanding of modern data warehousing ...

Engineer Hands-on experience with Microsoft Fabric (essential) Experience working within Azure data environments Strong SQL development skills Experience with Python and/or PySpark Knowledge of Azure Synapse Analytics and Azure Data Factory Experience building and maintaining data pipelines and data integration solutions Understanding of modern data warehousing ...

Lake Storage, Azure Data Factory, , MS Fabric ETL/ELT processes and designing, building and testing data pipelines Building data transformations using python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

Lake Storage, Azure Data Factory, , MS Fabric ETL/ELT processes and designing, building and testing data pipelines Building data transformations using python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

data warehouses or similar analytical platforms. Experience developing reliable ETL or ELT processes in the Medallion architecture. Experience with Microsoft Fabric data engineering, including Pyspark and dbt. Practical knowledge of data quality, testing, monitoring and operational support. Strong problem-solving skills and attention to detail. Clear communication ...

deep Azure Synapse/Databricks experience with clear ability to specialise quickly in Fabric. Expert SQL/T-SQL plus strong Python or PySpark, with a track record of building maintainable ELT/ETL pipelines and analytical data models. Strong Power BI and DAX experience, including semantic modelling, incremental ...

Good understanding of modern software platform architecture and the role of the data platform within the broader platform ecosystem Good Scala skills (or strong PySpark plus a genuine willingness to work primarily in Scala) Strong SQL skills for transformations, data modelling, and debugging data issues Experience building and supporting ...

enterprise-scale environments. Deep expertise in Azure Databricks , Azure Data Factory , Azure Synapse , and Azure Data Lake . Hands-on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. ...

REST, etc.)2 + years Hands-on experience with Redis-backed state management and event-driven processing.2 + years of experience in Apache Spark (PySpark/Spark SQL)2+ years building and deploying Cloud based solutions using - Azure Databricks with UC, Snowflake, Functions, Service Bus2+ years of experience ...

direct dependence on operational databases or replicas. Fabric Data Factory and pipelines with stronger orchestration, monitoring and automated deployment. Python, notebooks and Spark/PySpark where they are the right tool for modern analytical workloads. Direct Lake and other Fabric-native patterns where they provide a genuine benefit. Better ...

with Microsoft Fabric technologies including OneLake, Lakehouse, Warehouse, Data Factory, Spark and Power BI. Strong Engineering Skills - You're proficient in SQL, Python and PySpark, with experience developing scalable and maintainable data solutions. Modern Data Platform Experience - You’ve successfully built, enhanced or operated cloud‐based data platforms, ideally ...

deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good ...

ideally in enterprise-scale environments. Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse and Azure Data Lake. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. ...

assisted software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

data-science role, including delivery of reliable pipelines, reusable data assets and useful analytical models in Microsoft Fabric and Databricks. Advanced Python, SQL and PySpark, including experience with data transformation, analysis and common machine-learning libraries. Hands-on experience designing and operating scalable data pipelines, lakehouse data models, Delta ...

deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good ...

Data Lake, Lakehouse, Data Warehouse, Data Vault ETL/ELT processes and designing, building and testing data pipelines Building data transformations using Python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

management. Experience integrating education or CRM systems such as an LMS, HubSpot or similar platforms. Knowledge of Power BI, including semantic models. Python or PySpark for data transformation. Microsoft Fabric or Boomi certification. Why join Corndel Work flexibly and stay connected – most of our people work remotely, supported ...

client environments Advanced SQL and strong ETL/ELT pipeline experience Hands-on Azure, Databricks, Microsoft Fabric or comparable cloud platform experience Python or PySpark, plus Git, CI/CD, testing, monitoring and infrastructure as code Strong data modelling and client-facing communication skills Practical experience building AI-powered ...