1 to 25 of 77 PySpark Jobs

Engineer Hands-on experience with Microsoft Fabric (essential) Experience working within Azure data environments Strong SQL development skills Experience with Python and/or PySpark Knowledge of Azure Synapse Analytics and Azure Data Factory Experience building and maintaining data pipelines and data integration solutions Understanding of modern data warehousing ...

Engineer Hands-on experience with Microsoft Fabric (essential) Experience working within Azure data environments Strong SQL development skills Experience with Python and/or PySpark Knowledge of Azure Synapse Analytics and Azure Data Factory Experience building and maintaining data pipelines and data integration solutions Understanding of modern data warehousing ...

ideally in enterprise-scale environments. Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse and Azure Data Lake. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. ...

scale data platforms,lakehousesand data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured, semi-structured and unstructured data ...

pipeline failures, data inconsistencies, and performance issues. 2. Databricks & Data Processing Develop and maintain Databricks notebooks, jobs, and data transformation workflows. Use Python and PySpark to process, cleanse, validate, and transform large datasets. Implement reusable data processing logic and efficient ETL/ELT workflows. Perform data quality checks, error ...

assisted software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

Fabric, or Data Engineering are a plus. Technical Skills Required Preferred Required Preferred Power BI Microsoft Fabric SQL Azure Synapse Python Spark/PySpark Data Modeling Dataflows Gen2 DAX OneLake ETL/ELT Concepts Azure Data Factory Dashboard Development Azure Event Hub REST APIs AI/ML Tooling ...

such as Claude Code or equivalent tools, to improve engineering productivity while maintaining high standards of quality, governance and security. Desirable experience Regulated environment. PySpark for large-scale data transformations. Migrating legacy data estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven systems. Data-quality ...

Experience of core MS Fabric features including: OneLake, Lakehouses, Warehouses, Data Pipelines, Dataflows, Gen2, Notebooks, Eventstreams + Eventhouses. Using SQL, T-SQL, Spark SQL, PySpark + Python for data ingestion, data transformation, data optimisation, data modelling, and converting structured/unstructured source data into validatd Delta tables. Using Power ...

accelerate development across code, infrastructure, testing and documentation while maintaining appropriate quality and security controls. Desirable experience Financial services or another regulated environment. PySpark for large-scale batch or streaming transformations. Migrating SQL Server and SSIS estates to lakehouse or metadata-driven architectures. Customer Data Platforms or event-driven ...

Strong experience with BigQuery (SQL, partitioning, clustering, optimization) and Dataflow (Apache Beam) Strong programming skills in Python with experience in data manipulation libraries (PySpark, pandas) Expert-level SQL proficiency for complex transformations, optimization, and analysis Proficiency with Dataform for modular SQL-based data transformations and data pipeline management Solid ...

Strong experience with BigQuery (SQL, partitioning, clustering, optimization) and Dataflow (Apache Beam) Strong programming skills in Python with experience in data manipulation libraries (PySpark, pandas) Expert-level SQL proficiency for complex transformations, optimization, and analysis Proficiency with Dataform for modular SQL-based data transformations and data pipeline management Solid ...

Strong experience with BigQuery (SQL, partitioning, clustering, optimization) and Dataflow (Apache Beam) Strong programming skills in Python with experience in data manipulation libraries (PySpark, pandas) Expert-level SQL proficiency for complex transformations, optimization, and analysis Proficiency with Dataform for modular SQL-based data transformations and data pipeline management Solid ...

challenge Technical lead or lead engineer experience on enterprise scale Databricks platforms on Azure Strong hands on data engineering background with Python or PySpark and advanced SQL Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows and Auto Loader Declarative pipelines (Delta Live Tables or Lakeflow) and Structured Streaming ...

architecture Experience with cloud data platforms (AWS preferred - Redshift, Athena, S3, Glue. Azure accepted) Understanding of workflow orchestration and automation tools Python experience (including PySpark) is beneficial but not essential Exposure to BI platforms such as Power BI, Tableau or QuickSight, with an understanding of how backend data models ...

Knowledge of Azure Data Lake, Azure Data Factory, or other Azure data services. Experience with Power Platform, especially Power Apps. Exposure to Python or PySpark for data analysis is a plus. Experience with data governance, GDPR, documentation standards, and move-to-run processes. Experience delivering BI solutions from initial ...

patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data ...

/C# Solid grasp of database design for both transactional (OLTP) and analytical (OLAP) systems Desirable Proficiency with Azure Databricks/PySpark Previous experience building and deploying ML or AI solutions in Azure Previous technical experience with Power BI or Microsoft Fabric If this role sounds of interest ...

Data modelling and data warehousing Extract-Transform-Load (ETL) Creating data pipelines Ethics in data and AI Cloud and big data technologies (AWS, Snowflake, PySpark) Working with unstructured data Working with Generative AI – Large language models (LLMs) and image generation (ChatGPT, Gemini) Consulting Skills Effective teamwork Business acumen Time ...

stakeholders to translate business challenges into analytical solutions. YOUR SKILLS AND EXPERIENCE: You will bring strong capability in: Python and SQL skills, with strong PySpark experience Hands-on expertise with Microsoft Fabric and Databricks. Experience building production-grade data pipelines, reusable data assets and machine learning solutions. Experience communicating ...

development of scalable ETL/ELT pipelines. Proven experience working with SQL, relational databases and large datasets. Experience with data processing technologies such as PySpark, Spark or Pandas. Strong understanding of data quality, transformation and integration. Experience with Git, CI/CD and automated testing. Comfortable working in Linux ...

payment, and transaction analysis. Strong Python skills for statistical modelling, cohort analysis, weighting, matching, and reproducible analytics. Hands-on experience with Databricks, Spark/PySpark, Delta tables, and automated data workflows. Proficiency with Git, Azure DevOps, and controlled production release processes. Strong data quality, reconciliation, and audit-trail practices. ...

looking for: Previous experience as a Technical Lead (hands-on) Deep, hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data ...

Data Engineer Key skills: * Microsoft Fabric * Azure Cloud Technologies * Python * SQL Server/SQL * Data Warehousing & Data Modelling * Data Architecture * Data Pipelines & Integration Solutions * PySpark * ETL/ELT Development * Data Governance & Security * Source Control (Git) * Technical Leadership & Team Management You will be a Lead Data Engineer Developer with good ...

across systems Proven skills in project management, solution delivery, and budget oversight within an IT/Cloud environment Technical proficiency with SQL, Python/PySpark, and Microsoft Power Platform Experience with CI/CD, DevOps practices, and system security considerations Ability to analyse, troubleshoot, and validate data solutions, ensuring ...