1 to 25 of 116 PySpark Jobs

governance by design Data warehousing and ETL best practices CI/CD and version control using GitHub Additional Skills: (Nice to Have) Experience with PySpark Experience with Matillion or other modern ETL tools Working knowledge of core financial data concepts (general ledger, chart of accounts, financial reporting) Experience deploying ...

Good understanding of ETL/ELT, data modelling, and data transformation. Experience with CI/CD and modern engineering practices. Strong Python, SQL, or PySpark skills. Experience working in Defence, Public Sector, or other secure environments . Must hold active SC clearance . Desirable experience Experience with streaming ...

tooling Key skills Strong experience with Microsoft Fabric (Data Factory, Lakehouse, OneLake, Pipelines, SQL Warehouse) Solid data engineering background using SQL, Python, PySpark, Delta Lake E xperience building and optimising end-to-end data pipelines CI/CD experience for analytics and data platforms (Azure DevOps or GitHub) Exposure ...

facing or consulting environment (or strong desire to develop this) Ambition to move into team leadership/people management Nice to have: Experience with PySpark/Databricks Experience with streaming technologies like Kafka or AWS Kinesis Knowledge of data governance and quality frameworks ...

pipelines and working with lakehouse architectures Knowledge of Azure tools such as Data Factory and/or Fabric Data Pipelines Experience with Python or PySpark for data transformation Understanding of data modelling (dimensional modelling, medallion architecture) Strong communication skills and ability to work with non-technical stakeholders Desirable: Experience ...

pipelines and working with lakehouse architectures Knowledge of tools such as Azure Data Factory and/or Fabric Data Pipelines Experience with Python or PySpark for data transformation Understanding of data modelling (dimensional modelling, medallion architecture) Strong communication skills and ability to work with non-technical stakeholders Desirable: Experience ...

cloud, S3, cross-cloud connectivity, API Gateways GCP (BigQuery, GCS, Dataflow) Deep expertise in: Databricks (Lakehouse architecture, Delta Lake, Unity Catalog, Delta Sharing, Pyspark, Spark SQL etc.) Snowflake (data warehousing & data sharing, Iceberg, ) Strong understanding of: Data architecture patterns, industry benchmarking, Data integration patterns (ETL/ELT, streaming ...

backgrounds. The technology stack is flexible, but experience in some of the following are nice to have: Data Engineering & Programming: • Python • SQL • Spark/PySpark • Scala • Data modelling and warehousing concepts Cloud & Platforms: • AWS (S3, Redshift, Glue, EMR, Lambda) • Other cloud platforms (Azure/GCP considered) • Data lake ...

modern UI patterns, develop framework is essential Back-end: Expertise in C# .NET Core, REST APIs, and microservices-based development is essential Python (pyspark): Experience with data pipelines and analytics is a plus. Power BI: Familiarity with developing dashboards and data models. Architecture: Knowledge of microservices, API design ...

Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud Storage ). Exposure ...

Technical depth 10+ years in data engineering, data warehousing, or a senior data infrastructure role Expert‐level SQL and data manipulation skills using Spark, PySpark, or equivalent big data frameworks, with proven experience processing high‐volume, high‐velocity datasets Deep expertise in data lake and data warehousing technologies ...

experience in architecting and designing robust, secure and compelling data solutions using Microsoft Fabric, Purview and other Microsoft ecosystem toolsets. Programming skills, including Python, PySpark and T-SQL. Strong track record in business development, proposition shaping, and winning work. Proven experience leading and developing high-performing multi-disciplinary technology ...

Advanced SQL skills for complex query validation. Experience with Agile/DevOps delivery models and JIRA test management. Familiarity with automation frameworks (SQL/PySpark preferred). Knowledge of CI/CD and release management processes. Experience with Power BI, Alteryx, and VS Code , particularly in testing integrations with ...

Azure ML. Key skills and responsibilities, Demonstrated hands-on experience with Microsoft Fabric and Power BI. Extensive knowledge in data modelling, DAX, Python/PySpark, SQL/KQL. Proven record of technical leadership and effective stakeholder engagement. Design and deliver comprehensive Fabric solutions, including OneLake, Lakehouse, Warehouses, and Power ...

Pega, Adobe, and similar tools) using APIs, connectors, or data exports. Build and maintain scalable ETL/ELT pipelines using SQL, Python, and PySpark for batch and near real-time processing. Work with AWS services such as DMS, Glue, Lambda, Step Functions, S3, Redshift, Athena, RDS, EC2, and CloudWatch. ...

with CDPs that are SQL-native. Treasure Data is a significant advantage; exposure to mParticle, Tealium, Segment or Salesforce CDP will translate. Production-grade PySpark/Databricks experience — building scalable data pipelines, optimising performance, and operating Delta Lake/Lakehouse architectures in a real production environment. Solid DevOps craft ...

Lloyd's of London or London Market insurance sector Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling. Strong Python, PySpark and Spark SQL capability Solid understanding of data governance Proven track record of ownership, delivery and operating effectively in fast-paced environments. What ...

Databricks Defining data models and Databricks architecture standards Establishing data governance and security frameworks Aligning data architecture with enterprise cloud strategy Supporting development using PySpark, SQL and Python within Databricks Working within CI/CD and DevOps environments (Git-based) About You: Strong technical background in Databricks and Azure ...

Databricks Defining data models and Databricks architecture standards Establishing data governance and security frameworks Aligning data architecture with enterprise cloud strategy Supporting development using PySpark, SQL and Python within Databricks Working within CI/CD and DevOps environments (Git-based) About You: Strong technical background in Databricks and Azure ...

exposure to enterprise-level data platforms and best practices. The role will involve developing and maintaining data pipelines, transforming data using Python, SQL and PySpark , integrating data from various internal and external sources, and supporting day-to-day data operations. You’ll also contribute to data quality initiatives, documentation ...

cost optimisation Delivery Governance: programme planning, roadmap ownership, risk & dependency management, stakeholder alignment, executive reporting Data & Analytics: Data lakes/warehouses, ETL/ELT, PySpark, SQL, BI integration, data quality and lineage ...

Hands-on experience with Microsoft Purview • Strong Azure Data Platform experience: ADLS Gen2, Azure Databricks, Data Factory • Hands-on data engineering experience (Spark/PySpark, Delta Lake) • Knowledge of security and compliance controls, GDPR, PII/DLP, RBAC/ABAC • Experience leading Agile teams and managing risks, dependencies ...

workflows with DBT (core and Cloud), ensuring modular, testable, and reusable transformations. Build high-performance data pipelines in Snowflake and Databricks (PySpark, Delta Lake, Unity Catalog) for batch and streaming workloads. Orchestration using Apache Airflow and Amazon tools. Engineer scalable data ingestion pipelines into AWS (DBT, SQL, PythonS3, Kinesis … experience IT with some in Biotech/Pharma industry Expert-level proficiency in: DBT: advanced macros, Jinja, testing, exposures, dbt Cloud deployment. Databricks: Spark (PySpark, SQL), Delta Lake, Unity Catalog. Snowflake AWS: S3, Glue, Lambda, Step Functions, Datasync, EMR, Redshift, IAM and networking/security fundamentals. Data Visualization: Power ...

start up, but with resources and business opportunities that start ups can rarely match. What you’ll need Strong SQL, Python, and Spark/PySpark skills Experience in data pipeline design A quantitative degree - Maths, Stats, Physics, Operations research About the company You'll be working ...

/Ansible Work closely with engineering teams to deliver real-world ML systems across multiple sectors. Tech Stack You’ll Work With Python, SQL, PySpark PyTorch, TensorFlow, Scikit‐learn MLflow (core focus), Databricks, Azure IaC: Terraform, ARM, Ansible Git, Azure DevOps, Jenkins What We’re Looking For ✔️ 3+ years ...