76 to 100 of 109 PySpark Jobs

from scratch-this is the one. What You'll Be Doing Lakehouse Engineering (Azure + Databricks) Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines , PySpark , and Spark SQL across a full Medallion Architecture (Bronze Silver Gold) . Implement ingestion patterns for files, APIs, SaaS platforms (e.g. subscription billing … Work With Databricks : Lakeflow Declarative Pipelines, Workflows, Unity Catalog, SQL Warehouses Azure : ADLS Gen2, Data Factory, Key Vault, vNets & Private Endpoints Languages : PySpark, Spark SQL, Python, Git DevOps : Azure DevOps Repos, Pipelines, CI/CD Analytics : Power BI, Fabric What We're Looking For Experience 5-8+ years ...

Design, build and maintain scalable and secure data pipelines on the Azure platform. Develop and deploy data ingestion processes using Azure Data Factory, Databricks (PySpark), and Azure Synapse Analytics. Optimise ETL/ELT processes to improve performance, reliability and efficiency. Integrate multiple data sources including Azure Data Lake (Gen2 … Required Skills & Experience - Azure Data Engineer: Proven commercial experience as a Data Engineer delivering enterprise-scale solutions in Azure Azure Data Factory Azure Databricks (PySpark) SQL & Python Understanding of CI/CD in a data environment, ideally with tools like Azure DevOps. Experience working within consultancy or client-facing ...

with ambiguity and will want to dive deep and fix problems at scale. The environment looks a bit like this 👇 👉 Python (pandas, numpy, scipy, PySpark) 👉 SQL 👉 AWS or GCP (BigQuery/RedShift/Snowflake) 👉 Spark/Databricks 👉 Airflow/Dagster ...

independently and collaborate within cross-functional teams Some knowledge of programming languages such as Python is preferred Experience with data processing technologies such as PySpark is preferred Familiarity with AWS* services and cloud-based data workflows is preferred ...

Looker and enabling reporting for end-users within general guardrails Your Profile Essential Skills/Knowledge/Experience Python GCP Services GCS, GKE, BigQuery PySpark, DBT IAM Role & Groups CI/CD Tools Jenkins, Harness or equivalent Design & Architecture Data product design with in guard rails and estimating FinOps ...

able to develop the prototypes of the models, and do some feature engineering. The environment looks a bit like this 👇 👉 Python (pandas, numpy, scipy, PySpark) 👉 SQL 👉 AWS or GCP (BigQuery/RedShift/Snowflake ...

Defining and enforcing engineering standards across code quality, testing, CI/CD, IaC, and observability Building and optimising ELT/ETL pipelines using Glue (PySpark) , Lambda, Step Functions, and event-driven patterns Championing security-by-design in collaboration with security and assurance teams (IAM, KMS, encryption, auditability) Implementing … across data analytics and data engineering domains Strong hands-on experience with: Amazon S3 (data lake patterns, partitioning, lifecycle management) AWS Glue (Jobs, Crawlers, PySpark, orchestration) Amazon Redshift (data modelling, performance tuning, WLM, Spectrum) Advanced Python (PySpark) and SQL skills Experience implementing CI/CD pipelines and Infrastructure ...

expertise in data processing within a cloud-based data platform. Junior Data Engineer duties: Develop, test, and maintain scalable data pipelines using Python and PySpark within an Azure Databricks development environment Clean, aggregate, and transform complex datasets to meet business requirements. Assist with data quality checks and support data … Junior Data Engineer, you will have: Strong proficiency in Python, particularly for data manipulation and automation scripts; experience in libraries such as pandas, PySpark, numpy, pyodbc or associated would be beneficial Hands-on experience (or academic project experience) using PySpark. An understanding of ETL/ELT processes and/ ...

junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). The Role As a Tech Lead , you will play a critical role in designing, building, and maintaining data pipelines and infrastructure that enable … junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data ...

junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data … inclusive team environment that encourages continuous learning and improvement. Your Profile Essential skills/knowledge/experience: Knowledge of machine learning frameworks (e.g., PySpark, PyTorch) and model deployment tools (e.g., MLflow, TensorFlow Serving). Strong experience with data processing frameworks (e.g., Apache Spark, Flink). Expertise ...

Job Responsibilities: Payments and File Format Expertise Deep understanding of payment processing workflows, especially H2H and SWIFT. Hands-on experience with legacy formats (IDOC, EDIFACT) as well as ISO 20022 and GFF. Ability to interpret ...

treating data pipelines as production-grade software. A significant portion of the work involves designing and maintaining complex, test-driven Python data flows, with PySpark used as the execution engine rather than the primary focus. Strong Python fundamentals, testing discipline, and code quality are critical to success in this … strong emphasis on clean architecture, reusable Python modules, and testability Writing comprehensive unit tests and BDD tests (Behave), including mocking and patching Using PySpark to process large-scale datasets while keeping business logic in Python Creating, maintaining, and optimising Delta Lake tables for performance and reliability Building and running ...

support analytics and downstream use cases Developing scalable data solutions using Python , with a focus on maintainability and performance Working with Apache Spark/PySpark to process and transform large datasets Supporting the ingestion, transformation and validation of complex financial data Improving the performance, reliability and resilience of existing … Succeed Essential Commercial experience as a Data Engineer at a mid‐level Strong Python development skills Hands‐on experience with Apache Spark/PySpark Solid experience building ETL/ELT pipelines Background within the financial services industry (investment management experience desirable) Comfortable working with production systems in a regulated ...

innovative data platform. You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

working with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

Services : S3, Redshift, Glue, Athena, Lake Formation, IAM Data Engineering/ETL: Building and optimizing ETL pipelines Data ingestion, transformation & orchestration using AWS Glue (PySpark/Python) Working with structured/semi-structured telecom datasets (CDRs, network logs, subscriber data) Data Lake Technologies: Expertise in Apache Iceberg table format … Candidate Profile 3 10+ years of experience in AWS Data Engineering/Support Engineering. Strong telecom domain understanding. Hands-on with Iceberg, Athena, Glue (PySpark), Python, Redshift, S3, ETL frameworks. Strong troubleshooting mindset and ability to work in 24 7 or rotational support environments (if required). ...

processing, and analytics • Designing scalable, secure, and high performance architectures using Databricks and distributed data systems • Building and overseeing robust data ingestion pipelines using PySpark, with a strong focus on reliability and accuracy • Ensuring end to end data quality from raw ingestion through to curated datasets used for reporting … wider regulatory requirements • Evaluating and introducing new technologies where they add clear value to the data ecosystem What you will bring • Deep expertise in PySpark and distributed data processing at scale • Strong hands on experience designing, building, and optimising Databricks based platforms • Advanced SQL skills including performance optimisation ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

doing Design, build and deploy algorithms/data models supporting pricing, forecasting and optimisation use cases in production Develop and optimise distributed Spark/PySpark batch pipelines for large-scale data processing Write production-grade Python workflows implementing complex, explainable business logic Work with Databricks for job execution, orchestration … delivering production algorithms/data models (forecasting, pricing, optimisation or similar) Strong Python proficiency and modern data stack exposure (SQL, Pandas/NumPy + PySpark; Dask/Polars/DuckDB a bonus) build, schedule and optimise Spark/PySpark pipelines in Databricks (Jobs/workflows, performance tuning, production ...

tagging, monitoring and platform operations. • Architect and design end‐to‐end data platform solutions focused on scalability, security and reliability. • Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. • Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. • Collaborate closely with data … followed. Your Skills and Experience You will bring: • Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. • Proficiency in PySpark, Python, SQL and Spark SQL. • Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. • Experience with monitoring, logging and alerting solutions. ...

tagging, monitoring and platform operations. • Architect and design end‐to‐end data platform solutions focused on scalability, security and reliability. • Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. • Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. • Collaborate closely with data … followed. Your Skills and Experience You will bring: • Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. • Proficiency in PySpark, Python, SQL and Spark SQL. • Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. • Experience with monitoring, logging and alerting solutions. ...

tagging, monitoring and platform operations. * Architect and design end-to-end data platform solutions focused on scalability, security and reliability. * Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. * Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. * Collaborate closely with data … followed. Your Skills and Experience You will bring: * Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. * Proficiency in PySpark, Python, SQL and Spark SQL. * Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. * Experience with monitoring, logging and alerting solutions. ...

tagging, monitoring and platform operations. * Architect and design end-to-end data platform solutions focused on scalability, security and reliability. * Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. * Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. * Collaborate closely with data … followed. Your Skills and Experience You will bring: * Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. * Proficiency in PySpark, Python, SQL and Spark SQL. * Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. * Experience with monitoring, logging and alerting solutions. ...

Pyspark Engineer- (Data Engineering, AWS GLUE) SC Cleared OR Eligible Stevenage (Hybrid) 2-3 days onsite Up to £80,000 High-impact programme - Revolutionary platform I am looking for a Pyspark expert to take the reins on a range of highly ambitious Data Migration projects supporting a range … regulatory standards. Relevant exposure; Strong hands-on experience with ETL processes and tools (Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi) or scripting using Python, PySpark, and SQL. Proficient-level SQL skills for complex query development, performance tuning, indexing, and data transformation across on-premise databases and AWS cloud environments. Solid ...

Our client is an industry leading market-research company who use data-driven insights to provide unique and actionable data for some of the world's most recognised brands. Over the last 12 months they ...