51 to 75 of 81 Permanent PySpark Jobs

. Maintain/implement data warehousing solutions and manage large-scale data storage systems (e.g. Microsoft Fabric) Build and optimise SQL queries, stored procedures, PySpark notebooks and database objects to ensure data performance and reliability. Migrate and modernise legacy databases to cloud-based architectures. Database Administration Administer, monitor … normalisation, indexing, query optimisation). Strong experience with ETL/ELT tools. E.g. Azure Data Factory, Databricks, Synapse Pipelines, SSIS, etc. Experience with Python, PySpark, or Scala for data processing. Familiarity with CI/CD practices. Experience with Data lake, Data warehouse and Medallion architectures. Understanding of API integrations ...

extend AI-enabled workflows as part of the core product (e.g. decision support, automation, insight generation) Develop and maintain data pipelines using Python, PySpark, SQL , and distributed systems Make pragmatic engineering decisions that balance short-term delivery with long-term product quality Contribute to technical and architectural discussions ...

World’s Most Ethical Companies for the 14th time. DUTIES (included, but not limited to): Monitor and maintain Microsoft Fabric data pipelines (Dataflows, PySpark Notebooks etc.) Develop and support self-service data transformation functionality using Powerquery (M Code) Work with front-end reporting developers to provide them with cleansed ...

Operations, DevOps, or Site Reliability Engineering managing complex systems ✅ 2+ years hands-on Cloud platform infrastructure experience in Databricks environments (Azure/AWS) ✅ Strong PySpark, Python, and Linux scripting skills in production settings ✅ Proficiency with Terraform and Ansible for provisioning and configuration management ✅ Solid experience with *nix systems (Ubuntu ...

background in data analysis, data processing, data modeling, with excellent mathematical and problem-solving skills. A background in commercial optimisation analytics. Proficiency in Python, PySpark and SQL for data extraction, transformation, and analysis. Expertise in data-driven storage solutions and handling large, complex datasets. Good understanding of classical statistics ...

analysis, model development, and data interrogation. Strong knowledge of PyTorch or similar ML frameworks. Experience handling large datasets and distributed processing with PySpark or similar is desirable . Experience of experimentation tools such as MLflow is a bonus. Comfort with deploying ML models into production environments. Familiarity with ...

Data Engineer Data Pipelines | Python | PySpark | AWS | AWS Glue | London Location: London (hybrid - 2 days onsite per week) Salary: Up to £80K + Benefits We’re working with a tech-driven organisation looking to hire an experienced Data Engineer to help design and scale cloud-based data platforms that ...

just which tools you used Strong SQL/SSIS/Kingswaysoft and starting to use AI and automation for ETL. Python would be useful (PySpark/Pandas/NumPy) Desirable Experience migrating data into Dynamics (or similar ERP platforms) Experience working with messy, real-world data Why Join? High ...

engineering is done across the organisation. Skills & Experience Strong hands-on experience with Azure data platforms Advanced SQL skills Commercial experience using Databricks and PySpark Proven background building and maintaining scalable data pipelines If you're looking for a role where you can combine technical depth, ownership, and influence ...

data for critical client projects. How to develop and enhance your knowledge of agile ways of working and working in open source stack (PySpark/PySql). Quality engineering professionals utilise Accenture delivery assets to plan and implement quality initiatives to ensure solution quality throughout delivery. As a Data ...

data for critical client projects. How to develop and enhance your knowledge of agile ways of working and working in open source stack (PySpark/PySql). Quality engineering professionals utilise Accenture delivery assets to plan and implement quality initiatives to ensure solution quality throughout delivery. As a Data ...

from scratch-this is the one. What You'll Be Doing Lakehouse Engineering (Azure + Databricks) Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines , PySpark , and Spark SQL across a full Medallion Architecture (Bronze Silver Gold) . Implement ingestion patterns for files, APIs, SaaS platforms (e.g. subscription billing … Work With Databricks : Lakeflow Declarative Pipelines, Workflows, Unity Catalog, SQL Warehouses Azure : ADLS Gen2, Data Factory, Key Vault, vNets & Private Endpoints Languages : PySpark, Spark SQL, Python, Git DevOps : Azure DevOps Repos, Pipelines, CI/CD Analytics : Power BI, Fabric What We're Looking For Experience 5-8+ years ...

with ambiguity and will want to dive deep and fix problems at scale. The environment looks a bit like this 👇 👉 Python (pandas, numpy, scipy, PySpark) 👉 SQL 👉 AWS or GCP (BigQuery/RedShift/Snowflake) 👉 Spark/Databricks 👉 Airflow/Dagster ...

independently and collaborate within cross-functional teams Some knowledge of programming languages such as Python is preferred Experience with data processing technologies such as PySpark is preferred Familiarity with AWS* services and cloud-based data workflows is preferred ...

able to develop the prototypes of the models, and do some feature engineering. The environment looks a bit like this 👇 👉 Python (pandas, numpy, scipy, PySpark) 👉 SQL 👉 AWS or GCP (BigQuery/RedShift/Snowflake ...

expertise in data processing within a cloud-based data platform. Junior Data Engineer duties: Develop, test, and maintain scalable data pipelines using Python and PySpark within an Azure Databricks development environment Clean, aggregate, and transform complex datasets to meet business requirements. Assist with data quality checks and support data … Junior Data Engineer, you will have: Strong proficiency in Python, particularly for data manipulation and automation scripts; experience in libraries such as pandas, PySpark, numpy, pyodbc or associated would be beneficial Hands-on experience (or academic project experience) using PySpark. An understanding of ETL/ELT processes and/ ...

support analytics and downstream use cases Developing scalable data solutions using Python , with a focus on maintainability and performance Working with Apache Spark/PySpark to process and transform large datasets Supporting the ingestion, transformation and validation of complex financial data Improving the performance, reliability and resilience of existing … Succeed Essential Commercial experience as a Data Engineer at a mid‐level Strong Python development skills Hands‐on experience with Apache Spark/PySpark Solid experience building ETL/ELT pipelines Background within the financial services industry (investment management experience desirable) Comfortable working with production systems in a regulated ...

innovative data platform. You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

working with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

processing, and analytics • Designing scalable, secure, and high performance architectures using Databricks and distributed data systems • Building and overseeing robust data ingestion pipelines using PySpark, with a strong focus on reliability and accuracy • Ensuring end to end data quality from raw ingestion through to curated datasets used for reporting … wider regulatory requirements • Evaluating and introducing new technologies where they add clear value to the data ecosystem What you will bring • Deep expertise in PySpark and distributed data processing at scale • Strong hands on experience designing, building, and optimising Databricks based platforms • Advanced SQL skills including performance optimisation ...

deploying the Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring … logging and alerting for platform reliability. YOUR SKILLS AND EXPERIENCE: Deep expertise in Databricks, PySpark, Python and SQL. Proven experience designing and delivering Lakehouse solutions. Strong DevOps capabilities, including CI/CD and automated testing. Experience with Infrastructure as Code and cloud automation. Familiarity with monitoring, logging and alerting ...

doing Design, build and deploy algorithms/data models supporting pricing, forecasting and optimisation use cases in production Develop and optimise distributed Spark/PySpark batch pipelines for large-scale data processing Write production-grade Python workflows implementing complex, explainable business logic Work with Databricks for job execution, orchestration … delivering production algorithms/data models (forecasting, pricing, optimisation or similar) Strong Python proficiency and modern data stack exposure (SQL, Pandas/NumPy + PySpark; Dask/Polars/DuckDB a bonus) build, schedule and optimise Spark/PySpark pipelines in Databricks (Jobs/workflows, performance tuning, production ...

tagging, monitoring and platform operations. • Architect and design end‐to‐end data platform solutions focused on scalability, security and reliability. • Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. • Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. • Collaborate closely with data … followed. Your Skills and Experience You will bring: • Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. • Proficiency in PySpark, Python, SQL and Spark SQL. • Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. • Experience with monitoring, logging and alerting solutions. ...

tagging, monitoring and platform operations. • Architect and design end‐to‐end data platform solutions focused on scalability, security and reliability. • Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. • Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. • Collaborate closely with data … followed. Your Skills and Experience You will bring: • Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. • Proficiency in PySpark, Python, SQL and Spark SQL. • Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. • Experience with monitoring, logging and alerting solutions. ...

tagging, monitoring and platform operations. * Architect and design end-to-end data platform solutions focused on scalability, security and reliability. * Lead delivery using Databricks, PySpark, Spark SQL and Azure Data Factory. * Drive DevOps best practices including CI/CD, automated testing and infrastructure as code. * Collaborate closely with data … followed. Your Skills and Experience You will bring: * Strong commercial experience in Azure and Databricks, with deep knowledge of Lakehouse architecture. * Proficiency in PySpark, Python, SQL and Spark SQL. * Strong DevOps engineering skills including CI/CD, automated testing and deployment pipelines. * Experience with monitoring, logging and alerting solutions. ...