151 to 175 of 343 PySpark Jobs

pipelines using modern cloud-native data engineering technologies. Provide technical direction across data modelling, ingestion, transformation, orchestration and integration. Apply practical experience of Spark, PySpark, Python and SQL within enterprise data environments. Governance, Security & Quality Embed effective data governance, security, quality and access controls within cloud data environments. Ensure … Fabric or another cloud data platform. Strong understanding of modern data architecture patterns, data modelling and ETL/ELT approaches. Practical experience with Spark, PySpark, Python and SQL and the design of scalable batch and real-time data pipelines. Experience designing appropriate data governance, security and data quality controls. ...

analytics and data science across Veezu. Reporting into the Director of Data Engineering, you’ll work hands‐on with Microsoft Fabric, SQL, Python/PySpark and Power BI, bringing data in from our booking systems, databases and third‐party APIs and shaping it into trusted, well‐modelled datasets that … more! What you’ll do: Design, build and maintain ETL/ELT pipelines in Microsoft Fabric using Pipelines, Notebooks, Lakehouses, Warehouses and OneLake. Write PySpark notebooks to ingest and transform data from SQL Server databases, REST APIs, SaaS applications and file‐based exports. Develop and optimise SQL transformations, stored ...

analytics and data science across Veezu. Reporting into the Director of Data Engineering, you'll work hands‐on with Microsoft Fabric, SQL, Python/PySpark and Power BI, bringing data in from our booking systems, databases and third‐party APIs and shaping it into trusted, well‐modelled datasets that … more! What you’ll do: Design, build and maintain ETL/ELT pipelines in Microsoft Fabric using Pipelines, Notebooks, Lakehouses, Warehouses and OneLake. Write PySpark notebooks to ingest and transform data from SQL Server databases, REST APIs, SaaS applications and file‐based exports. Develop and optimise SQL transformations, stored ...

management (Metadata Driven Ingestion). Good working experience of database technologies and database integration, such as SQL, Postgres, MySQL or Oracle. Experience in Python (PySpark) Experience in working with DevOps capabilities like version control, automated builds, testing and release management capabilities using Microsoft DevOps and Git repositories. Good numerical … technology roles, such as architecture, operations, and design related disciplines. At least three years of experience in Azure (Azure Synapse, Azure Databricks, Fabric, PySpark). Comprehensive experience with data modelling Experience across a range of other IT functional disciplines will be of benefit, including: Data Platform tools CI/ ...

resolve data engineering challenges, supporting continuous platform improvement. Migrate data from legacy systems into Azure-based data lakes and data warehouses. Develop and maintain PySpark notebooks and data processing solutions. Improve data quality, reliability, and performance across data pipelines. Leverage AI-driven tools to enhance coding, testing, and documentation … data lakes, and end-to-end data pipelines. Proficiency in SQL, stored procedures, and working with relational and NoSQL databases. Hands-on experience with PySpark and Microsoft Fabric. Knowledge of DevOps, CI/CD, Agile delivery, and data quality best practices. Strong analytical, problem-solving, and troubleshooting skills. Experience ...

Only - No C2C/No Third Parties Key Skills: Databricks, LeanIX Data Lake, Unity Catalog, Iceberg REST Catalogs, Databricks Asset Bundles, Spark Declarative Pipelines, PySpark, SQL, Python, API Ingestion, Data Governance, Data Quality, Power BI, Data Mesh. Role Summary: Seeking an experienced Sr. Data Analyst/Data Engineer ...

best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs Use of Jupyter notebooks and/or Amazon Athena to query and validate data Data analysis ...

data processing & Spark Amazon Bedrock - Generative AI & foundation models AWS Lambda - Serverless data & AI workflows Amazon QuickSight - BI, dashboards & visualisation Python, SQL, R & PySpark - Data science & analytics As an equal opportunities’ employer, we welcome applications from individuals of all backgrounds. However, for you to be eligible for this role ...

knowledge (you don't need everything) Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

maintain the data platform powering reporting, analytics and data science across the business. You’ll work hands‐on with Microsoft Fabric, SQL, Python/PySpark and Power BI, shaping data from booking systems, databases and APIs into trusted, well‐modeled datasets. You’ll join a hybrid team, collaborating with ...

Location: London; UKWork type: HybridSeniority: SeniorPrimary skill: Data Software EngineeringPosted: 2026-09-29We're looking for a Senior Data Engineer – Tech Lead (Databricks, PySpark) to join our team in London, UK, in a hybrid working mode. In this role, you will lead the design, development and optimization … scalable cloud-native data architectures, focusing on Azure Databricks, PySpark and Lakehouse principles. You will work hands-on to deliver performant data solutions for high-volume workloads, ensuring governance, reliability and best practices for enterprise-grade platforms. As a technical leader, you will define data strategies, drive modernization initiatives ...

Strong, demonstrable experience running A/B testing and causal inference in a product-focused environment. Core Technical Stack: Proficiency with SQL , Python/PySpark , and Snowflake . Experience with data visualisation tools (e.g. Tableau, Power BI) Hands-on experience with A/B testing and experimentation Experience with ...

experience with Databricks — running queries, navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation. Experience with Databricks and PySpark for large‐scale data processing. Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring. Experience implementing responsible ...

ODCS/ODPS), data product portals, lineage tools. Experience in a regulated financial services company. Experience leading a migration or a large platform change. PySpark or other distributed processing experience. We're building a place where everyone truly feels they belong. Even if your past experience doesn't align ...

target reconciliation, validate financial calculations, and ensure 100% data integrity across Bronze/Silver/Gold layers. Responsibilities include building automated frameworks with PyTest, PySpark, and advanced SQL, integrating checks into Azure DevOps pipelines, and monitoring schema drift and #J-18808-Ljbffr ...

data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers … latest data engineering technologies and best practices. Key Responsibilities Data Engineering & Development Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark Build and optimise ETL/ELT solutions processing large-scale datasets Develop high-quality, reusable code and engineering frameworks Deliver robust, performant and maintainable data ...

limited to, the duties listed below: Databricks Engineering Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake. Data Lakehouse Architecture Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers … Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities. Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic. Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation. AWS and/or Databricks certifications, such ...

agreed requirements. What youll work on The core environment is Microsoft Fabric (Lakehouse, Warehouse, OneLake, pipelines and notebooks) on a medallion architecture, alongside Python, PySpark and advanced SQL. Youll build and refine the pipelines, models and integrations that feed reporting, analytics and wider business data needs, including: Developing … like to hear how your solutions performed after launch and what you changed to make them better. Youll also have strong Python/PySpark and advanced SQL, a solid grasp of data modelling, orchestration and data quality, and real experience supporting production data workloads. An understanding of data governance ...

work Microsoft Fabric sits at the centre of the platform: Lakehouse, Warehouse, OneLake, pipelines and notebooks, built around a medallion architecture. Youll use Python, PySpark and advanced SQL to build the pipelines, models and integrations behind the businesss reporting, analytics and wider data needs. Youll also: Integrate internal … least a year in that role working with Fabric, so youve lived with your own design decisions. Youll also bring: Strong Python/PySpark and advanced SQL Sound data modelling, ETL/ELT, orchestration and data quality experience A good grasp of data governance, privacy and security ...

We are looking for an experienced Data Engineer to join our IT team and play a key role in the development of our firmwide data capability. This is a newly created position and our first ...

/GCP Exposure to LLM models like ChatGPT, Google Bard etc. Exposure to Langchain, RAG and Fine Tuning of the LLM Models. Exposure to PySpark and Databricks for Data pipeline creation. Familiarity with modern front-end technologies Exposure to cloud technologies Experience with ReactJS front-end development ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications Advanced Data Management – Strong SQL design, query ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query ...

financial services. You should have practical knowledge of: View21 – Core View and Pre-View Anti-Money Laundering (AML) Way4 Cards Auto Loans AWS PySpark SAP BODS Informatica AWS Glue EMR and Serverless Spark IAM Secrets Manager CloudWatch Microsoft SQL Server You will also have strong SQL, data validation ...