151 to 175 of 322 Permanent PySpark Jobs

Only - No C2C/No Third Parties Key Skills: Databricks, LeanIX Data Lake, Unity Catalog, Iceberg REST Catalogs, Databricks Asset Bundles, Spark Declarative Pipelines, PySpark, SQL, Python, API Ingestion, Data Governance, Data Quality, Power BI, Data Mesh. Role Summary: Seeking an experienced Sr. Data Analyst/Data Engineer ...

best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs Use of Jupyter notebooks and/or Amazon Athena to query and validate data Data analysis ...

data processing & Spark Amazon Bedrock - Generative AI & foundation models AWS Lambda - Serverless data & AI workflows Amazon QuickSight - BI, dashboards & visualisation Python, SQL, R & PySpark - Data science & analytics As an equal opportunities’ employer, we welcome applications from individuals of all backgrounds. However, for you to be eligible for this role ...

knowledge (you don't need everything) Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

maintain the data platform powering reporting, analytics and data science across the business. You’ll work hands‐on with Microsoft Fabric, SQL, Python/PySpark and Power BI, shaping data from booking systems, databases and APIs into trusted, well‐modeled datasets. You’ll join a hybrid team, collaborating with ...

Location: London; UKWork type: HybridSeniority: SeniorPrimary skill: Data Software EngineeringPosted: 2026-09-29We're looking for a Senior Data Engineer – Tech Lead (Databricks, PySpark) to join our team in London, UK, in a hybrid working mode. In this role, you will lead the design, development and optimization … scalable cloud-native data architectures, focusing on Azure Databricks, PySpark and Lakehouse principles. You will work hands-on to deliver performant data solutions for high-volume workloads, ensuring governance, reliability and best practices for enterprise-grade platforms. As a technical leader, you will define data strategies, drive modernization initiatives ...

experience with Databricks — running queries, navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation. Experience with Databricks and PySpark for large‐scale data processing. Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring. Experience implementing responsible ...

ODCS/ODPS), data product portals, lineage tools. Experience in a regulated financial services company. Experience leading a migration or a large platform change. PySpark or other distributed processing experience. We're building a place where everyone truly feels they belong. Even if your past experience doesn't align ...

target reconciliation, validate financial calculations, and ensure 100% data integrity across Bronze/Silver/Gold layers. Responsibilities include building automated frameworks with PyTest, PySpark, and advanced SQL, integrating checks into Azure DevOps pipelines, and monitoring schema drift and #J-18808-Ljbffr ...

data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers … latest data engineering technologies and best practices. Key Responsibilities Data Engineering & Development Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark Build and optimise ETL/ELT solutions processing large-scale datasets Develop high-quality, reusable code and engineering frameworks Deliver robust, performant and maintainable data ...

limited to, the duties listed below: Databricks Engineering Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake. Data Lakehouse Architecture Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers … Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities. Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic. Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation. AWS and/or Databricks certifications, such ...

agreed requirements. What youll work on The core environment is Microsoft Fabric (Lakehouse, Warehouse, OneLake, pipelines and notebooks) on a medallion architecture, alongside Python, PySpark and advanced SQL. Youll build and refine the pipelines, models and integrations that feed reporting, analytics and wider business data needs, including: Developing … like to hear how your solutions performed after launch and what you changed to make them better. Youll also have strong Python/PySpark and advanced SQL, a solid grasp of data modelling, orchestration and data quality, and real experience supporting production data workloads. An understanding of data governance ...

work Microsoft Fabric sits at the centre of the platform: Lakehouse, Warehouse, OneLake, pipelines and notebooks, built around a medallion architecture. Youll use Python, PySpark and advanced SQL to build the pipelines, models and integrations behind the businesss reporting, analytics and wider data needs. Youll also: Integrate internal … least a year in that role working with Fabric, so youve lived with your own design decisions. Youll also bring: Strong Python/PySpark and advanced SQL Sound data modelling, ETL/ELT, orchestration and data quality experience A good grasp of data governance, privacy and security ...

We are looking for an experienced Data Engineer to join our IT team and play a key role in the development of our firmwide data capability. This is a newly created position and our first ...

/GCP Exposure to LLM models like ChatGPT, Google Bard etc. Exposure to Langchain, RAG and Fine Tuning of the LLM Models. Exposure to PySpark and Databricks for Data pipeline creation. Familiarity with modern front-end technologies Exposure to cloud technologies Experience with ReactJS front-end development ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications Advanced Data Management – Strong SQL design, query ...

Machine Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query ...

financial services. You should have practical knowledge of: View21 – Core View and Pre-View Anti-Money Laundering (AML) Way4 Cards Auto Loans AWS PySpark SAP BODS Informatica AWS Glue EMR and Serverless Spark IAM Secrets Manager CloudWatch Microsoft SQL Server You will also have strong SQL, data validation ...

Experience debugging moderately complex systems and improving reliability/performance Strong fundamentals: data structures, testing, version control, Linux basics Nice to have: Spark/PySpark experience Hadoop ecosystem exposure (HDFS/HBase) Workflow orchestration (Airflow/Dagster/NiFi) Search/indexing (OpenSearch, MongoDB) Kubernetes and infrastructure‐as‐code ...

adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience supporting data processing and integration work using Python and/or PySpark, such as batch jobs, reconciliation pipelines, or platform integrations Experience with data engineering and orchestration tooling such as Databricks and Airflow or similar platforms Exposure ...

troubleshooting of Azure DatabricksDatabricks Serverless configuration and workload optimisationDatabricks SQL and Delta Lake developmentCluster management, compute policies, autoscaling, and workload monitoringData EngineeringStrong Python, PySpark, and SQL development skillsDesign and build ingestion and transformation pipelinesFull and incremental data loadingCDC (Change Data Capture) implementationData quality controls, schema evolution, reconciliation, and error ...

STACK Infrastructure EMEA is seeking a Senior Databricks Data Engineer to build and manage the enterprise data platform, applying PySpark, Python, SQL and Delta Lake in production. You will design reliable pipelines across Finance, Delivery, Operations, HR and PMO, implement governance with Unity Catalog, and optimise performance and costs. ...

This 12-month, onsite contract (5 days/week) offers an ASAP or 4-week start and focuses on enterprise data lakehouse architectures, Python, PySpark, and dbt to deliver analytics-ready data products. You will collaborate with data architects and stakeholders to ensure data governance, quality, and cost efficiency ...

trends using Tableau, Power BI and AWS Connect Customer, collaborating with business stakeholders to productionise solutions. The role requires strong SQL and Python/PySpark skills, plus experience leading technical teams and mentoring #J-18808-Ljbffr ...