51 to 75 of 195 PySpark Jobs in London

years of relevant in data governance, compliance, or automation programs. Strong proficiency in Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet . Advanced SQL skills; experience with Oracle databases . Experience designing ...

/reporting outputs such as Tableau dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Hands-on exposure to DCRM tooling and operational exception management processes. Experience with governance/ ...

Azure Resource Manager/Bicep, or Google Cloud Deployment Manager) to build, maintain, and scale complex cloud environments Strong coding proficiency in Python and PySpark/Spark: you must demonstrate live coding, debugging, and solution‐building skills Deep expertise in distributed data systems architecture: designing scalable pipelines, streaming architectures ...

Azure Resource Manager/Bicep, or Google Cloud Deployment Manager) to build, maintain, and scale complex cloud environmentsStrong coding proficiency in Python and PySpark/Spark: you must demonstrate live coding, debugging, and solution-building skillsDeep expertise in distributed data systems architecture: designing scalable pipelines, streaming architectures, lakehouse patterns ...

ingest, cleanse and model data for BI reporting, ML and AI, ensuring quality and governance. The role emphasizes strong Azure skills (ADF, ADLS, Databricks), PySpark, Python and SQL, with a DevOps and CI/CD mindset. Hybrid work with three days in the office weekly is offered. #J ...

troubleshooting, and collaboration with analysts and engineers to raise data quality. The ideal candidate has 6+ years in data engineering, strong SQL, Python/PySpark, and experience with Databricks and Azure, with a passion #J-18808-Ljbffr ...

tools, and emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated ...

also a key driver in translating Vision and Data Strategy into IT solutions and delivering on them. Strong hands-on expertise in Databricks, PySpark, Python and SQL is central to this role. Key Skills Azure Stack ELT Experience Data Modeling Data Integration & Ingestion Data Manipulation and Processing GitHub, GitHub ...

about its values Values | PA ConsultingWe use a range of different tools and technologies depending on the specific task, these include: Data Bricks, Python, PySpark, R, Excel, Palantir Foundry and PowerBI.QualificationsEssential requirements Even if you don't meet every requirement below, feel free to still apply ...

PostgreSQL, MySQL, and Aurora. Integrating and migrating data using AWS DMS, Glue Connectors, EventBridge, SNS, and SQS. Strong programming capability in Python and PySpark for large-scale data processing. Proficiency in writing complex SQL queries, analytical functions, and performance tuning for large datasets. Familiarity with NoSQL databases such ...

looking for a Senior Data Engineer - Tech Lead (Databricks, PySpark) to join our team in London, UK, in a hybrid working mode. In this role, you will lead the design, development and optimization of scalable cloud-native data architectures, focusing on Azure Databricks, PySpark and Lakehouse principles. … design and build of large-scale data platforms using Azure Databricks and modern cloud technologies Implement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principles Enhance performance, manage cloud costs and ensure platform reliability for structured streaming workloads Define data governance, security ...

knowledge (you don't need everything) Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

management, network design and access control; Databricks and OCI, or comparable platforms Strong Python and SQL, and experience building and monitoring production pipelines using PySpark or Spark Experience with infrastructure as code, CI/CD and a Git and GitHub workflow A track record of setting engineering standards ...

data engineering experience BigQuery, including partitioning, clustering and performance tuning Dataflow, Dataproc and Pub/Sub Cloud Composer/Apache Airflow Python and PySpark SQL Cloud Storage Terraform Git and CI/CD Experience with structured and unstructured data For more information or to apply directly, please email ...

client sites or in our office, with hybrid work plan of at least two days per week. Use tools such as Data Bricks, Python, PySpark, R, Excel, Palantir Foundry and PowerBI to deliver scalable, high‐performance applications. Qualifications Essential requirements A degree, Masters or PhD in data science, mathematics ...

Data Lake Storage (Gen2), Purview, Cosmos DB, OpenAI, AzureML, AIFoundry, Kubernetes Understanding of software engineering tools and concepts including experience in Python, Scala or PySpark, database technologies, data modelling and SQL Confident communicator who is able to explain technical terms to non‐technical audiences and mentor junior colleagues Leads ...

fields.· Proven track record of designing, developing, and deploying AI systems in production environments.· Proficient in Python and key ML libraries (e.g. PyTorch, PySpark, scikit-learn, Hugging Face Transformers).· Hands-on experience with modern data platforms and AI tooling such as Azure ML, Databricks, MLflow, LangChain, LangGraph.· Proven ...

experience with cloud data platforms such as Azure or AWS, with a focus on Databricks. Highly skilled with Pythonic data processing tools such as Pyspark and SparkSQL. Strong understanding of data modelling concepts relational and non-relational databases. Proficiency in SQL and experience with data integration and ETL tools. ...

cloud-native experience Preferred Qualifications, Capabilities, and Skills: Experience with modern data technologies such as Databricks or Snowflake Hands-on experience with Spark/PySpark and other big data processing technologies Demonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial intelligence, machine ...

securityPractical cloud-native experiencePreferred Qualifications, Capabilities, and Skills: Experience with modern data technologies such as Databricks or SnowflakeHands-on experience with Spark/PySpark and other big data processing technologiesDemonstrated proficiency in software applications and technical processes within disciplines such as data engineering, cloud, artificial intelligence, machine learning ...

MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management. Strong Python experience, with helpful knowledge of Spark or PySpark for working with large‐scale data and machine learning systems. An understanding of feature stores and related data technologies used to support operational machine ...

knowledge (you don't need everything) Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/ ...

truth. You will design homogeneous data repositories and oversee pipelines, working within a hybrid office setup at One New Change, London. You will apply PySpark, Python, SQL, and Kimball modelling to support BI, ML, and analytics while following DevOps practices and data governance standards. Strong collaboration and communication ...

complex data led organisation* Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI* Expertise in Streaming Data, Event Driven Design ...

complex data led organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design ...