176 to 200 of 250 PySpark Jobs in the UK

query, validate and optimise data across cloud-based analytical platforms such as Athena, Redshift and Azure SynapseBuild scalable data processing solutions using PySpark, Python and cloud-native orchestration servicesImplement data quality checks, validation processes, monitoring and logging to ensure trusted and reliable data deliverySupport the performance tuning and optimisation … data engineering, integration and analytics challengesCollaborate with experienced engineers, architects and data specialists in supportive, multidisciplinary teamsDevelop expertise across cloud-native data services, PySpark, SQL and modern engineering practicesAccess mentoring, knowledge sharing and development opportunities to help you grow your careerContribute to solutions that deliver measurable value and meaningful ...

visible role where you'll combine strong technical skills with commercial thinking to drive meaningful product improvements. Your Skills & Experience Strong SQL skills Python (PySpark experience is beneficial) Experience in Product Analytics Strong knowledge of A/B testing and experimentation Excellent statistical analysis skills, with exposure to modelling ...

data.Knowledge of Bayesian methods for causal analysis and decision-making.Experience in gaming, e-commerce, or subscription-based products.Experience working with large-scale data using PySpark or equivalent distributed data processing tools.Familiarity with production environments, MLOps, or data pipelines.Benefits: Discretionary bonus opportunityPrivate Medical InsuranceDental Scheme25 days holiday per yearOn Site ...

global organization through resourcing/location strategy. Experience and Qualifications Proven track record delivering solutions in complex organizations. Experience with AWS, Databricks, DBT, PySpark, React, JavaScript, Terraform, etc. Fast learner, thrives in ambiguity, translates vision into product scope. Passion for automation and user experience innovation. Builds high‐performing teams ...

location strategy.“I am the person Capital Group is looking for.” Proven track record delivering solutions in complex organizations. Experience with AWS, Databricks, DBT, PySpark, React, JavaScript Terraform, etc. Fast learner, thrives in ambiguity, translates vision into product scope. Passion for automation and user experience innovation. Builds high-performing ...

including architecture, development, deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture (Bronze/Silver … reviews. Skills & Experience Strong experience in dataengineering, big data, or solution architecture. Strong hands-onexperience with Databricks Platform . Expert knowledge of: Apache Spark PySpark/Scala Delta Lake SQL Data Lakehouse architecture ETL/ELT development Experience designing anddelivering enterprise Databrickssolutions. Strong experience withcloud platforms: Azure Databricks (preferred ...

DescriptionRole –Senior Databricks ArchitectTechnology – Databricks (Spark, PySpark/SQL) and data lake architecturesLocation – London, UKCompensation – Competitive (including bonus)Job DescriptionWe are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role focuses on integrating legacy … systems with cloud-native platforms to support analytics, reporting, and domain-owned data products.Primary Skills• Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures.• Hands-on experience with streaming technologies, especially Kafka/Confluent Kafka.• Experience integrating legacy data sources such as DB2 (mainframe ...

Data Engineer £54-59,000 + bonus Hybrid Working – 1 day on-site per week West Midlands based I'm recruiting for a Data Engineer to join an established data team, working on the development ...

modelling, Kimball, or Data Vault Exposure to modern cloud data platforms, ideally within the Microsoft ecosystem Familiarity with tools such as DBT, Python, or PySpark is beneficial Experience working in data transformation or modernisation programmes Understanding of architecture principles or a clear interest in developing into a Data Architect ...

experience building production services, data pipelines, APIs, workflow systems, or developer tools.Experience with at least one distributed data or compute system such as Spark, PySpark, Flink, Kafka, Ray, Airflow, Kubernetes, BigQuery, or similar technologies.Familiarity with ML data workflows such as feature generation, training dataset creation, batch processing, real-time ...

business questions and deliver custom data products seamlesslyThis is the perfect job for someone who has:At least 2 years experience with SQL/pySpark for data analysis - MustExperience in Python scripting - MustExcellent verbal and communication skills in English - MustAWS Infrastructure experience - Big AdvantageWorking with clients - Big Advantage.Proficiency ...

showcase data-powered products and engage with stakeholders to identify new opportunities. The role requires expertise in SQL and data analytics platforms (Databricks, PySpark, Tableau), and a Bachelor's in Engineering, Computer Science, Statistics, or Mathematics. #J-18808-Ljbffr ...

passion for realtime data, The Data Scientist must have: • Experience in building complex data pipelines/ETL/ELT scripts • Python/PySpark as our Data pipelines will be built in PySpark • SQL • Knowledge of AWS and components such as S3, Glue, Athena, RDS • Experience with Data Warehouse ...

technology sprawl while improving security, supportability and delivery speed. 4. Data Engineering and Integration Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake. Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls. Design reusable integration … Databricks Serverless and compute/workload optimisation Azure identity, networking, security, secrets, monitoring and private connectivity Databricks SQL, Delta Lake and performance optimisation Python, PySpark and SQL Jobs/workflows, incremental processing, CDC and data quality REST/API and external data integration patterns FinOps, cost attribution, tagging, budgets ...

technology sprawl while improving security, supportability and delivery speed. 4. Data Engineering and Integration Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake. Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls. Design reusable integration … Databricks Serverless and compute/workload optimisation Azure identity, networking, security, secrets, monitoring and private connectivity Databricks SQL, Delta Lake and performance optimisation Python, PySpark and SQL Jobs/workflows, incremental processing, CDC and data quality REST/API and external data integration patterns FinOps, cost attribution, tagging, budgets ...

robust and standardised data platform experience.What you’ll be doingDesigning, building, and maintaining scalable data pipelines using Python and Scala, leveraging Spark and PySpark within an Azure ecosystem (including Azure Data Factory and Databricks).Developing and maintaining reusable data engineering templates, frameworks, and tooling to support data teams across … succeed.QualificationsStrong experience as a Data Engineer building scalable data platformsDeep expertise in Azure (ADF, ADLS, Databricks)Proficiency in Python and/or Scala(PySpark/Spark) for large-scale data processingHands-on experience with Databricks and Delta LakeSolid understanding of the end-to-end data lifecycle (ingestion transformation serving ...

Senior Data Scientist Welwyn Garden City, Hertfordshire or Slough, Berkshire | Hybrid (2 Days Office) | Python | SQL | PySpark | Azure | Databricks | Machine Learning | Statistics | Time Series Forecasting | Propensity Models Role: Senior Data Scientist, Lead Data Scientist, ML Scientist, AI Data Scientist, Statistical Data Scientist, Predictive Analytics Key Skills: Senior Data Scientist … Data Scientist, Machine Learning, Artificial Intelligence, Predictive Analytics, Statistical Modelling, Python, SQL, PySpark, Azure, Databricks, MLflow, GitHub Actions, CI/CD, Time Series Forecasting, Propensity Modelling, Feature Engineering, Data Engineering, Azure Machine Learning, Azure Data Platform, Experiment Design, A/B Testing, Hypothesis Testing, Predictive Modelling, Customer Analytics, Marketing ...

knowledge, with the ability to write stable, scalable and maintainable code. Experience with data science Python libraries such as Sckit-learn, Pandas, NumPy, PyTorch, PySpark, LightGBM. Have worked with a cloud service, such as AWS. Familiarity developing Infrastructure as code (e.g Terraform, cloudformation). Have some experience with data … engineering, building data pipelines with PySpark, SQL (E.g in Databricks, Glue) to power machine learning applications. Comfortable working with Docker and containerised applications. Experience leveraging AI native engineering tooling. Passion for building products that meet customer needs and business objectives. Strong sense of responsibility and a track record ...

closely with architects, senior engineers, and analysts to deliver scalable and reliable migration solutions, using technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML configurations. You should be comfortable working in a collaborative, delivery-focused environment and have a strong interest in data migration, cloud … pipelines on AWS services, including AWS Glue and Apache IcebergSupport validation of data pipeline migrations to AWS Data Lakehouse architecturesTest pipelines using:Python/PySpark transformationsSQL-based validation logicYAML-driven configurationsExecute and validate:Initial bulk data loadsIncremental/delta data processingWrite and run SQL queries to validate:Data completenessData ...

Senior Data Engineer, Lead Data Engineer, Data Engineer, ETL, AWS, AWS Glue, AWS Lambda, AWS S3, AWS Redshift, Amazon EMR, Python, SQL, Pipeline Orchestration, PySpark, data warehousing, data lakehouse, data architecture, Agile, Senior Data Engineer, Lead Data Engineer, Data Engineer) Our client is a global innovator and world leader … with AWS services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR. Proven expertise in Python, SQL, Pipeline Orchestration and PySpark is also expected, as is experience with data lakehouse architectures and data warehousing solutions. Agile experience is also key, as are excellent communication skills ...

ongoing performance and reliability of the SQL database estate. This role would suit a Data Engineer with strong Databricks, SQL and Python/PySpark expertise who enjoys solving technical challenges, improving data platforms and working collaboratively with both technical and non-technical stakeholders. Key Responsibilities Design, develop and maintain … Oxfordshire HQ Opportunity to join a growing Data & Innovation function Hands-on exposure to modern Microsoft data technologies including Databricks, SQL, Python and PySpark Opportunity to influence the development and maturity of the organisation's data platform Collaborative working environment with exposure to a broad range of data ...

/ELT processes for data ingestion and transformation. Supporting and enhancing a business-critical reporting and analytics platform. Developing SQL and Python/PySpark-based data solutions. Managing and optimising SQL database environments, ensuring performance, reliability and scalability. Collaborating with Data Analysts, BI professionals and stakeholders across the organisation. … skills and experience working with complex relational databases. Hands-on experience building and supporting ETL/ELT pipelines. Strong Python and/or PySpark development experience. Experience with SQL Server or similar database technologies. Excellent problem-solving skills and the ability to engage effectively with both technical ...

cause understandingMaintenance of accurate documentation, lineage and operational run books to support ongoing platform supportKey ResponsibilitiesDevelop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL)Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta LakeSupport and operate Azure Databricks and Azure Synapse platforms, including … standards, CI/CD pipelines and code reviewsWhat you'll need to succeedEssentialProven experience as a Data EngineerStrong hands-on experience with AzureDatabricks, Python, PySpark and SQLExperience supporting Azure data platforms in productionUnderstanding of medallion architecture and Delta LakeDesirableExperience with Azure Synapse AnalyticsExperience with Azure Data FactoryExperience in financial ...

delivery. What You'll Be Doing Design, develop and maintain data pipelines and data engineering solutions using Databrick sBuild scalable data processing solutions using PySpark and SQL Work across Databricks Lakehouse environments, supporting implementation, optimisation and ongoing improvement Develop and optimise batch and data processing workflows Contribute to technical … transformation journeys What We're Looking For Strong commercial experience in Data Engineering Hands-on experience with Databricks and/or Apache Spark Strong PySpark and SQL development skills Experience designing, building and supporting data pipelines Experience working with Azure or AWS Good understanding of modern data platforms ...

Cyber Analytics team.You will need: A strong passion for big data, data modelling, data quality and ETLs. Solid programming experience with Python/PySpark along with proficiency in SQL. Extensive experience working with ETL and ELT frameworks and orchestration tools such as Airflow and dbt. Experience building robust data … Cyber Analytics team. You will need A strong passion for big data, data modelling, data quality and ETLs. Solid programming experience with Python/PySpark along with proficiency in SQL. Extensive experience working with ETL and ELT frameworks and orchestration tools such as Airflow and dbt. Experience building robust ...