51 to 75 of 104 PySpark Jobs in the UK

understanding of DAG (Directed Acyclic Graphs) and their role in pipeline orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic ...

Azure Data Platform and Databricks ecosystem. The ideal candidate should have strong hands-on experience with Structured Streaming, Delta Live Tables (DLT), PySpark architecture, Python, and SQL, and a proven track record of delivering large-scale enterprise data platforms. Youll be responsible for: Leading the design, development, and implementation ...

ongoing basis. Contribute to the planning, research, recommendation, and implementation of current and future IT strategies. PERSON SPECIFICATION: Essential Demonstrable experience of Spark (PySpark) and SQL. Demonstrable experience of Fabric components. Demonstrable experience of working on enterprise environments. Experience of designing architectural Fabric processes. Experience of Azure Data Factory. ...

experiment design and conducting A/B tests Bonus Points – Experience building shared or platform‐style ML systems Bonus Points – Experience with Databricks and PySpark Bonus Points – Experience working with AWS or another cloud platform (GCP/Azure) Additional Information Health + Mental Wellbeing: PMI and cash plan healthcare ...

through best practices, mentoring, and thoughtful technical direction. \n Job responsibilities \n \n Design scalable, reusable data processing and data quality frameworks using Python, PySpark, and dbt \nBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow … data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence and reporting workloadsWrite production-grade Python/PySpark code with disciplined testing, performance tuning, and maintainable object-oriented designImplement infrastructure-as-code (e.g., Terraform) to provision and manage cloud-based data platform ...

/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata management, lineage, and monitoring. Leadership & Stakeholder Engagement Work closely with business leaders and product teams to translate … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

practices. Strong AWS knowledge across IAM, S3, KMS, and CI/CD tooling such as CodePipeline, CodeBuild, or equivalent. Expert Python development skills, with PySpark experience highly desirable. Proven experience designing enterprise MLOps frameworks, including model registries, monitoring, governance, and deployment automation. Strong understanding of statistical validation and model ...

recommendation models Experience with deep learning & large language models Experience with experiment design and conducting A/B tests Experience with Databricks and PySpark Experience working with AWS or another cloud platform (GCP/Azure) Additional Information Health + Mental Wellbeing PMI and cash plan healthcare access with Bupa ...

machine learning operations. Excellent communication skills with the ability to influence cross-functional teams and stakeholders. Experience with Python, SQL, Java, HQL, Hive, Pyspark, or similar programming languages preferred Well-versed in design thinking principles Great problem solving and analytical skills The total cash range for this position ...

between security and broader technology teams to demonstrably improve the overall security posture. Desirable skills and experience: Scripting and automation capabilities, particularly utilising Python, PySpark, and SQL to streamline analytical workflows. Experience navigating cloud-native security environments (AWS, Azure, or GCP) alongside familiarity with the Microsoft 365 ecosystem. Advanced ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

delivery lifecycle from requirements through support. \n Job responsibilities \n \n Build and maintain scalable, reusable data processing and data quality frameworks using Python, PySpark, and dbt \nBuild and operate batch and streaming data pipelines with strong scalability, performance, and fault toleranceDevelop and manage workflow orchestration using tools such … well-scheduled data movement and transformationsImplement and optimize data models and warehouse structures to support analytics and business intelligence workloadsWrite clean, testable Python/PySpark code using object-oriented principles and unit testingImplement infrastructure-as-code for the data platform using TerraformContainerize and deploy services using Docker, Kubernetes ...

datalake migration initiatives, including consolidation of legacy data, medallion architecture design, schema evolution, and cutover planningProvide hands-on data engineering expertise, building and optimising PySpark pipelinesEnsure data is secure, high-quality, accessible, and structured for maximum business valueDrive best practice in data governance, cataloguing, lineage, and standards across … operations, analytics, and reporting teamsEnsure compliance with data protection regulations and internal policiesWhat you'll bring Strong hands-on experience in data engineering, including PySpark and production data pipelinesExperience with Microsoft Fabric, Azure Synapse, or similar platforms (e.g. Databricks, AWS)Proven ability to design and manage scalable data pipelines ...

migration initiatives, including consolidation of legacy data, medallion architecture design, schema evolution, and cutover planning Provide hands-on data engineering expertise, building and optimising PySpark pipelines Ensure data is secure, high-quality, accessible, and structured for maximum business value Drive best practice in data governance, cataloguing, lineage, and standards … reporting teams Ensure compliance with data protection regulations and internal policies What you'll bring Strong hands-on experience in data engineering, including PySpark and production data pipelines Experience with Microsoft Fabric, Azure Synapse, or similar platforms (e.g. Databricks, AWS) Proven ability to design and manage scalable data pipelines ...

Market Risk, CCR, PnL, Trade Lifecycle and Reconciliations \n Knowledge of derivatives, swaps, futures, options and fixed-income products \n\n Desirable \n \n PySpark or distributed data processing \n Exposure to quantitative models and pricing systems \n Global banking environment experience \n 9-15 years' relevant experience ...

Data Intelligence Platform, run hands-on Spark demos, and map technical value to business outcomes. The role requires strong data science/ML knowledge, PySpark/Python/Scala/Java coding, cloud experience (AWS, GCP, Azure), and travel 20–30% across the UK. #J-18808-Ljbffr ...

clearance to work on a UK government account. Must be UK based and hold a valid and current passport. Technology requirements: Python 3/Pyspark 3/4 Python behave - for Behaviour Driven Development and testing Python coverage - Code coverage Strong knowledge of Databricks, including delta parquet data format ...

within the Microsoft ecosystem or Azure Exposure to tools such as Sparx Enterprise Architect or similar modelling tools Familiarity with technologies such as DBT, PySpark or Python is beneficial Interest in progressing into a data architecture focused role Ability to engage with both technical and non technical stakeholders APPLY ...

context AWS CI/CD tooling (CodePipeline/CodeBuild or equivalent) — advanced; you've automated model promotion across environments General and technical Python/PySpark — expert; production-quality code, not just notebook scripts Statistical/parity testing methodology — advanced; you can design and execute parity sign-off on migrated ...

engineering is done across the organisation. Skills & Experience Strong hands-on experience with Azure data platforms Advanced SQL skills Commercial experience using Databricks and PySpark Proven background building and maintaining scalable data pipelines If you're looking for a role where you can combine technical depth, ownership, and influence ...

structured and unstructured datasets Validate large-scale data ingestion, transformation, and processing workflows Ensure data integrity, transformation accuracy, reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis … DynamoDB, AWS CloudWatch, AWS IAM, Python, SQL, Java, Bash, GitLab CI/CD, Jupyter Notebooks, Apache Spark/PySpark, Terraform, Docker Key Skills Strong proficiency in Python, SQL, Java, and Scripting languages such as Bash Hands-on experience testing Big Data platforms and Data Lake solutions Strong experience with ...

well as familiarity with software engineering best practices (such as version control, unit testing, CI/CD) and big data and cloud technologies (PySpark preferred). Experience training and deploying deep learning models in cloud environments (including GPU-enabled workflows) is a strong advantage. \n What ...

years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data … databases, financial data providers) into the Azure Databricks platform. Optimise data pipelines for performance, efficiency, and cost-effectiveness. Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … Microsoft Fabric (Lakehouse, Warehouse, pipelines). Azure Data Factory experience will also be considered if no working experience with fabric Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

exposure to Kotlin, Java, or Scala, or a willingness to work with these technologies Familiarity with cloud environments and scalable data processing, such as PySpark Background in advertising technology, marketing analytics, or auction-based modelling is highly beneficial Interest in bridging data science with production and data engineering environments ...