176 to 200 of 222 PySpark Jobs

closely with architects, senior engineers, and analysts to deliver scalable and reliable migration solutions, using technologies such as AWS Glue, Apache Iceberg, Python/PySpark, SQL, and YAML configurations. You should be comfortable working in a collaborative, delivery-focused environment and have a strong interest in data migration, cloud … pipelines on AWS services, including AWS Glue and Apache IcebergSupport validation of data pipeline migrations to AWS Data Lakehouse architecturesTest pipelines using:Python/PySpark transformationsSQL-based validation logicYAML-driven configurationsExecute and validate:Initial bulk data loadsIncremental/delta data processingWrite and run SQL queries to validate:Data completenessData ...

Senior Data Engineer, Lead Data Engineer, Data Engineer, ETL, AWS, AWS Glue, AWS Lambda, AWS S3, AWS Redshift, Amazon EMR, Python, SQL, Pipeline Orchestration, PySpark, data warehousing, data lakehouse, data architecture, Agile, Senior Data Engineer, Lead Data Engineer, Data Engineer) Our client is a global innovator and world leader … with AWS services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR. Proven expertise in Python, SQL, Pipeline Orchestration and PySpark is also expected, as is experience with data lakehouse architectures and data warehousing solutions. Agile experience is also key, as are excellent communication skills ...

Senior Data Engineer, Lead Data Engineer, Data Engineer, ETL, AWS, AWS Glue, AWS Lambda, AWS S3, AWS Redshift, Amazon EMR, Python, SQL, Pipeline Orchestration, PySpark, data warehousing, data lakehouse, data architecture, Agile, Senior Data Engineer, Lead Data Engineer, Data Engineer)Our client is a global innovator and world leader … with AWS services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR. Proven expertise in Python, SQL, Pipeline Orchestration and PySpark is also expected, as is experience with data lakehouse architectures and data warehousing solutions. Agile experience is also key, as are excellent communication skills ...

Responsibilities Design, develop and maintain modern data platform solutions. Build, optimise and manage data ingestion pipelines. Develop data transformation and modelling solutions using Python, PySpark and SQL. Support the implementation of data lakes, data warehouses and modern analytics architectures. Ensure high standards of data quality, security and performance. Work … date with emerging technologies. Requirements Previous experience working as a Data Engineer or within a similar role. Strong experience with Python or PySpark . Experience with cloud data platforms such as Microsoft Fabric, Databricks or Snowflake . Knowledge of SQL and data transformation techniques. Experience building and maintaining data ...

cause understandingMaintenance of accurate documentation, lineage and operational run books to support ongoing platform supportKey ResponsibilitiesDevelop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL)Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta LakeSupport and operate Azure Databricks and Azure Synapse platforms, including … standards, CI/CD pipelines and code reviewsWhat you'll need to succeedEssentialProven experience as a Data EngineerStrong hands-on experience with AzureDatabricks, Python, PySpark and SQLExperience supporting Azure data platforms in productionUnderstanding of medallion architecture and Delta LakeDesirableExperience with Azure Synapse AnalyticsExperience with Azure Data FactoryExperience in financial ...

Job Advertisement: Software Developer/Senior Data Engineer Location: London Contract Length: 12 Months Daily Rate: £700 (Umbrella) Are you ready to be part of a transformative journey? Our client is embarking on a major ...

Job TitleSenior Data Engineer EMEAJob Description SummaryAbout the teamWe are a Centre of Excellence within a larger EMEA team, with deep specialization in data engineering. Our mission is to build the platform that defines how ...

analysis into practical recommendations for senior and non-technical stakeholders.Experience using at least one data-focused programming or analytical language, such as Python, R, PySpark or similar.What's in it for youBeing a part of M&S is exactly that – playing your part to bring the magic ...

existing pipelines. Working within a growing data team, you'll design scalable data models, build robust ELT pipelines in Databricks using SQL, Python and PySpark, and integrate data from multiple internal and third-party systems. You'll collaborate closely with Analytics Engineers to ensure data is reliable, well-governed … architecture and help define long-term data standards, rather than simply delivering against predefined requirements. You'll need experience with: Databricks, SQL, Python and PySpark Building and optimising ELT/data pipelines Data modelling, lakehouse architecture and modern data platforms Azure data services API and third-party system integration ...

data transformationsTest framework and supporting toolingLead testing of AWS‐based Lakehouse and data pipeline solutions, including: AWS GlueApache IcebergAI‐enabled ETL acceleratorsNeo4jPython/PySpark, SQL and YAML‐driven configurations Quality, Defect & Risk Management Ensure quality issues, risks issues and data defects are identified early, clearly documented and effectively triaged.Monitor … data creation, masking and environment readiness. What experience you'll bring: Extensive experience in data engineering/test management.Strong technical capability in: Python/PySpark, SQL, Oracle, HDFS and YAMLBPSS minimum, SC not mandated but strong likelihood may be requiredStrong experience in leading test automation in data testing, ideally ...

Senior Databricks Architect Technology: Databricks (Spark, PySpark/SQL) and data lake architectures. Location: London, UK. Compensation: Competitive (including bonus). Job Description We are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role … integrating legacy systems with cloud-native platforms to support analytics, reporting, and domain-owned data products. Primary Skills Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures. Hands‐on experience with streaming technologies, especially Kafka/Confluent Kafka. Experience integrating legacy data sources such ...

capabilities, and skills Hands-on experience delivering system design, application development, testing, and operational stability at enterprise scale Expertise in Python and/or PySpark Advanced knowledge of software application development and technical processes, with depth in disciplines such as cloud, AI/ML, or data engineering Experience … scalable solutions Preferred qualifications, capabilities, and skills Experience with modern data platforms such as Databricks or Snowflake Deep hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks ...

Senior Databricks Architect - London (Hybrid) - Databricks - Spark - PySpark - AWS - Up to £115,000 + Bonus Our client, a large-scale organisation undergoing significant investment in data and cloud technologies, is seeking an experienced Senior Databricks Architect to join its expanding Data & AI practice. This is an exciting opportunity … time data pipelines from legacy and modern source systems Implementing real-time and event-driven data processing solutions Developing data transformation frameworks using Spark, PySpark and SQL Designing Lakehouse architectures using Databricks and Medallion Architecture principles Creating scalable, reusable data products to support modern data operating models Integrating data ...

star schema and other best practices to optimise data usability.Sets best practices and standards on how we use SQL & dbt/Python/Pyspark and/or other languages to solve business problems.Becomes an expert in the business subject area the team are working on, holding expertise on both … team click. We’re a group that chooses to lift each other up and think smart every day.Extensive experience in SQL/Python/Pyspark and/or other languages relevant to data processingExtensive experience in dbt and ideally Airflow (or other orchestration tools)Extensive experience in Entity-Relationship ...

**Take the lead on meaningful, high-impact projects as a Senior Data Engineer.** We are seeking a Senior Data Engineer to join a growing organisation based in London, offering a hybrid work model. This role ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

Principal Consultant - Lead Data Analyst/Data Modeller Company: Transform Together Contract: 6-month Fixed Term Contract, with option to extend or become a permanent member of the team Compensation: OTE £90,000 Location: Hybrid ...

delivery of enterprise-scale data and AI platformsBuild cloud-native solutions using Databricks, Azure, AWS and GCPDevelop scalable ETL/ELT pipelines with Python, PySpark and SQLDefine engineering standards and mentor a team of Data & Platform EngineersWork on AI deployment, orchestration and monitoring in productionCollaborate with clients, architects, data … product teams to deliver high-impact solutionsWhat we're looking for:Strong experience building cloud-based data platformsExpertise with Python, SQL, Spark/PySpark and DatabricksExperience designing scalable data architectures and modern data pipelinesPrevious technical leadership or mentoring experienceAzure, AWS or GCP experienceWhat's in it for you Salary ...

adoption of modern data mesh principles across complex enterprise environments. What You'll Be Doing Architect and deliver modern data solutions using Databricks, Spark, PySpark and SQL Design scalable batch and real-time data pipelines that ingest data from legacy and modern platforms Lead the development of cloud-based … making Provide technical leadership and architectural direction on large-scale transformation programmes Essential Experience Strong hands-on expertise with Databricks Extensive experience using Spark, PySpark and SQL Proven track record designing and implementing data lake architectures Experience building both batch and streaming data pipelines Strong knowledge of Kafka Good ...

ability to contribute across the length and breadth of a projectProficiency in one or more of the following: Java/Spring Boot, Python/PySpark, GraphQL, or mobile development frameworks, with openness and aptitude to work across others as neededExperience with cloud platforms (preferably AWS) and working knowledge … knowledge of the financial services industry and their IT systemsPreferred qualifications, capabilities, and skillsExperience with high-volume batch and real-time data processing using PySpark or similar frameworks, including performance tuning and troubleshooting of large-scale data pipelinesIn-depth knowledge of relational and distributed databases (e.g., Oracle, Databricks) with ...

data processing solutions Work with modern data platforms such as Microsoft Fabric, Databricks, or similar technologies Build and optimise data solutions using Python, Spark, PySpark, and/or SQL Support data integration, transformation, and performance optimisation Collaborate with stakeholders to ensure high-quality, reliable data architecture Essential Skills & Experience … Hands-on experience with Microsoft Fabric, Databricks, or equivalent modern data platforms Strong coding experience in Python and/or Spark/PySpark Solid SQL skills and experience working with structured datasets Desirable Experience Knowledge of relational database design , normalisation , and data modelling Experience designing efficient, scalable data architectures ...

systems, including LLM-based applications and agentic architectures in regulated environments Proficiency in one or more programming languages, particularly Python and/or PySpark Advanced knowledge of software application development and technical processes in disciplines such as cloud, AI/ML, or data engineering Experience in large-scale data … analytical skills Preferred qualifications, capabilities, and skills Experience with modern data platforms such as Databricks or Snowflake Hands-on expertise with Spark/PySpark and big data processing technologies Knowledge of open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model ...

locks and idempotency patterns. Serverless Framework v3, Docker (container-based Lambdas), Terraform, Jenkins CI/CD. Apache Airflow DAG authoring and AWS Glue with PySpark for batch ETL. pytest and moto (offline AWS service mocking), Black formatter, ruff linter. Datadog APM tracing and structured JSON logging with Lambda Powertools. … infrastructure experience: Serverless Framework v3, Terraform, ECR, IAM. Experience deploying Docker container-based Lambda functions. Data engineering background with Airflow DAGs and PySpark/Glue ETL pipelines is an advantage. Multi-tenant SaaS platform experience at scale is desirable. Familiarity with telecom, OEM device management, or IoT platforms ...

environment Technical SkillsYou’ll have experience with some of the following:Cloud & Data Platforms: AWS, Azure, Google Cloud Platform, Databricks Programming Languages: Python, R, PySpark Agentic & GenAI Platforms: Microsoft Copilot Studio, OpenAI GPT-5 Agents, UiPath, Adept AI, Orby AI, Beam AI If you’ve held any of these … Specialist, Python Developer, AI Transformation Consultant, Databricks Consultant, Azure AI Consultant, AWS Data Scientist, GCP Data Consultant, Agentic AI Consultant, OpenAI GPT Specialist, PySpark Developer. ...

Staff Data Engineer | AWS | PySpark | Data Modernisation | London | Contract | Inside IR35 We're supporting a global, technology‐led organisation undertaking a large‐scale cloud data transformation programme. This is not a traditional data engineering role. The organisation is modernising hundreds of complex data pipelines, establishing reusable engineering patterns … culture. Responsibilities Defining technical standards and engineering patterns for a large‐scale data modernisation programme Re‐engineering complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality ...