1 to 25 of 45 Spark SQL Jobs in the UK

Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems Develop and implement data solutions leveraging Microsoft Fabric, OneLake … providing sponsorship for this vacancy Minimum of 2 years of experience in Data Engineering and Big Data technologies Deep expertise in Apache Spark (PySpark/Spark SQL) Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models Strong SQL ...

Purview, Apache Atlas, Amundsen, DataHub or similar). Proven ability to build lineage for complex platforms: data lakes, warehouses, marts, and distributed processing (Spark‐based pipelines). Strong proficiency in SQL for tracing transformations and validating mappings across layers. Working knowledge of ETL/ELT patterns … data modeling (dimensional + normalized), and batch scheduling dependencies. Ability to interpret data transformation logic from pipelines (Spark SQL/PySpark/Hive queries/orchestration configs). Strong documentation capability: source‐to‐target mappings, lineage diagrams, data dictionaries, metadata standards, and control evidence packs. Technical ...

technical SME across Databricks, Fabric, Synapse, and Azure. Own solution design, implementation strategy, and technical delivery. Build and optimise pipelines using PySpark , Spark SQL , ADF , Synapse Pipelines , or Fabric Pipelines . Define engineering best practice, conduct code reviews, and uplift quality. Develop secure, performant SQL … bring Proven leadership of Data Engineering teams. Strong commercial experience with Databricks , Microsoft Fabric , or Azure Synapse . Deep expertise in PySpark , Spark SQL , and SQL optimisation. ETL/ELT pipeline development and orchestration. Experience building scalable cloud-based data platforms. Nice to Have ...

department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis, and documentation … efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks ...

bonus and an extensive benefits package. What you'll be doingDesign, build and maintain scalable data pipelines and data products using Databricks, PySpark, Spark SQL and Delta Lake.Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems.Shape enterprise … help mature a growing data function.What you need to bring“Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc)Advanced SQL and Python coding skills.A track record of building and optimising enterprise-scale data pipelines ...

bonus and an extensive benefits package. What you'll be doingDesign, build and maintain scalable data pipelines and data products using Databricks, PySpark, Spark SQL and Delta Lake.Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems.Shape enterprise … help mature a growing data function.What you need to bring“Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc)Advanced SQL and Python coding skills.A track record of building and optimising enterprise-scale data pipelines ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

analytics roadmap (S/4HANA Embedded Analytics, SAP Datasphere, SAP Analytics Cloud). Lead pilot projects and migrations (e.g., Cognos to Power BI, SQL DW to Fabric OneLake). Collaborate with stakeholders across all business levels and our Accenture partners. Education & Qualifications Bachelor's degree in Computer Science … Data Factory, Synapse, and cloud data services. Expert‐level skills in data modelling, ETL processes, and semantic layer design. Advanced programming capabilities in SQL (T‐SQL/Spark SQL), Python, and PySpark. Hands‐on experience with Power BI (Desktop and Service ...

Formation - Medallion Architecture based ETL processes using S3, Iceberg and Athena - Azure DevOps/AWS CodeCommit/AWS CodePipeline - AWS Step functions - Athena SQL ; Spark SQL ; Python - AWS CLI - Release management (CI/CD) - Knowledge and Experience - desirable - AWS Databrew - AWS RDS/Redshift ...

transformation of its data estate. As part of a wider modernisation programme, the business is migrating critical real estate data capabilities from legacy SQL Server environments into a modern Databricks platform, creating a more scalable, governed and future-ready foundation for analytics, reporting and investment decision-making. This … platform architecture, data governance and reporting capabilities across a complex investment environment. What You'll Do Lead the assessment and modernisation of legacy SQL Server, ETL and reporting workloads into a Databricks-based architecture. Design scalable data models, Delta Lake structures and transformation pipelines supporting real estate investment ...

transformation of its data estate. As part of a wider modernisation programme, the business is migrating critical real estate data capabilities from legacy SQL Server environments into a modern Databricks platform, creating a more scalable, governed and future-ready foundation for analytics, reporting and investment decision-making. This … platform architecture, data governance and reporting capabilities across a complex investment environment. What You'll Do Lead the assessment and modernisation of legacy SQL Server, ETL and reporting workloads into a Databricks-based architecture. Design scalable data models, Delta Lake structures and transformation pipelines supporting real estate investment ...

data management, information systems industry approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications) Additional Information Tech Stack SQL Server, SSIS, T-SQL ADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lake #J ...

technical stakeholders, including customers, sales, and product teams Experience with deep learning statistical models such as Transformers B2B product experience desirable Experience with Spark SQL, PySpark, and Databricks desirable Experience with CausalAI modelling frameworks such as DoWhy and EconML desirable Core Competencies Demonstrates expertise in designing … Science Techniques Behavioural Insights Soft Skills Creative Problem Solving Collaboration Communication Industry Keywords B2B Product Experience Agentic AI Behavioural Data Science Tools & Technologies Spark SQL PySpark Databricks #J-18808-Ljbffr ...

this role is to deliver new, and implement changes to existing data solutions, data extraction and transform (ETL), using the full Microsoft SQL stack, for identified and agreed business requirements. In addition, the Senior Data Engineer will be responsible for translating business requirements into Azure Cloud-based … Azure SQL, Azure Data Lake, Datra bricks, Purview and Azure App Service is essential Proficiency in Python/SQL/SparkSQL/PySpark is essential Strong experience of Azure delivery experience including analysis/development/configuration/deployments essential Expert level technical skill ...

techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL.Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution design … building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologiesProficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability.Designing and building solutions on a cloud data warehouse or lakehouse ...

deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL. Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution … building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologies Proficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability. Designing and building solutions on a cloud data warehouse ...

cleansed, conformed business layers) while securely handling identifiable data.Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark.Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across …/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats.Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications)Additional InformationTech StackSQL Server, SSIS, T-SQLADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lakeSummaryType: Full-timeFunction: Analyst ...

Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing ...

middleware, stabilise it, and begin replacing it with a cleaner, sustainable solution built around modern data tooling (Microsoft Fabric, Databricks, or equivalent, plus SQL and Power BI). Phase 2 (ongoing): Expand to include reporting. Once the core system is stable, the role broadens into wider business reporting … gather requirements from senior stakeholders across the business. Simultaneous: Bring Data Hub and Data Warehouse closer in design standards. Must-haves: Strong SQL Microsoft Fabric, Databricks, Synapse, or similar modern data platform experience Power BI (building and owning reports/dashboards, not just consuming) Comfortable working with ambiguous ...

Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into … need from you 8+ years in data engineering, including 3+ years hands on with Azure Databricks Strong Python and PySpark, backed by advanced SQL Deep grounding in data warehousing, data modelling and integration patterns Unity Catalog and Delta Lake in a production environment Data quality and governance experience ...