25 of 25 Permanent Spark SQL Jobs

department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis, and documentation … efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks ...

Azure | Databricks | SQL | Outside IR35 | Hybrid THE COMPANY We're partnering with a global mining organisation that is investing heavily in the evolution of its enterprise data platform. Following significant growth and a wider transformation programme, they're looking to bring in an experienced Senior Data Engineer … stakeholders, understand technical and commercial requirements and translate them into scalable Azure data solutions. The platform is built around Azure and Databricks, with SQL sitting at the centre of the engineering approach. The team strongly believes in solving transformation challenges through Spark SQL wherever ...

performance, cost, reliability, and maintainability. What you'll do Design, build, and operate production ETL pipelines on Databricks using Delta Live Tables and Spark, applying medallion architecture patterns across Finance, Payments, Product, and GTM domains. Own data modeling in different layers — defining governed, reusable table structures that serve … maintaining production pipelines and data models. Deep proficiency with Databricks or similar platforms — Delta Live Tables, Unity Catalog, Delta Lake, and PySpark or Spark SQL as primary tools. Strong SQL skills: complex transformations, incremental materialization, schema evolution, and data quality validation. Solid AWS experience ...

approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications)Additional InformationTech StackSQL Server, SSIS, T-SQLADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lakeSummaryType: Full-timeFunction: Analyst ...

Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing ...

acquire & transform raw data (bronze) into high-quality, BI-ready models (gold), ensure data quality and lineage, and enable fast, secure analytics through SQL warehouses and Unity Catalogues.The ideal candidate will work closely with key business stakeholders and our SI partners to gather requirements, optimise business processes … scalable solutions that drive business value.What does this role involve day to day Data Engineering· Design and implement batch/streaming pipelines (PySpark, Spark SQL, DLT, Auto Loader) across bronze, silver & gold layers.·Implement CDC patterns, incremental loads, tune partitioning, Z Order, OPTIMIZE/VACUUM.· Build ...

real-time pipelines powering trading decisions, risk management, and market analytics.Key Responsibilities Data Architecture: Build and optimize ETL/ELT pipelines using Databricks, Spark (PySpark/Scala), and Delta Lake for structured/unstructured energy market data.Trading Analytics: Partner with Quants, Risk Managers, and Traders to implement models … curve generation, position tracking, and risk metrics (e.g., VaR). Performance Optimization: Tune clusters and SQL warehouses for low-latency, time-sensitive trading environments. Data Governance: Enforce governance, access controls, and compliance with UK/EU financial and energy regulations via Unity Catalog. Consultative Leadership: Provide architectural best ...

with modern cloud data platforms (e.g. Databricks or Snowflake), ideally within AWS environments. Advanced programming and data engineering skills in Python, PySpark and SQL, with a solid understanding of ETL pipelines and data transformation. Proven experience in data modelling (end-to-end), including performance optimisation using tools such ...

experience designing canonical data models or enterprise logical data models. Strong understanding of data modelling techniques, including conceptual, logical and physical modelling. Strong SQL skills and ability to interrogate complex relational data structures. Experience working with cloud data platforms, ideally AWS and Databricks. Understanding of modern data architecture … platforms. Experience with regulatory, client or operational reporting. Experience supporting data migration, legacy system modernisation or platform replacement. Experience with Databricks, Delta Lake, Spark SQL or PySpark. Experience defining validation frameworks, data quality dashboards or metadata-driven rule engines. Experience working in consultancy or client-facing ...

years Location: United Kingdom - Glasgow, London Skills: Apache Airflow, AWS, Data Lake/Lakehouse, Databricks Certified (Data Engineer, ML), DBT, Python, SQL We are seeking a highly skilled Сonsultant Data Engineer to be the driving force behind this massive architectural shift. You will be hands‐on in moving … Databricks. Requirements Required Skills & Technologies Cloud Platforms: Hands‐on engineering experience within the AWS ecosystem (S3, IAM, Glue) and deep expertise in Databricks (SparkSQL, PySpark, governed cataloging). Modern Data Stack: Strong proficiency in dbt (Core or Cloud) for data transformation and understanding of modern table formats, specifically Apache Iceberg. ...

years Location United Kingdom - Glasgow, London Skills Apache Airflow, AWS, Data Lake/Lakehouse, Databricks Certified (Data Engineer, ML), DBT, Python, SQL We are seeking a highly skilled Сonsultant Data Engineer to be the driving force behind this massive architectural shift. You will be hands-on in moving … Databricks. Requirements Required Skills & Technologies Cloud Platforms: Hands-on engineering experience within the AWS ecosystem (S3, IAM, Glue) and deep expertise in Databricks (SparkSQL, PySpark, governed cataloging). Modern Data Stack: Strong proficiency in dbt (Core or Cloud) for data transformation and understanding of modern table formats, specifically Apache Iceberg. ...

emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building ...

Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

operate Databricks pipelines using Python, Lakeflow Declarative Pipelines, and Jobs/Workflows Use Azure Data Factory for ingestion/orchestration; integrate and optimize SQL/NoSQL databases Ensure pipeline quality via versioning, testing, monitoring, SLAs/SLOs, and automated deployments Manage platform costs and pipeline performance. Support establishment … group-wide use Profil Degree in Computer Science, Software Engineering, Data Engineering/Data Science, or equivalent experience Hands-on with Databricks (Python, Spark SQL, Lakehouse), Declarative Pipelines, and Jobs/Workflows Experience with Azure Data Factory, data modeling, performance tuning, and production data systems DataOps ...

columnar databases. Experience in data integration with ETL techniques and frameworks Experience in Big Data querying tools, such as Hive, Impala and Spark SQL Work Schedule: MULTIPLE SHIFTS AVAILABLE Work Type: Full Time ...

Data Engineer to join our Debt Solutions team and help shape the future of data at DWP. You'll need experience of SQL-based data transformation, ETL pipeline development, and building user-facing dashboards and reports Our DWP Digital teams use fresh ideas and leading-edge tech … Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Data Engineer to join our Debt Solutions team and help shape the future of data at DWP. You'll need experience of SQL-based data transformation, ETL pipeline development, and building user-facing dashboards and reports Our DWP Digital teams use fresh ideas and leading-edge tech … Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Data Engineer to join our Debt Solutions team and help shape the future of data at DWP. You'll need experience of SQL-based data transformation, ETL pipeline development, and building user-facing dashboards and reports Our DWP Digital teams use fresh ideas and leading-edge tech … Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Data Engineer to join our Debt Solutions team and help shape the future of data at DWP. You'll need experience of SQL-based data transformation, ETL pipeline development, and building user-facing dashboards and reports Our DWP Digital teams use fresh ideas and leading-edge tech … Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

Data Engineer to join our Debt Solutions team and help shape the future of data at DWP. You'll need experience of SQL-based data transformation, ETL pipeline development, and building user-facing dashboards and reports Our DWP Digital teams use fresh ideas and leading-edge tech … Ability to optimise data processing, storage, and queries to improve performance and data quality. Experience designing and automating ETL pipelines using PySpark and Spark SQL.A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java. You and your role ...

drive to make a positive impact on the world Experience working within a finance team and senior stakeholders Proficiency in writing robust, performant SQL queries Some experience building robust data pipelines, using Python and dbt Strong analytical skills, commercial interest, and the ability to distinguish between what matters … tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here Wondering what the salary for this role is Just ask us! On a call with ...