11 of 11 Spark SQL Jobs in London

technical SME across Databricks, Fabric, Synapse, and Azure. Own solution design, implementation strategy, and technical delivery. Build and optimise pipelines using PySpark , Spark SQL , ADF , Synapse Pipelines , or Fabric Pipelines . Define engineering best practice, conduct code reviews, and uplift quality. Develop secure, performant SQL … bring Proven leadership of Data Engineering teams. Strong commercial experience with Databricks , Microsoft Fabric , or Azure Synapse . Deep expertise in PySpark , Spark SQL , and SQL optimisation. ETL/ELT pipeline development and orchestration. Experience building scalable cloud-based data platforms. Nice to Have ...

department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis, and documentation … efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

transformation of its data estate. As part of a wider modernisation programme, the business is migrating critical real estate data capabilities from legacy SQL Server environments into a modern Databricks platform, creating a more scalable, governed and future-ready foundation for analytics, reporting and investment decision-making. This … platform architecture, data governance and reporting capabilities across a complex investment environment. What You'll Do Lead the assessment and modernisation of legacy SQL Server, ETL and reporting workloads into a Databricks-based architecture. Design scalable data models, Delta Lake structures and transformation pipelines supporting real estate investment ...

cost-effectiveness. o Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: o Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. o Develop and maintain data processing logic for cleaning, enriching, and aggregating data. o Ensure data consistency … accuracy throughout the data life cycle. Azure Databricks Implementation: o Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. o Implement best practices for Databricks development and deployment. o Optimise Databricks workloads for performance and cost. o Need ...

Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing ...

interested in engineers with strong hands-on Databricks experience. Essential Skills Commercial experience building solutions within Databricks. Strong hands-on expertise with PySpark, Spark SQL, and Delta Lake. Experience designing and supporting production-grade ETL/ELT pipelines. Strong Python and SQL development skills. ...

experience designing canonical data models or enterprise logical data models. Strong understanding of data modelling techniques, including conceptual, logical and physical modelling. Strong SQL skills and ability to interrogate complex relational data structures. Experience working with cloud data platforms, ideally AWS and Databricks. Understanding of modern data architecture … platforms. Experience with regulatory, client or operational reporting. Experience supporting data migration, legacy system modernisation or platform replacement. Experience with Databricks, Delta Lake, Spark SQL or PySpark. Experience defining validation frameworks, data quality dashboards or metadata-driven rule engines. Experience working in consultancy or client-facing ...

your duties, you will be responsible for: Design, build, and deploy scalable data pipelines using Azure Databricks , Azure Data Factory , and Azure SQL Database . Optimise Databricks environments for performance, cost, and security. Develop and maintain data integration , ETL/ELT processes, and data quality frameworks. Translate business … environments. 8–10 years’ experience in data engineering and integration , ideally in enterprise‐scale environments. Hands‐on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working ...

drive to make a positive impact on the world Experience working within a finance team and senior stakeholders Proficiency in writing robust, performant SQL queries Some experience building robust data pipelines, using Python and dbt Strong analytical skills, commercial interest, and the ability to distinguish between what matters … tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here Wondering what the salary for this role is Just ask us! On a call with ...

experience in data/network security would be a nice-to-have Datadog/Grafana/Prometheus Data related products (Airflow, Jupyter, Spark, dbt etc) The projects will be varied and we’re looking for someone who can work autonomously and proactively to scope problems and solve … formats on S3 for data lake storage Postgres/Aurora for our relational databases Spark for data processing dbt for data modelling SparkSQL for analytics Streamlit for data applications Are you ready for a career with us? We want to ensure you have the right tools and environment ...