1 to 25 of 44 Spark SQL Jobs in London

platform. Optimise data pipelines for performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure … Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need to program using the languages such as SQL, Python, R, YAML and JavaScript Data Integration: Integrate data from various sources … practices. Essential Skills & Experience: 10+ years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data Factory, Azure Blob Storage More ❯

platform. Optimise data pipelines for performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure … Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need to program using the languages such as SQL, Python, R, YAML and JavaScript Data Integration: Integrate data from various sources … practices. Essential Skills & Experience: 10+ years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data Factory, Azure Blob Storage More ❯

comprehensive logging, monitoring, and alerting tools to manage the platform, ensuring resilience and optimal performance are maintained. Data Integration and Transformation Integrate and transform data from multiple organisational SQL databases and SaaS applications using end-to-end dependency-based data pipelines, to establish an enterprise source of truth. Create ETL and ELT processes using Azure Databricks, ensuring audit … ready financial data pipelines and secure data exchange with Databricks Delta Sharing and SQL Warehouse endpoints. Governance and Compliance Ensure compliance with information security standards in our highly regulated financial landscape by implementing Databricks Unity Catalog for governance, data quality monitoring, and ADLS Gen2 encryption for audit compliance. Development and Process Improvement Evaluate requirements, create technical design documentation … Metadata Driven Platform, Event-driven architecture. Proven experience of ETL/ELT, including Lakehouse, Pipeline Design, Batch/Stream processing. Strong working knowledge of programming languages, including Python, SQL, PowerShell, PySpark, Spark SQL. Good working knowledge of data warehouse and data mart architectures. Good experience in Data Governance, including Unity Catalog, Metadata Management, Data Lineage, Quality More ❯

ensure high availability and accessibility Requirements Minimum of 8 years of experience in data engineering At least 5 years of hands-on experience with Azure data services (Apache Spark, Azure Data … Factory, Synapse Analytics, RDBMS such as SQL Server) Proven leadership and management experience in data engineering teams Proficiency in PySpark, Python (with Pandas), T-SQL, SparkSQL, and experience with CI/CD pipelines Strong understanding of data modeling, ETL processes, and data warehousing concepts Knowledge of version control systems like Git Excellent problem-solving, analytical, communication … manage multiple projects and meet deadlines Azure certifications such as Microsoft Certified: Azure Data Engineer Associate or Azure Solutions Architect Nice to Have Experience with Scala for Apache Spark Knowledge of other cloud platforms like AWS or GCP Our Benefits Include Group pension plan, life assurance, income protection, and critical illness cover Private medical insurance and dental care More ❯

ideas and solutions to improve our data infrastructure and capabilities. What is needed to succeed: Technical skills: Problem-solving team player with an analytical mind. Strong knowledge of SQL and Spark SQL. Understanding of dimensional data modelling concepts. Experience with Azure Synapse Analytics. Understanding of streaming data ingestion processes. Ability to develop/manage Apache Spark More ❯

Design and deliver quality solutions independently • Leading a team of Data Engineers and deliver solutions as a team Key skills/knowledge/experience: • Proficient in PySpark, Python, SQL with atleast 5 years of experience • Working experience in Palantir Foundry platform is must • Experience designing and implementing data analytics solutions on enterprise data platforms and distributed computing (Spark … customer requirements into a best-fit design and architecture. • Demonstrated experience in end-to-end data management, data modelling, and data transformation for analytical use cases. • Proficient in SQL (Spark SQL preferred). • Experience with JavaScript/HTML/CSS a plus. Experience working in a Cloud environment such as Azure or AWS is More ❯

experience - Experience with data modeling, warehousing and building ETL pipelines - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) - Experience with one or more scripting language (e.g., Python, KornShell) PREFERRED QUALIFICATIONS - Experience with big data technologies such as: Hadoop, Hive, Spark, EMR - Experience with any ETL More ❯

be part of a team that’s transforming how data powers retail, this is your opportunity. Your Role (Key Responsibilities) Design, build, and optimise robust data pipelines using PySpark, SparkSQL, and Databricks to ingest, transform, and enrich data from a variety of sources. Translate business requirements into scalable and performant data engineering solutions, working closely with squad members and stakeholders. … Community, ensuring standards, practices and principles are built into everything you do. About You (Experience & Qualifications) Experience building and maintaining data solutions. Experience with languages such as Python, SQL and, preferably, PySpark. Hands-on experience with Microsoft Azure data services, including Azure Data Factory, and a good understanding of cloud-based data architectures (e.g., data lakes, data vaults More ❯

lead projects on your own. Your qualifications and experience You are a pro at using SQL for data manipulation (at least one of PostgreSQL, MSSQL, Google BigQuery, SparkSQL) Modelling & Statistical Analysis experience, ideally customer related Coding skills in at least one of Python, R, Scala, C, Java or JS Track record of using data manipulation and machine learning … libraries in one or more programming languages. Keen interest in some of the following areas: Big Data Analytics (e.g. Google BigQuery/BigTable, Apache Spark), Parallel Computing (e.g. Apache Spark, Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Data Storytelling with (Jupyter) Notebooks, Graph Computing, Microservices Architectures Modelling & Statistical Analysis experience, ideally customer related More ❯

teams to extract, transform, and load data from a wide variety of data sources Implement data structures using best practices in data modeling, ETL/ELT processes, and SQL, Redshift. Model data and metadata for ad-hoc and pre-built reporting Interface with business customers, gathering requirements and delivering complete reporting solutions Build robust and scalable data integration … ETL) pipelines using SQL, Python and Spark. Build and deliver high quality data sets to support business analyst, data scientists, and customer reporting needs. Continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers Participate in strategic & tactical planning discussions A day in the life As a Data Engineer, you will be working … - Experience with data modeling, warehousing and building ETL pipelines - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) - Experience with one or more scripting language (e.g., Python, KornShell) PREFERRED QUALIFICATIONS - Experience with big data technologies such as: Hadoop, Hive, Spark, EMR - Experience with any ETL More ❯

and WW) use cases on our data platform 5. Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL, Amazon and AWS big data technologies 6. Must possess strong verbal and written communication skills, be self-driven, and deliver high quality results in a fast-paced environment. 7. …/dash-boarding d) empowering business with self-service tools to manage data and generate insights. BASIC QUALIFICATIONS - 1+ years of data engineering experience - Experience writing and optimizing SQL queries with large-scale, complex datasets - Experience with data modeling, warehousing and building ETL pipelines … Experience with one or more scripting language (e.g., Python, KornShell) - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) PREFERRED QUALIFICATIONS - Experience with big data technologies such as: Hadoop, Hive, Spark, EMR - Experience with any ETL tool like, Informatica, ODI, SSIS, BODI, Datastage, etc. Our inclusive More ❯

service support for datasets. • Triage many possible courses of action in a high-ambiguity environment, making use of both quantitative analysis and business judgment. BASIC QUALIFICATIONS - Experience with SQL - 1+ years of data engineering … experience - Experience with data modeling, warehousing and building ETL pipelines - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) - Experience with one or more scripting language (e.g., Python, KornShell) PREFERRED QUALIFICATIONS - Experience with big data technologies such as: Hadoop, Hive, Spark, EMR - Experience with any ETL More ❯

development lifecycle understanding Agile methodologies experience Experience with CI/CD tools Key Technologies: Azure Databricks, Data Factory, Storage, Key Vault Source control experience (e.g., Git) Proficiency in SQL (Spark SQL) and Python (PySpark) Certifications: Microsoft Certified: Azure Fundamentals (AZ-900), Azure Data Fundamentals (DP-900) Curiosity about technology and adaptability Agile mindset, optimism More ❯

Agile methodologies Experience working with CI/CD tools Key Technology: Experience with source control systems, such as Git dbt (Data Build Tool) for transforming and modelling data SQL (Spark SQL) & Python (PySpark) Certifications: You will need to be you. Curious about technology and adaptable to new technologies Agile-minded, optimistic, passionate, and pragmatic More ❯

years of data engineering experience. Experience with data modeling, warehousing, and building ETL pipelines. Proficiency in query languages such as SQL, PL/SQL, HiveQL, SparkSQL, or Scala. Experience with scripting languages like Python or KornShell. Knowledge of writing and optimizing SQL queries for large-scale, complex datasets. Experience with big data technologies such … as Hadoop, Hive, Spark, EMR. Experience with ETL tools like Informatica, ODI, SSIS, BODI, or DataStage. Our inclusive culture empowers Amazon employees to deliver the best results for our customers. If you have a disability and need workplace accommodations during the application and hiring process, please visit this link for more information. If your country or region isn More ❯

years of data engineering experience. Experience with data modeling, warehousing, and building ETL pipelines. Experience with query languages such as SQL, PL/SQL, HiveQL, SparkSQL, or Scala. Experience with scripting languages like Python or KornShell. Knowledge of writing and optimizing SQL queries for large-scale, complex datasets. PREFERRED QUALIFICATIONS Experience with big data … technologies such as Hadoop, Hive, Spark, EMR. Experience with ETL tools like Informatica, ODI, SSIS, BODI, or DataStage. We promote an inclusive culture that empowers Amazon employees to deliver the best results for our customers. If you have a disability and require workplace accommodations during the application, hiring, interview, or onboarding process, please visit this link for more More ❯

analysis, testing, and release management. Understanding of Agile methodologies, CI/CD tools, and full software development lifecycle. Proficiency with Azure Databricks, Data Factory, Storage, Key Vault, Git, SQL (Spark SQL), and Python (PySpark). Certifications: Azure Fundamentals (AZ-900), Azure Data Fundamentals (DP-900). Curiosity about technology, adaptability, agility, and a collaborative More ❯

expertise/knowledge in the following (in rough priority order): AWS Kubernetes (EKS) Data/network security Python Docker Grafana Postgres CDC systems Data related products (airflow, jupyter, spark, etc) The projects will be varied and we’re looking for someone who can work autonomously and proactively to scope problems and solve and deliver pragmatic solutions Our Tech … platform Parquet and Delta file formats on S3 for data lake storage Postgres/aurora for our relational databases Spark for data processing dbt for data modelling SparkSQL for analytics Streamlit for data applications Why else you'll love it here Wondering what the salary for this role is? Just ask us! On a call with one of More ❯

AWS. Build out real-world architectures that business engineering teams buy into and build their applications around. Required Qualifications, Capabilities, and Skills: Experience across the data lifecycle with Spark-based frameworks for end-to-end ETL, ELT & reporting solutions using key components like Spark SQL & Spark Streaming. Strong knowledge of multi-threading … mindset geared towards a fantastic end-to-end engineering experience supported by excellent tooling and automation. Preferred Qualifications, Capabilities, and Skills: Good understanding of the Big Data stack (Spark/Iceberg). Ability to learn new technologies and patterns on the job and apply them effectively. Good understanding of established patterns, such as stability patterns/anti-patterns More ❯

working with large scale of data. Key job responsibilities • Own the design, development, and maintenance of last mile data sets • Manipulate/mine data from database tables (Redshift, Apache Spark SQL) • Conduct deep dive investigations into issues related to incorrect and missing data • Identify and adopt best practices in developing data pipelines and tables: data integrity … experience - Experience with data modeling, warehousing and building ETL pipelines - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) - Experience with one or more scripting language (e.g., Python, KornShell) - Speak, write, and read fluently in Japanese - Speak, write, and read fluently in English PREFERRED QUALIFICATIONS - Experience with big … data technologies such as: Hadoop, Hive, Spark, EMR - Experience with any ETL tool like, Informatica, ODI, SSIS, BODI, Datastage, etc. Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding More ❯

autonomy to lead, space to innovate, and a team who values support, balance and shared success. Technologies EON Next loves: Languages: Python, SQL & Spark (PySpark, SparkSQL) Tools: Git (GitHub/GitLab) DataBricks, AWS services (inc. Athena/Glue, S3, SageMaker, Lambda, Transcribe, EC2) & Tableau Datastores: Postgres, DataBricks, Parquet/Delta Here's what else you need More ❯

autonomy to lead, space to innovate, and a team who values support, balance and shared success. Technologies EON Next loves: Languages: Python, SQL & Spark (PySpark, SparkSQL) Tools: Git (GitHub/GitLab) DataBricks, AWS services (inc. Athena/Glue, S3, SageMaker, Lambda, Transcribe, EC2) & Tableau Datastores: Postgres, DataBricks, Parquet/Delta Here's what else you need More ❯

the data infrastructure and maintain a highly scalable, reliable and efficient data system to support the fast growing business. You will work with analytic tools, can write excellent SQL scripts, optimize performance of SQL queries and can partner with internal customers to answer key business questions. We look for candidates who are self-motivated, flexible, hardworking … experience - Experience with data modeling, warehousing and building ETL pipelines - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) - Experience with one or more scripting language (e.g., Python, KornShell) - Knowledge of AWS Infrastructure - Knowledge of writing and optimizing SQL queries in a business environment with large More ❯

the data infrastructure and maintain a highly scalable, reliable and efficient data system to support the fast growing business. You will work with analytic tools, can write excellent SQL scripts, optimize performance of SQL queries and can partner with internal customers to answer key business questions. We look for candidates who are self-motivated, flexible, hardworking … experience - Experience with data modeling, warehousing and building ETL pipelines - Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala) - Experience with one or more scripting language (e.g., Python, KornShell) - Knowledge of AWS Infrastructure - Knowledge of writing and optimizing SQL queries in a business environment with large More ❯

requires security clearance at SC level. You must be SC cleared to be considered for the role. Tasks and Responsibilities: Engineering: Ingestion configuration. Write python/pyspark and spark SQL code for validation/curation in notebook. Create data integration test cases. Implement or amend worker pipelines. Implement Data validation/curation rules. Convert data model … App, AKS, Azure App Service, Web App. Good knowledge in real-time streaming applications, preferably with experience in Kafka real-time messaging or Azure Functions, Azure Service Bus. Spark processing and performance tuning. File formats partitioning (e.g., Parquet, JSON, XML, CSV). Azure DevOps/GitHub. Hands-on experience in at least one of Python with knowledge of More ❯