25 of 25 Spark SQL Jobs

ResponsibilitiesImplement end-to-end Microsoft Fabric solutions based on approved architecture, including OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL) and Real-Time IntelligenceBuild medallion architectures (Bronze/Silver/Gold), including incremental loads, change data capture, slowly changing dimensions … error handlingSet up ingestion from diverse sources (SQL databases, APIs, files, SaaS applications, SAP, streaming sources) using pipelines, shortcuts and mirroringWrite clean, reusable and tested code in PySpark, Spark SQL, T-SQL and KQL, optimizing Delta tablesAdd data quality checks, validation rules ...

DescriptionRole – Senior ConsultantTechnology – Scala, Apache Spark, SQL, PythonNice to Have - AWS, UNIX shell scriptingLocation – London, UKContext and Job Description: Infosys provides Integration/middleware services for global clients covering a multitude of Integration technologies and products. The services are rendered typically following a global delivery model … fundamental Data Structures and their usage At least 5+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. Having expertise in Spark Core, Spark SQL and Spark Streaming. Experience with ...

scalable Data Engineering solutions. Experience of core MS Fabric features including: OneLake, Lakehouses, Warehouses, Data Pipelines, Dataflows, Gen2, Notebooks, Eventstreams + Eventhouses. Using SQL, T-SQL, Spark SQL, PySpark + Python for data ingestion, data transformation, data optimisation, data modelling, and converting ...

support decommission planning for legacy components Core Technical Skills Azure Synapse & Data Platform Mandatory hands-on expertise with: Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool) Azure Data Lake Storage Gen2 (ADLS Gen2) Delta Lake on Azure (Synapse Lakehouse patterns) Oracle Golden Gate Replication … SQL-driven generation from mapping documents) Databricks & Lakehouse Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred) Strong Apache Spark skills (PySpark/Spark SQL) Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

design scalable data models that enable reliable and consistent analytics. The ideal candidate will bring strong hands-on experience in data modeling, SQL, and data modeling tools such as ERwin, along with a solid understanding of healthcare revenue cycle processes and the ability to translate business requirements into … logic, business rules, metadata, and data lineage. Collaborate with data engineering teams to translate models into implementable data structures and integration workflows. Use SQL for data profiling, relationship validation, reconciliation, discrepancy analysis, and data quality assessment. Partner with business stakeholders to establish consistent definitions for revenue cycle metrics ...

hands-on Senior Databricks Solutions Architect to support a complex, high-performance data platform within a global organisation. Youll design and optimise Spark workloads, build production streaming … pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements: Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft ...

PySpark , and SQL/Spark SQL for building production-grade ETL/ELT pipelines Solid knowledge of Apache Spark , including DataFrames, joins, aggregations, partitioning, caching, and performance optimisation Experience working within Azure Databricks , including clusters, job clusters, notebooks, repositories, jobs, and Databricks ...

platforms,lakehousesand data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured … semi-structured and unstructured data sources. Migration and modernisation of legacy platforms such as Oracle, SQL Server/SSIS or Synapse. Power BI semantic models, curated datasets and supporting analytics platforms. Data quality, governance, metadata management and lineage. Data security, classification and access control approaches. Automated testing, validation ...

interfaces, Search) Evaluating the performance and applicability of multiple tools against customer requirements Technical Skills and expertise required Experience on ADLS, Azure SQL DB and Datawarehouse Strong working experience in Implementation of Azure cloud components using Azure Data Factory , Azure Data Analytics, Azure Data Lake, Azure Data Catalogue … data applications on cloud solutions on Azure. Hands on experience in performance tuning and optimizing code running in Databricks environment. Good understanding of SQL, T-SQL and/or PL/SQL Should have experience working in Agile projects with knowledge in Jira. Good ...

approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications)Additional InformationTech StackSQL Server, SSIS, T-SQLADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lakeSummaryType: Full-timeFunction: Analyst ...

clients. This is a hands-on, client-facing role for someone who enjoys getting stuck into the data. Key Responsibilities Write and run SQL to check, profile and reconcile data across source systems, staging, data warehouse and lakehouse layers Test ETL/ELT pipelines end to end, validating … than lagging behind Our QA tooling and processes improve because gaps are named and fixed, not just worked around Competencies & Behaviours Essential Strong SQL: confident writing joins, CTEs, aggregations and window functions to interrogate, profile and reconcile data Data pipeline testing: experience testing ETL/ELT processes ...

PySpark Developer Location: Irving, TX (Hybrid 3 Days Onsite) Experience: 7+ Years Job Type: Full-Time Required Skills: PySpark, Python, Apache Spark, SparkSQL, Hadoop, SQL, Unix/Linux, Shell Scripting, and Apache Airflow. Interested candidates, please DM me or share your updated resume. VITS Consulting ...

capacity (F-SKU) sizing and licensingDesign end-to-end Microsoft Fabric solutions: OneLake, Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Notebooks (PySpark/Spark SQL), Real-Time Intelligence (Eventstreams, Eventhouse/KQL) and Power BI + Power BI CoPilotDesign governance, security and operating frameworks: data governance … operating and support models and handover documentationPlan and run migrations from older platforms (Azure Synapse, Azure Data Factory, SSIS/SSAS, on-premises SQL Server, Snowflake, Databricks, Power BI Premium P-SKUs, Tableau/Qlik, etc.) to FabricPlan Fabric capacity (F-SKUs) and look after cost, performance ...

Databricks) Leeds (Hybrid) 6 Month Contract £(Apply online only)/day (Inside IR35) Data Engineer needed with strong hands-on Databricks, PySpark, AWS, SQL + Python experience. 6 Month Contract based in Leeds (Hybrid). Start ASAP in October 2026. Hybrid Working Model - primarily working from home with … work with a leading global IT transformation on a large-scale Government project: Data Engineer with strong Databricks, AWS, Python, PySpark + SQL skills. Hands-on Data Engineer to run + maintain production data pipelines on Databricks and AWS. Keeping scheduled data processing reliable, resolve incidents, fixes + ...

work with a leading global IT transformation on a large-scale Government project: Data Engineer with strong Databricks, AWS, Python, PySpark and SQL skills. Hands-on Data Engineer to run + maintain production data pipelines on Databricks and AWS. Keeping scheduled data processing reliable, resolve incidents, fixes + … role combines L2/L3 production support and Data Engineering tasks. Databricks experience covering notebooks, scheduled jobs, cluster configuration, driver/executor logs, Spark UI + performance troubleshooting. Supporting + maintaining production data pipelines, including incident investigation, safe recovery, root cause analysis + permanent remediation. Experience of Spark ...

your duties, you will be responsible for: Design, build, and deploy scalable data pipelines using Azure Databricks , Azure Data Factory , and Azure SQL Database . Optimise Databricks environments for performance, cost, and security. Develop and maintain data integration , ETL/ELT processes, and data quality frameworks. Translate business … Deep expertise in Azure Databricks , Azure Data Factory , Azure Synapse , and Azure Data Lake . Hands-on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working ...

your duties, you will be responsible for: Design, build and deploy scalable data pipelines using Azure Databricks , Azure Data Factory and Azure SQL Database . Optimise Databricks environments for performance, cost and security. Develop and maintain data integration , ETL/ELT processes and data quality frameworks. Translate retail … environments. Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse and Azure Data Lake. Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design and data pipeline orchestration. Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. Experience working with ...

data quality and governance. Create and maintain ETL/ELT processes using Data Factory, Data Pipelines, Notebooks, and Spark. Develop, maintain, and enhance SQL scripts, stored procedures, views, and data models. Monitor and troubleshoot performance issues within Fabric environments. Implement security, access controls, and governance standards. Support … architects, and stakeholders to translate requirements into technical solutions. Produce technical documentation and knowledge transfer materials. What you will have Technical Skills Strong SQL development skills. Experience designing and implementing ETL/ELT solutions. Knowledge of Lakehouse and Data Warehouse architectures. Experience with Data Pipelines, Dataflows Gen2 ...

data quality and governance. Create and maintain ETL/ELT processes using Data Factory, Data Pipelines, Notebooks, and Spark. Develop, maintain, and enhance SQL scripts, stored procedures, views, and data models. Monitor and troubleshoot performance issues within Fabric environments. Implement security, access controls, and governance standards. Support … architects, and stakeholders to translate requirements into technical solutions. Produce technical documentation and knowledge transfer materials. What you will have Technical Skills Strong SQL development skills. Experience designing and implementing ETL/ELT solutions. Knowledge of Lakehouse and Data Warehouse architectures. Experience with Data Pipelines, Dataflows Gen2 ...

data quality and governance. Create and maintain ETL/ELT processes using Data Factory, Data Pipelines, Notebooks, and Spark. Develop, maintain, and enhance SQL scripts, stored procedures, views, and data models. Monitor and troubleshoot performance issues within Fabric environments. Implement security, access controls, and governance standards. Support … architects, and stakeholders to translate requirements into technical solutions. Produce technical documentation and knowledge transfer materials. What you will have Technical Skills Strong SQL development skills. Experience designing and implementing ETL/ELT solutions. Knowledge of Lakehouse and Data Warehouse architectures. Experience with Data Pipelines, Dataflows Gen2 ...

deliver measurable value. Job Details End-to-end product life cycle Python Spark/PySpark SQL/MySQL/SparkSQL Tableau Databricks AWS Glue Linear Regression - GLM - Random Forrest GenAI/LLM No sponsorship or C2C Interested in hearing more? Easy Apply now by clicking ...