1 to 25 of 56 Permanent Spark SQL Jobs

using Microsoft Fabric • Build scalable data pipelines using Dataflow Gen2 and Fabric Notebooks • Develop and maintain Lakehouse solutions using Medallion architecture • Work with SQL, Python, Spark SQL and Delta Lake • Integrate data from APIs, databases, CRM, ERP and third party platforms • Support both batch … wider business stakeholders Essential experience • Strong commercial experience with Microsoft Fabric • Hands on experience with Dataflow Gen2, Fabric Notebooks and Lakehouse architecture • Strong SQL and Python skills • Experience with Spark and Spark SQL • Proven experience designing scalable ETL and ELT pipelines • Data ...

Purview, Apache Atlas, Amundsen, DataHub or similar). Proven ability to build lineage for complex platforms: data lakes, warehouses, marts, and distributed processing (Spark‐based pipelines). Strong proficiency in SQL for tracing transformations and validating mappings across layers. Working knowledge of ETL/ELT patterns … data modeling (dimensional + normalized), and batch scheduling dependencies. Ability to interpret data transformation logic from pipelines (Spark SQL/PySpark/Hive queries/orchestration configs). Strong documentation capability: source‐to‐target mappings, lineage diagrams, data dictionaries, metadata standards, and control evidence packs. Technical ...

reporting, analytics, and operational decision-making across the project programme. Working within a multidisciplinary Data & Analytics team, the successful candidate will use Databricks, SQL, Python, and Microsoft technologies to deliver robust, scalable, and maintainable data solutions. They will collaborate closely with Data Architects and Performance Analysts to transform … business intelligence solutions. Strong proven commercial experience with Databricks. Strong experience developing ETL/ELT pipelines and data integration solutions. Experience using Apache Spark, PySpark, and Spark SQL. Strong SQL development and optimisation skills. Experience working with large and complex datasets from multiple source ...

Title: Senior Apache Spark Developer Location: Jersey city ,NJ (Only Local candidates who can come for in person interview ) Work Arrangement: Hybrid Client Industry: Banking Duration: 12 Months Contract Key Focus Area Spark SQL, Structured Streaming , Apache Kafka, AMPS About the Role … hiring an Senior Apache Spark Developer Required Qualifications 5+ years of hands-on development experience with Apache Spark. Strong expertise in: Spark SQL Structured Streaming DataFrames/Datasets Performance tuning and optimization Experience integrating Spark with Apache Kafka. Strong development skills ...

department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis, and documentation … efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks ...

documentation, and deployment of configuration changes Develop and support integrations using Node.js Support related data integration work using Python, AWS Glue, PySpark, and Spark SQL as needed Support production issues as needed Primary Requirements: Approximately 3-4+ years of relevant experience Strong hands-on experience … code configuration and API support Desired Skills: Direct Newgen platform experience Healthcare provider-domain experience Experience with Python, AWS Glue, PySpark, Spark, and Spark SQL If you have the described qualifications and are interested in this exciting opportunity, please apply! Ranked a Top Staffing ...

hands-on architecture role combining technical leadership, solution design, stakeholder engagement, and delivery oversight. Essential Technical Skills Data Integration & Development Python PySpark SQL and T-SQL Spark SQL XML XSD (XML Schema Definition) XPath XSLT JSON REST APIs Data warehousing methodologies Master … Strong stakeholder engagement and consultancy experience. Candidate Profile Summary The ideal candidate will possess deep expertise in Azure Data Factory, Azure Databricks, PySpark, SQL, XML, XSD, XPath and XSLT, with a proven track record of architecting and delivering enterprise-scale ETL/ELT solutions across hybrid and cloud ...

strategic asset that's as beautiful as it is actionable. If you dream in data layers and speak the languages of both SQL and stakeholder, this is your stage. Who We Are: The Architects of What Connects Us You find the signal in the noise of data. … engineering teams monitor, measure, and optimize the performance of the very networks we design. Automate Intelligence: Unleash the power of Tableau, Alteryx, SQL, and Python/R to automate complex analysis, transforming raw data streams into a wellspring of proactive, business-ready insights-freeing our minds from manual ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data ...

knowledge‐sharing sessions to uplift team capability. Proven experience working as an Analytics Engineer, Data Engineer, or similar role. Deep proficiency in SQL for data transformation and analysis. Strong hands‐on experience with dbt (Data Build Tool) on Databricks, including advanced modelling techniques, test writing, and performance optimisation. … projects. Tools & Technologies You’ll Work With: Databricks (primary data platform for modelling and transformation) dbt for data modelling and transformation on Databricks SQL (Databricks SQL/Spark SQL) VS Code UI for SQL and dbt coding Git/GitHub ...

analytics roadmap (S/4HANA Embedded Analytics, SAP Datasphere, SAP Analytics Cloud). Lead pilot projects and migrations (e.g., Cognos to Power BI, SQL DW to Fabric OneLake). Collaborate with stakeholders across all business levels and our Accenture partners. Education & Qualifications Bachelor's degree in Computer Science … Data Factory, Synapse, and cloud data services. Expert‐level skills in data modelling, ETL processes, and semantic layer design. Advanced programming capabilities in SQL (T‐SQL/Spark SQL), Python, and PySpark. Hands‐on experience with Power BI (Desktop and Service ...

Formation - Medallion Architecture based ETL processes using S3, Iceberg and Athena - Azure DevOps/AWS CodeCommit/AWS CodePipeline - AWS Step functions - Athena SQL ; Spark SQL ; Python - AWS CLI - Release management (CI/CD) - Knowledge and Experience - desirable - AWS Databrew - AWS RDS/Redshift ...

driven culture within the organisation, mentoring team members, and contributing to the continuous improvement of the Engineering Practice You Will Demonstrate: Proficiency in SQL and Python: You are highly proficient in SQL and Python, enabling you to handle complex data problems with ease. Understanding of Data … Lakehouse Architecture: You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions: You possess experience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or even on-premise Spark clusters, using PySpark ...

techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

3+ years with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema ...

3+ years with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema ...

conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics ...

platforms,lakehousesand data warehouses. Data modelling and dimensional modelling techniques such as Kimball. ETL/ELT development and large-scale data integration. PySpark, Spark SQL and modern transformation techniques. Advanced SQL and relational databases including SQL Server, PostgreSQL and Oracle. Integrating structured … semi-structured and unstructured data sources. Migration and modernisation of legacy platforms such as Oracle, SQL Server/SSIS or Synapse. Power BI semantic models, curated datasets and supporting analytics platforms. Data quality, governance, metadata management and lineage. Data security, classification and access control approaches. Automated testing, validation ...

Implement automated testing, CI/CD patterns + deployment best practices for Foundry-based data assets. Technical Environment: Data Engineering, Palantir Foundry, Python, SQL, PySpark, Spark SQL + Cloud (AWS, Azure, GCP), CI/CD, Git based workflows. Benefits ...

team. This role is central to delivering Canopius' data strategy through development of new data capabilities and supporting the final transition from legacy SQL Server-based environments. Alongside hands-on delivery, the role provides technical leadership across data engineering initiatives, helping to establish a future-ready, cloud-native … secure, scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines to ensure ...

utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities. Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices. Design enterprise monitoring, logging, operational dashboards, alerting … Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security; Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL); Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage ...

approved; relevant data solutions development qualifications (e.g. Microsoft certification; Databricks Certification; data apprenticeship qualifications)Additional InformationTech StackSQL Server, SSIS, T-SQLADF, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, Delta lakeSummaryType: Full-timeFunction: Analyst ...

processes and solutions. Lead the execution of Microsoft Fabric migration activities. Lead and guide migration engineers and QA resources. Develop Python, PySpark and Spark SQL code for data validation and curation within notebooks. Create data integration test cases and support testing activities. Implement and amend data … Azure Functions o Azure App Service o Logic Apps o Azure Kubernetes Service (AKS) o Web Apps Hands-on expertise in PySpark, SQL and Python. Good knowledge of real-time streaming applications, preferably including Kafka, Azure Functions or Azure Service Bus. Strong understanding of Spark processing ...