76 to 100 of 160 Remote/Hybrid PySpark Jobs

environment. What You'll Need: Strong experience as a Senior Azure Data Engineer An expert in Azure Databricks and Medallion Architecture Strong skills in PySpark, Python, Delta Lake and Unity Catalog Knowledge of Kimball data modelling, such as star schema, snowflake, etc. Strong SQL skills A flexible approach ...

methodology. Platforms & Tools Experience with Databricks (preferred) or similar platforms: Snowflake, BigQuery, Redshift, Synapse, or equivalent Strong proficiency with: SQL (advanced level) Spark/PySpark Data pipeline orchestration tools (e.g., Airflow, ADF, etc.) Data Modelling Extensive experience designing: Fact and dimension tables Star and snowflake schemas Aggregation layers ...

experience with Databricks — running queries, navigating Unity Catalog, reading Spark job outputs and understanding what they mean for data quality Working knowledge of PySpark or Spark SQL — enough to read pipeline code, understand transformations, and trace where data issues originate Understanding of Lakehouse/medallion architecture (bronze-silver-gold ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation. Experience with Databricks and PySpark for large-scale data processing. Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring. Experience implementing responsible ...

interrogate data andverify correctness Hands-on experience with Databricks — running queries, navigatingUnity Catalog, reading Spark job outputs and understanding what they Working knowledge of PySpark or Spark SQL — enough to read pipelinecode, understand transformations, and trace where data Understanding of Lakehouse/medallion architecture (bronze-silvergold) and how data ...

management (Metadata Driven Ingestion). Good working experience of database technologies and database integration, such as SQL, Postgres, MySQL or Oracle. Experience in Python (PySpark) Experience in working with DevOps capabilities like version control, automated builds, testing and release management capabilities using Microsoft DevOps and Git repositories. Good numerical … technology roles, such as architecture, operations, and design related disciplines. At least three years of experience in Azure (Azure Synapse, Azure Databricks, Fabric, PySpark). Comprehensive experience with data modelling Experience across a range of other IT functional disciplines will be of benefit, including: Data Platform tools CI/ ...

Artefact is a new generation of data service provider, specialising in data consulting and data-driven digital marketing, dedicated to transforming data into business impact across the entire value chain of organisations. We are proud ...

Microsoft Azure. You will build production-grade data pipelines with Delta Lake, Unity Catalog, and Databricks tools. The role requires hands-on expertise in PySpark, Python, SQL, and Spark architecture, plus experience with GitLab-based CI/CD, Databricks Asset Bundles, and Azure Data Lake Storage. #J-18808-Ljbffr ...

troubleshooting of Azure DatabricksDatabricks Serverless configuration and workload optimisationDatabricks SQL and Delta Lake developmentCluster management, compute policies, autoscaling, and workload monitoringData EngineeringStrong Python, PySpark, and SQL development skillsDesign and build ingestion and transformation pipelinesFull and incremental data loadingCDC (Change Data Capture) implementationData quality controls, schema evolution, reconciliation, and error ...

successful candidate will design and implement modern data lakehouse architectures, ensuring high performance and scalability. Key responsibilities include developing data pipelines using Python and PySpark, working with AWS services, and implementing CI/CD pipelines. Strong experience in AWS Data Engineering and knowledge of data quality frameworks are essential. ...

understanding of DAG (Directed Acyclic Graphs) and their role in pipeline orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation.Experience with Databricks and PySpark for large-scale data processing.Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring.Experience implementing responsible AI practices ...

Delta Lake, with governance, security, and real-time processing at scale. The role sits in the Enterprise Data Team and requires strong SQL and PySpark, plus experience with cloud-native data platforms. Hybrid working available, with 2 days in the office and 3 days from home. #J-18808-Ljbffr ...

functional teams to deliver high-quality technical solutions. Support the ongoing maintenance and optimisation of platform services. Essential Skills & Experience Hands-on experience with PySpark for data transformation and processing. Experience working with AWS data services , including: AWS Glue Amazon S3 Experience using Terraform for Infrastructure as Code. Experience ...

Declarative Pipelines with Expectations, including failure handling and observability Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns Production-level PySpark and SQL development skills Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM Experience with Databricks Workflows, orchestration ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

will be doing Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into … into engineered solutions What we need from you 8+ years in data engineering, including 3+ years hands on with Azure Databricks Strong Python and PySpark, backed by advanced SQL Deep grounding in data warehousing, data modelling and integration patterns Unity Catalog and Delta Lake in a production environment Data ...

Choose a partner with intimate knowledge of your industry and first-hand experience of defining its future.Your locationYour locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.Birmingham ...

Engineer to join the Data & Analytics team. The role is hybrid, focusing on building scalable data pipelines on Databricks and AWS, with Python/PySpark expertise. You will design data architectures, work with data warehousing concepts and contribute to analytics and AI initiatives, while collaborating with stakeholders ...

with a leading cloud-based software innovator to shape data analytics, AI-driven automation and scalable data pipelines. You will craft Python, SQL, and PySpark solutions, design analytics architectures, and validate AI-assisted outputs to guide business decisions. The role blends data engineering, analytics, and responsible AI practices ...

turn data into revenue. The role combines hands-on development with architecture decisions in a hybrid London setting. You will work with Python (pyspark), AWS, Databricks and dbt/Fivetran to deliver scalable data pipelines, apply governance, and drive #J-18808-Ljbffr ...

Kubernetes Experience in creating production grade coding and SOLID programming principles, including test-driven development (TDD) approaches Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Experience in source-control software, e.g., GitHub Proficient at communicating results in a concise manner both verbally and written Experience in data ...

production settings Experience with multimodal modelling (e.g. text + image) Familiarity with human-in-the-loop systems or moderation workflows Experience with Databricks, PySpark, or large-scale data processing systems Additional Information Health + Mental Wellbeing PMI and cash plan healthcare access with Bupa Subsidised counselling and coaching with ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … Microsoft Fabric (Lakehouse, Warehouse, pipelines). Azure Data Factory experience will also be considered if no working experience with fabric Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

solutions, ensuring alignment with business priorities and best practice Implementing robust ETL/ELT pipelines using Azure Data Factory, Synapse Pipelines and Azure Databricks (PySpark/Spark SQL) to support batch and streaming data processing Designing and delivering real-time and batch data ingestion solutions using Azure Event Hubs … Synapse Analytics, Azure SQL, Azure Data Lake, Datra bricks, Purview and Azure App Service is essential Proficiency in Python/SQL/SparkSQL/PySpark is essential Strong experience of Azure delivery experience including analysis/development/configuration/deployments essential Expert level technical skill ...