1 to 25 of 78 Remote PySpark Jobs

Essential Skills for the AWS Data Engineer: Extensive hands-on experience with AWS data services Strong programming skills in Python (including libraries such as PySpark or Pandas) Solid understanding of data modelling, warehousing and architecture design within cloud environments Experience building and managing ETL/ELT workflows and data

security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations

data sources into Azure Databricks. Optimize pipelines for performance, reliability, and cost, incorporating data quality checks. Develop complex transformations and processing logic using Spark (PySpark/Scala) for cleaning, enrichment, and aggregation, ensuring accuracy and consistency across the data lifecycle. Work extensively with Unity Catalog, Delta Lake, Spark

Azure Storage. Excellent SQL and data modelling (star/snowflake, dimensional modelling). Knowledge of Power BI dataflows, DAX, and RLS. Experience with Python, PySpark, or T-SQL for transformations. Understanding of CI/CD and DevOps (Git, YAML pipelines). Strong grasp of data governance, security, and performance

Azure Storage. Excellent SQL and data modelling (star/snowflake, dimensional modelling). Knowledge of Power BI dataflows, DAX, and RLS. Experience with Python, PySpark, or T-SQL for transformations. Understanding of CI/CD and DevOps (Git, YAML pipelines). Strong grasp of data governance, security, and performance

office. About you : You have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology

office. About you : You have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology

Experience as a Senior Data Engineer or similar Strong knowledge of Azure Cloud architecture and Azure Databricks, DevOps and CI/CD. Experience with PySpark, Python, SQL and other data engineering development tools. Experience with metadata driven pipelines and SQL serverless data warehouses. Knowledge of querying API systems. Experience

Experience as a Senior Data Engineer or similar Strong knowledge of Azure Cloud architecture and Azure Databricks, DevOps and CI/CD. Experience with PySpark, Python, SQL and other data engineering development tools. Experience with metadata driven pipelines and SQL serverless data warehouses. Knowledge of querying API systems. Experience

Experience as a Senior Data Engineer or similar Strong knowledge of Azure Cloud architecture and Azure Databricks, DevOps and CI/CD. Experience with PySpark, Python, SQL and other data engineering development tools. Experience with metadata driven pipelines and SQL serverless data warehouses. Knowledge of querying API systems. Experience

Days Cycle to work scheme And More.. DATA ENGINEER - ESSTENTIAL SKILLS Proven experience building data pipelines using Databricks . Strong understanding of Apache Spark (PySpark or Scala) and Structured Streaming . Experience working with Kafka (MSK) and handling real-time data . Good knowledge of Delta Lake/Delta

pipelines to ingest, transform, and load data from diverse sources (APIs, databases, files) into Azure Databricks. Implement data cleaning, validation, and enrichment using Spark (PySpark/Scala) and related tools to ensure quality and consistency. Utilize Unity Catalog, Delta Lake, Spark SQL, and best practices for Databricks development, optimization

processes Develop and maintain ELT/ETL pipelines using AWS Glue, Step Functions, Lambda, DMS, and AppFlow Automate transformations and model refresh using Python, PySpark and SQL Implement end-to-end source-to-target data integration, mapping and transformation across data source ? Raw ? Curated ? Redshift Proficient in Redshift

demand delivery Provide technical guidance for team members Providing 2nd or 3rd level technical support About You Experience using SQL, SQL Server DB, Python & PySpark Experience using Azure Data Factory Experience using Data Bricks and Cloudsmith Data Warehousing Experience Project Management Experience The ability to interact with the operational

Fabric. Migrate on-prem data to Fabric Lakehouse/Warehouse, enabling Power BI analytics. Develop robust data pipelines, notebooks, and data models using: Python, PySpark, SQL, Synapse, and Data Warehousing principles. Implement CI/CD processes with GitHub integration for lifecycle management. Ensure governance, security, and best practices across

Fabric. Migrate on-prem data to Fabric Lakehouse/Warehouse, enabling Power BI analytics. Develop robust data pipelines, notebooks, and data models using: Python, PySpark, SQL, Synapse, and Data Warehousing principles. Implement CI/CD processes with GitHub integration for lifecycle management. Ensure governance, security, and best practices across

patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data quality, data governance processes, Git version control + Agile development environments. Azure

patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data quality, data governance processes, Git version control + Agile development environments. Azure

orchestration. Experience with Apache Airflow DAG development and scheduling best practices. Strong understanding of data warehouse and data lake concepts. Knowledge of Spark or PySpark for big data processing. Familiarity with CI/CD pipelines, Git, and Terraform for infrastructure automation.

Continuously improve data systems and maintain awareness of best practice and emerging approaches Essential Skills Experience with large-scale analytics engines (e.g. Spark/PySpark) and scripting languages such as Python or Scala. Hands-on use of cloud data stacks (e.g. SageMaker Notebooks, S3, Glue, Athena) and modern storage

data models and analytics; exposure to predictive modelling and machine learning is a plus. Proficient in SQL and Python, with bonus points for PySpark, SparkSQL, and Git. Skilled in data visualisation with tools such as Tableau or Power BI. Confident writing efficient code and troubleshooting sophisticated queries. Clear

Methodology or other similar dimensional modelling standards, is essential to the role. Technical experience building and deploying models and reports utilizing the following tools: PySpark Microsoft Fabric or Databricks Power BI Git CI/CD pipelines (Azure DevOps experience preferred) An understanding of the structure and purpose

agentic capabilities, able to build production-grade applications not just prototypes. Strong proficiency in at least one relevant programming language such as Python or PySpark, Java, Typescript or SQL. Solid understanding of data engineering, data modelling, security and governance in enterprise settings. Experience with software engineering best practices including

manipulate data and building additional process to support data processing activities Experience manipulating, processing and storing data using tools such as SQL, dbt, pyspark and SSIS Comfortable building out and administrating data models, ensuring that they are accessible and used appropriately, leveraging platforms such as Databricks and Power

mentor other engineers. Must-Have Experience Proven experience as a Senior Data Engineer. Strong Python software engineering foundation. Hands-on Spark experience in production (PySpark essential). Real-world experience using Databricks for data pipelines (Spark depth matters most). Experience with large-scale or parallel data processing. Ownership