9 of 9 PySpark Jobs in the North East

Databricks ecosystem. What you'll need: Strong experience building data pipelines using Azure Data Factory, Databricks and Azure Data Lake. Advanced SQL and Python (PySpark) skills for data transformation, automation and optimisation. Hands-on experience with Delta Lake, Spark, Databricks Workflows and Structured Streaming. Knowledge of data warehousing, lakehouse ...

storage, data pipelines to ingest and transform data, and querying & reporting of analytical data. You've worked with technologies such as Python, Spark, SQL, Pyspark, PowerBI etc. You’ve got a background in software engineering, including Front End technologies like JavaScript. You’re a problem-solver, pragmatically exploring options ...

best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs Use of Jupyter notebooks and/or Amazon Athena to query and validate data Data analysis ...

Delta Lake, with governance, security, and real-time processing at scale. The role sits in the Enterprise Data Team and requires strong SQL and PySpark, plus experience with cloud-native data platforms. Hybrid working available, with 2 days in the office and 3 days from home. #J-18808-Ljbffr ...

functional teams to deliver high-quality technical solutions. Support the ongoing maintenance and optimisation of platform services. Essential Skills & Experience Hands-on experience with PySpark for data transformation and processing. Experience working with AWS data services , including: AWS Glue Amazon S3 Experience using Terraform for Infrastructure as Code. Experience ...

Data Engineer in the United Kingdom to design, build, and maintain data pipelines on a modern AWS-native stack. You will use Spark/PySpark for distributed processing, Airflow for orchestration, and Terraform for infrastructure as code in a hybrid work arrangement with weekly travel to Newcastle office. ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

Hybrid, 3 days onsite). This 12-month fixed-term contract role focuses on building and supporting modern cloud-based data platforms using AWS, PySpark, and IaC. You will work with Terraform, GitLab CI/CD, and collaborate with engineers to deliver secure, scalable cloud solutions. You’ll join ...

Strong knowledge of migrating and transforming data from legacy and on-premise systems to cloud platforms. Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments. You and your role As a Data Engineer, you will design build and enhance scalable …/ELT pipelines using Python and PySpark, ensuring data solutions are reliable, efficient and aligned with engineering best practice. You'll work closely with stakeholders to understand complex business requirements, transforming data into trusted, high-quality datasets that enable valuable insights, reporting and decision-making. You will manage ...