8 of 8 Permanent PySpark Jobs in the East of England

Data modelling and data warehousing Extract-Transform-Load (ETL) Creating data pipelines Ethics in data and AI Cloud and big data technologies (AWS, Snowflake, PySpark) Working with unstructured data Working with Generative AI – Large language models (LLMs) and image generation (ChatGPT, Gemini) Consulting Skills Effective teamwork Business acumen Time ...

successful candidate will design and implement modern data lakehouse architectures, ensuring high performance and scalability. Key responsibilities include developing data pipelines using Python and PySpark, working with AWS services, and implementing CI/CD pipelines. Strong experience in AWS Data Engineering and knowledge of data quality frameworks are essential. ...

supporting the Intelligence Community or Department of Defense environments Experience with Python and working with databases Experience with big data frameworks such as Spark, PySpark, or Elastic Search Experience with data visualization platforms such as Kibana, Foundry Workshop, or Palantir Knowledge of secure coding practices and development in classified ...

operational decisions. Join a flexible, collaborative team moving toward a new UK-based Support Centre in St Albans. You’ll work with Python, PySpark, SQL, Azure and Databricks in a modern data environment, communicating clearly with both technical and non-technical colleagues to drive value. #J-18808-Ljbffr ...

including architecture, development, deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture (Bronze/Silver … reviews. Skills & Experience Strong experience in dataengineering, big data, or solution architecture. Strong hands-onexperience with Databricks Platform . Expert knowledge of: Apache Spark PySpark/Scala Delta Lake SQL Data Lakehouse architecture ETL/ELT development Experience designing anddelivering enterprise Databrickssolutions. Strong experience withcloud platforms: Azure Databricks (preferred ...

operational decisions. Applying statistical and machine learning techniques to build predictive and prescriptive models, test hypotheses and solve real business problems. Using Python, PySpark and SQL to analyse data and develop high‐quality, maintainable solutions. Creating and maintaining data pipelines and automated workflows, following agreed ETL and data governance … domain. Proven expertise in using data, statistical analysis and machine learning to solve business challenges and generate actionable insight. Strong technical capability across Python, PySpark and SQL, with experience working with complex datasets. Strong knowledge of machine learning algorithms, including classification, regression, supervised learning and unsupervised learning. Proven expertise ...

Lakes*** Experience with **Apache Airflow/MWAA** for workflow orchestration* Strong knowledge of **dbt (data build tool)** for data transformation* Experience with **EMR/PySpark** for large-scale data processing* Ability to create **Architecture Design Documents (ADDs)*** Deep understanding of **cloud-native design patterns*** Experience in **performance tuning** across ...

scalable, secure, and high-performance data platforms.Key Responsibilities Design and implement scalable AWS Data Lakehouse solutions Develop and maintain data pipelines using Python and PySpark Work with AWS services including: S3, EMR, Lambda, IAM, EKS, MWAA Build and manage Apache Iceberg tables for efficient data storage and querying Implement … engineersRequired Skills & Experience Strong experience in AWS Data Engineering Hands-on expertise with: AWS services: IAM, Lambda, EKS, S3, EMR, MWAA Apache Iceberg Python & PySpark Solid understanding of data lakehouse architecture principles Experience with CI/CD pipelines and Terraform Knowledge of data quality frameworks and data security standards ...