1 to 25 of 152 Permanent PySpark Jobs in London

optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow Experience with containerisation ...

Azure Databricks Azure Function Apps& Logic Apps Azure Stream Analytics Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell) Strong PySpark, Delta Lake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software ...

models in data warehouses, lakehouse environments or analytics tools such as Power BI. Familiarity with modern data engineering frameworks and tools (e.g. Spark/PySpark, Pandas, orchestration tools). Experience with cloud data platforms on AWS or Azure (e.g. Databricks, Redshift, Microsoft Fabric). Understanding of software engineering fundamentals ...

construction, programme controls, PMO, or consultancy environments. Experience with cloud-native architectures, DevOps, CI/CD pipelines, and DataOps practices. Knowledge of Python, Spark, PySpark, Databricks, or equivalent data engineering technologies. Information Management in a complex construction or infrastructure project/programme Experience developing enterprise data products and supporting ...

Good understanding of modern software platform architecture and the role of the data platform within the broader platform ecosystem Good Scala skills (or strong PySpark plus a genuine willingness to work primarily in Scala) Strong SQL skills for transformations, data modelling, and debugging data issues Experience building and supporting ...

influencing skills. Ability to think creatively and propose innovative solutions. To qualify for this role, you must have: Proven experience and knowledge with PySpark and Apache Spark including the fundamentals of how it works. Core experience with AWS, with substantial and mature Azure platform offering. Experience with other cloud ...

/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft Purview. ...

Snowflake warehouse. Coach and mentor junior engineers, and lead technical discussions and design reviews. Design, build, and optimize data pipelines using Databricks, Python and PySpark, with event-driven components (e.g. Kafka). Ensure pipelines are auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls ...

Quality, Data Integration, Data Analytics, Data Visualisation, Data Science and Business Intelligence. Confident in delivering solutions using Databricks & developing and optimising data pipelines using PySpark within Databricks environments. Proven experience building, publishing, and supporting dashboards and reports using Power BI Strong experience and programming skills in languages such ...

deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good ...

complex data workflows Excellent communication and collaboration skills in agile, cross-functional teams Ability to mentor junior engineers Nice to have: Experience with Databricks, PySpark, and Delta Lake Условия Competitive salary based on location, experience, and skills Comprehensive benefits package for employees and their families Continuous learning, structured development ...

data engineering or integration environments. 8–10 years’ experience in data engineering and integration , ideally in enterprise‐scale environments. Hands‐on experience with Spark (PySpark/SparkSQL) , ETL/ELT design, and data pipeline orchestration . Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing ...

data-science role, including delivery of reliable pipelines, reusable data assets and useful analytical models in Microsoft Fabric and Databricks. Advanced Python, SQL and PySpark, including experience with data transformation, analysis and common machine-learning libraries. Hands-on experience designing and operating scalable data pipelines, lakehouse data models, Delta ...

Data Lake, Lakehouse, Data Warehouse, Data Vault ETL/ELT processes and designing, building and testing data pipelines Building data transformations using Python/PySpark Azure Cloud Version control and CI/CD tools, specifically Azure DevOps Service Analytics and MI products including MS Power BI Data catalogue & governance ...

only on ETL. However, we will consider candidates who bring Cloud + CI/CD experience alongside technologies such as: Proficient SQL; familiarity with PySpark is beneficial dbt Microsoft Fabric SQLMesh 5. Core Engineering Skills Strong Python, with good documentation and unit testing practices Comfortable in an agile delivery ...

following technical expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both ...

Experience leading or mentoring teams of engineers to deliver high‐quality scalable data solutions. Technical Expertise Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow). Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent. Strong ...

Skills Significant experience leading the design, development, and operation of enterprise‐scale data engineering platforms and teams. Strong technical expertise in SQL and Python, PySpark, or equivalent technologies for data ingestion, transformation, and validation. Experience with Microsoft Fabric or similar modern cloud‐based analytics and data platforms. Strong experience ...

/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls for sensitive or identifiable data, utilizing Azure Key Vault, Azure Monitor, Log Analytics, and Microsoft Purview. ...

storage Proficiency with Terraform (ideally the Databricks provider) and Gitbased CI/CD workflows Solid programming skills in Python and SQL; comfortable with PySpark and shell scripting Experience tuning Spark jobs and Delta tables (partitioning, Working knowledge of SSO/SCIM, identity providers and secretsmanagement Understanding of data security ...

Data Stack, critically ADF and Synapse (experience with Microsoft Fabric is a plus) Highly developed python and data pipeline development knowledge, must include substantial PySpark experience Demonstrable DevOps and DataOps experience with an understanding of best practices for engineering, test and ongoing service delivery An understanding of Infrastructure ...

Need to Be Successful Deep, hands‐on experience building and running production data platforms and products A good level of experience in Python (particularly pyspark) for building data pipelines Strong AWS and Terraform experience, with infrastructure‐as‐code as your default way of working rather than something ...

data warehousing, dimensional modelling, Kimball principles and data architecture Advanced SQL/T-SQL skills, including query development, optimisation and troubleshooting Strong Python, PySpark or similar programming skills for data manipulation and automation Experience integrating data from APIs, databases, files and other sources Experience with data quality, pipeline monitoring ...

Star Schema concepts (Fact and Dimension tables), from the ground up. Git: You're fluent in Git workflows — branching, troubleshooting, code reviews. Python/PySpark/SQL: You write clean, efficient code across all three, and you understand how it performs at scale. Database management: Experience with SQL/ ...

Analytics capabilities. What you'll do Data Engineering Delivery: Design, build andoptimisescalable data pipelines, ETL/ELT workflows and data transformation solutions using Databricks,PySpark, SQL anddbtCloud, delivering trusted, high-quality data products that support reporting, analytics, AI and data science. Data Architecture & Modelling: Develop and maintain robustlakehousearchitectures ...