1 to 25 of 37 PySpark Jobs in the City of London

like Azure DevOps or Jenkins and following solid engineering practices. What you need to bring Strong hands-on experience with Python (including Pandas) and PySpark or similar distributed frameworks. Confident SQL across TSQL, ANSI SQL, Spark SQL. Real experience with cloud data services (Azure Data Factory, Databricks, Synapse ...

security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations ...

Spark Programming Ability to write Spark code for large scale data processing, including RDDs, Data Frames, and Spark SQL Python/Notebook programming PySpark programming Spark Streaming/batch processing Delta table Optimization Fabric spark jobs Java programming language, OOP knowledge Database knowledge, including Relational Database ...

Engineering, with a minimum of 3 years of hands-on Azure Databricks experience delivering production-grade solutions. Strong programming proficiency in Python and Spark (PySpark) or Scala, with the ability to build scalable and efficient data processing applications. Advanced understanding of data warehousing concepts, including dimensional modelling, ETL/ ...

pipelines, PowerShell scripting, Terraform/Bicep for IaC Skills and knowledge on Security & Compliance: Key Vault, RBAC, encryption, vulnerability management Programming skills: Python/PySpark, SQL; familiarity with .NET or Java is a plus Cloud Networking & Identity experience: VNET, NSG, IAM, MFA, SAML/OAuth/OIDC. Desirable skills ...

overview of many different sectors. What We’re Looking For 5 + years, hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools. Strong experience engineering in a front-office/capital markets environment. Previous experience ...

overview of many different sectors. What We’re Looking For 10 + years, hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools. Strong experience engineering in a front-office/capital markets environment. Previous experience ...

solutions integrated into SaaS products Proficiency in Python with libraries like TensorFlow, PyTorch, or Scikit-learn for ML, and Pandas, PySpark, or similar for data processing. Experience designing and orchestrating data pipelines with tools like Apache Airflow, Spark, or Kafka. Strong understanding of SQL, NoSQL, and data modeling. Familiarity ...

Looking For: Proven hands-on experience in data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like ...

Databricks , Azure , and secure cloud environments (if not AWS experience accepted) Strong communication and problem-solving skills Tech Stack Required Databricks DBT (ideally) Python PySpark SQL Azure/AWS Please apply directly ...

technical documentation including architecture diagrams and data flows To be successful in this role, you will have: Deep experience with Azure Databricks, SparkSQL and PySpark Experience working with Azure technologies such as Data Factory, Synapse and Azure Data Lake Storage Strong SQL skills Solid understanding of modern data modelling ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data ...

using Python, Glue, S3, Airflow, DBT, Redshift, RDS, etc. Extensive Python API design experience, preferably Fast API Strong SQL knowledge Nice to have skills Pyspark Databricks ETL design ...

looking for: 3+ years’ experience in data engineering and cloud infrastructure Hands on experience with Fivetran of similar data integration platforms Expertise in Python (PySpark), SQL, and dbt (or similar) Some Cloud Infrastructure knowledge: DevOps practices, CI/CD, AWS. Data warehousing experience (PostgreSQL or similar) Experience with event ...

5+ years of experience in data engineering or similar data-intensive roles. Strong Python skills and experience with data libraries (e.g., Pandas, Polars, Dask, PySpark). Expertise in SQL and relational databases; familiarity with columnar formats (Parquet/Arrow). Experience with distributed systems and messaging tools (e.g., Kafka ...

building out the businesses first data warehouse solution. Required Experience: Hands-on Microsoft Fabric and Azure data platform expertise. Extensive experience with Python/PySpark, and ETL skills. Exposure working within Investment Management/Alternative Investments Strong SQL/T-SQL skills Experience utilising Power BI Leadership and mentoring ...

this Senior Data Engineering role, you’ll design, develop, and maintain data platforms using technologies such as Snowflake, Databricks, Synapse/Fabric, and PySpark, ensuring the scalability, security, and performance of all data systems. As the Senior Data Engineer your role will be to include establishing and championing best ...

Ontology, pipelines, apps, AIP) Some experience in object-oriented design, software engineering, and experience in at least one relevant coding language (e.g. Python/PySpark, R, Java, Typescript, SQL) Experience of UX design and/or data visualisation Strong understanding of data engineering, data modelling, and platform integration across ...

solutions. This will be a long term project with scope for extensions and possible transition to permanent staff. Core Skills Needed: Strong Python development PySpark and SQL expertise Solid understanding of Azure infrastructure Knowledge of Containers, Microservices, and functional design patterns Experience working in Agile environments Terraform ...

outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Support ML workflows and data-driven product features Collaborate closely with engineering, product, and leadership to deliver fast, meaningful results Tech stack includes Python, PySpark, GCP, Azure Synapse, AWS. What They're Looking For Solid experience as a Senior Data Engineer , ideally in fast-moving or startup environments Strong ...

governance , schema validation, and master data management. Strong grasp of real-time vs batch processing trade-offs . Beneficial Experience CockroachDB deployment and management. PySpark or similar for large-scale processing. SAP data structures and PIM systems. E-commerce and B2B data integration patterns. Why Apply? Fully remote contract ...

client meetings (approx. 2 short trips per quarter). Desirable Experience with B2B products and enterprise-scale data solutions. Knowledge of Spark SQL, PySpark, and related ML libraries. Exposure to distributed data platforms (e.g., Hive, Databricks) and cloud services (AWS preferred). Understanding of advanced AI concepts such ...

/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata management, lineage, and monitoring. Leadership & Stakeholder Engagement Work closely with business leaders and product teams to translate … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

models and manages controlled, production-grade releases to ensure platform stability, scalability, and trusted analytics consumption. Key Responsibilities Design, govern, and operationalize Fabric pipelines, PySpark transformations, and Lakehouse-based Medallion architecture. Translate business requirements and approved data roadmaps into executable technical designs, sprint backlogs, and production-grade delivery plans. ...