1 to 25 of 37 PySpark Jobs in the City of London

like Azure DevOps or Jenkins and following solid engineering practices. What you need to bring Strong hands-on experience with Python (including Pandas) and PySpark or similar distributed frameworks. Confident SQL across TSQL, ANSI SQL, Spark SQL. Real experience with cloud data services (Azure Data Factory, Databricks, Synapse ...

security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations ...

Spark Programming Ability to write Spark code for large scale data processing, including RDDs, Data Frames, and Spark SQL Python/Notebook programming PySpark programming Spark Streaming/batch processing Delta table Optimization Fabric spark jobs Java programming language, OOP knowledge Database knowledge, including Relational Database ...

Engineering, with a minimum of 3 years of hands-on Azure Databricks experience delivering production-grade solutions. Strong programming proficiency in Python and Spark (PySpark) or Scala, with the ability to build scalable and efficient data processing applications. Advanced understanding of data warehousing concepts, including dimensional modelling, ETL/ ...

pipelines, PowerShell scripting, Terraform/Bicep for IaC Skills and knowledge on Security & Compliance: Key Vault, RBAC, encryption, vulnerability management Programming skills: Python/PySpark, SQL; familiarity with .NET or Java is a plus Cloud Networking & Identity experience: VNET, NSG, IAM, MFA, SAML/OAuth/OIDC. Desirable skills ...

solutions integrated into SaaS products Proficiency in Python with libraries like TensorFlow, PyTorch, or Scikit-learn for ML, and Pandas, PySpark, or similar for data processing. Experience designing and orchestrating data pipelines with tools like Apache Airflow, Spark, or Kafka. Strong understanding of SQL, NoSQL, and data modeling. Familiarity ...

overview of many different sectors. What We’re Looking For 5 + years, hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools. Strong experience engineering in a front-office/capital markets environment. Previous experience ...

Databricks , Azure , and secure cloud environments (if not AWS experience accepted) Strong communication and problem-solving skills Tech Stack Required Databricks DBT (ideally) Python PySpark SQL Azure/AWS Please apply directly ...

overview of many different sectors. What We’re Looking For 10 + years, hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools. Strong experience engineering in a front-office/capital markets environment. Previous experience ...

Looking For: Proven hands-on experience in data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like ...

using Python, Glue, S3, Airflow, DBT, Redshift, RDS, etc. Extensive Python API design experience, preferably Fast API Strong SQL knowledge Nice to have skills Pyspark Databricks ETL design ...

Purview, Cosmos DB, Open AI, Azure ML, AI Foundary, Kubernetes, Understanding of software engineering tools and concepts including experience in Python, Scala or PySpark, database technologies, data modelling and SQL Confident communicator who is able to explain technical terms to non-technical audiences and mentor junior colleagues Leads small ...

looking for: 3+ years’ experience in data engineering and cloud infrastructure Hands on experience with Fivetran of similar data integration platforms Expertise in Python (PySpark), SQL, and dbt (or similar) Some Cloud Infrastructure knowledge: DevOps practices, CI/CD, AWS. Data warehousing experience (PostgreSQL or similar) Experience with event ...

5+ years of experience in data engineering or similar data-intensive roles. Strong Python skills and experience with data libraries (e.g., Pandas, Polars, Dask, PySpark). Expertise in SQL and relational databases; familiarity with columnar formats (Parquet/Arrow). Experience with distributed systems and messaging tools (e.g., Kafka ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data ...

building out the businesses first data warehouse solution. Required Experience: Hands-on Microsoft Fabric and Azure data platform expertise. Extensive experience with Python/PySpark, and ETL skills. Exposure working within Investment Management/Alternative Investments Strong SQL/T-SQL skills Experience utilising Power BI Leadership and mentoring ...

this Senior Data Engineering role, you’ll design, develop, and maintain data platforms using technologies such as Snowflake, Databricks, Synapse/Fabric, and PySpark, ensuring the scalability, security, and performance of all data systems. As the Senior Data Engineer your role will be to include establishing and championing best ...

Ontology, pipelines, apps, AIP) Some experience in object-oriented design, software engineering, and experience in at least one relevant coding language (e.g. Python/PySpark, R, Java, Typescript, SQL) Experience of UX design and/or data visualisation Strong understanding of data engineering, data modelling, and platform integration across ...

solutions. This will be a long term project with scope for extensions and possible transition to permanent staff. Core Skills Needed: Strong Python development PySpark and SQL expertise Solid understanding of Azure infrastructure Knowledge of Containers, Microservices, and functional design patterns Experience working in Agile environments Terraform ...

outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Support ML workflows and data-driven product features Collaborate closely with engineering, product, and leadership to deliver fast, meaningful results Tech stack includes Python, PySpark, GCP, Azure Synapse, AWS. What They're Looking For Solid experience as a Senior Data Engineer , ideally in fast-moving or startup environments Strong ...

governance , schema validation, and master data management. Strong grasp of real-time vs batch processing trade-offs . Beneficial Experience CockroachDB deployment and management. PySpark or similar for large-scale processing. SAP data structures and PIM systems. E-commerce and B2B data integration patterns. Why Apply? Fully remote contract ...

deliver outcomes, not experiments Helping define the standards, patterns, and ways of working as the platform evolves What you’ll bring Strong Python/PySpark and solid SQL coding experience Proven delivery of production systems at scale, not just prototypes Cloud experience and modern engineering practices, CI/ ...

/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata management, lineage, and monitoring. Leadership & Stakeholder Engagement Work closely with business leaders and product teams to translate … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

quality and structure of data Translate business needs into clear technical plans Experience required: Strong skills in data modelling SQL with either Python or PySpark Databricks or similar cloud data platform Experience communicating with both technical and non-technical users Leadership or mentoring experience Insurance or regulated sector experience ...