Distribution architecture and technology stack within our new cloud-based datalake-house.be a hands-on contributor, senior lead in the big data and data lake space, capable to collaborate and influence architectural and design principles across batch and real time flows • have a continuous improvement mindset, who is always … extracting information and identifying linkages across disparate datasets. • Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and DeltaLake • Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD)/ETL tools (Azure Data factory, Informatica) • Experience in data modelling more »
ETL processes to read, transform, and verifying data using tools such as SQL and PySpark Cloud technologies and architecture including Databricks, Kubernetes, Data/DeltaLake, Azure Machine Learning, Data Factory. Azure is preferable, but AWS and GCP experience is welcome Familiar with all parts of the Machine more »
Java, Scala, GoLang. Significant experience with Hadoop, Spark and other distributed processing platforms and frameworks. Experience working with Open table/storage formats like deltalake, apache iceberg or apache hudi. Experience of developing and managing real time data streaming pipelines using Change data capture (CDC), Kafka and more »
the project. There is a massive emphasis on Pyspark and Databricks for this particular role. Technical Skills Required: Azure (ADF, Functions, Blob Storage, Data Lake Storage, Azure Data Bricks) Databricks Spark DeltaLake SQL Python PySpark ADLS Day To Day Responsibilities: Extensive experience in designing, developing, and more »
nigelfrank.com or alternatively, give me a call on 0191 3387551. Keywords: Azure Data Factory, Azure Databricks, Databricks Lakehouse, MS Power BI, Power BI, Spark, DeltaLake, T-SQL, DevOps, ETL, Data Modelling, DAX, Data Warehousing, London more »
and constructing robust data pipelines using the best of open-source data engineering and scientific Python toolset. Tech Stack: Airbyte AWS Glue Pandas Pyspark DeltaLake PostgreSQL The team follows agile ways of working and you engage with various stakeholders across the business. The role is Hybrid more »
Airbyte for data ingestion, Prefect for pipeline orchestration, AWS Glue for managed ETL, along with Pandas and PySpark for pipeline logic implementation. We utilize DeltaLake and PostgreSQL for data storage, emphasizing the importance of data integrity and version control in our workflows. Day-to-day, you will more »
City of London, London, United Kingdom Hybrid / WFH Options
Concept Resourcing
success of various IT salesforce projects. Key Responsibilities: Architecture, modelling and leadership skills Strong Azure data skills, including: Azure Data Factory V2, Azure Data Lake Storage V2, Azure Databricks, Azure Function Apps& Logic Apps, Azure Stream Analytics, Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell … Strong PySpark, DeltaLake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software development practices, such as SOLID principals, structuring code, testing, IOC, etc. Strong repositories, CI/CD skills. Strong knowledge of Kimball more »
data architecture expertise to craft scalable data solutions aligned with business requirements. Design and fine-tune data storage and retrieval systems using Azure Data Lake Storage, Azure SQL Database, and other Azure DB services. Develop, optimize, and maintain data pipelines for automated data integration from various sources, including API … industry as an Azure Data Engineer and/or BI Developer. Proficiency in Azure data tools such as Data Factory, Synapse, Databricks, SQL, Data Lake or Delta Lake. Strong data engineering and data modeling skills, with the ability to craft complex data models. In-depth understanding of Azure more »
SE1, Tower of London, Greater London, United Kingdom Hybrid / WFH Options
Avanti Recruitment
working anywhere in the world for up to 30 days! Requirements: Azure Data Bricks Data Strategy Data Transformation Data Cloud Migration Data API Design DeltaLake/Lakehouse Strong communication skills and also being able to translate to a non-technical audience Previous Finance/Fintech/Forex more »
alternatively, give me a call on (phone number removed). Keywords: Azure Data Factory, Azure Databricks, Databricks Lakehouse, MS Power BI, Power BI, Spark, DeltaLake, T-SQL, DevOps, ETL, Data Modelling, DAX, Data Warehousing, London more »
