201 to 212 of 212 PySpark Jobs

Doing Designing end-to-end Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. Leading client workshops and guiding technical … technologies. What We're Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. CI/CD expertise with Azure DevOps ...

Data Engineer - Active SC, Databricks, PySpark Up to £475 per day (Inside IR35) Remote/London 6 months My client is an International Consultancy who are recruiting for an Data Engineer with Active Security Clearance (SC) and strong Databricks and Azure experience to deliver and optimise Data engineering solutions. … requirements: Proven experience as a Data Engineer with Active Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with Azure services: Data Lake Storage (Gen2 ...

Data Engineer - Active SC, Databricks, PySpark Up to £475 per day (Inside IR35) Remote/London 6 months My client is an International Consultancy who are recruiting for an Data Engineer with Active Security Clearance (SC) and strong Databricks and Azure experience to deliver and optimise Data engineering solutions. … requirements: Proven experience as a Data Engineer with Active Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with Azure services: Data Lake Storage (Gen2 ...

Pyspark Engineer- (Data Engineering, AWS GLUE) SC Cleared OR Eligible Stevenage (Hybrid) 2-3 days onsite Up to £80,000 High-impact programme - Revolutionary platform I am looking for a Security-Cleared Pyspark expert to take the reins on a range of highly ambitious Data Migration projects supporting … databases and AWS cloud environments. Strong hands-on experience with ETL processes and tools (Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi) or scripting using Python, PySpark, and SQL. Solid understanding of data warehousing and modelling techniques (Star Schema, Snowflake Schema). Familiarity with security frameworks such as GDPR, HIPAA ...

generative AI applications. Perform Exploratory Data Analysis (EDA), data mining, and visualization to extract insights. Design and implement Big Data solutions using Hadoop, Spark, PySpark, and Data Lake architectures. Develop and deploy ML models (supervised, unsupervised, tree-based, ensemble methods). Implement deep learning architectures (CNN, RNN, LSTM) using … systems. Required Skills: Analytical Tools: EDA, Data Mining, Visualization (Plotly, Matplotlib, Seaborn) Statistical & Multivariate Analysis Big Data: Hadoop, MapReduce, HDFS, DataBricks, Spark, PySpark DataLake Architecture, AWS Redshift, Kinesis, EMR Machine Learning Supervised Models: Nave Bayes, Logistic Regression, SVM, Linear Regression, KNN, Tree-Based Models: Decision Trees, Random Forest, Gradient ...

efficiency of our existing ML systems and processes. Build, administer and scale ML processing pipelines. Be comfortable navigating the following technology stack: Python3, Pyspark, scripting (Bash/Python), Hadoop, SQL, S3 etc. Should be able to understand internals of ML models such as Random Forest, CNN, Regression models … systems/engineering/related field 8+ years of experience in software engineering with a minimum of 4+ years in ML Good experience in Pyspark Expert level understanding of Python with design patterns and object-oriented programming. Experience debugging and reasoning about production issues is desirable. A good understanding ...

efficiency of our existing ML systems and processes. Build, administer and scale ML processing pipelines. Be comfortable navigating the following technology stack: Python3, Pyspark, scripting (Bash/Python), Hadoop, SQL, S3 etc. Should be able to understand internals of ML models such as Random Forest, CNN, Regression models … systems/engineering/related field 8+ years of experience in software engineering with a minimum of 4+ years in ML Good experience in Pyspark Expert level understanding of Python with design patterns and object-oriented programming. Experience debugging and reasoning about production issues is desirable. A good understanding ...

responsible for: Designing end-to-end data architecture aligned with modern best practices. Building and managing ingestion pipelines using Databricks and related tools. Developing PySpark/Spark SQL notebooks for complex transformations and cleansing. Applying governance, security, and CI/CD best practices across cloud environments. Leading technical discussions … producing professional documentation. To be successful in this role, you will have: Hands-on experience with Databricks including Unity Catalog. Strong PySpark/Spark SQL skills for large-scale transformations. Experience integrating with diverse data sources such as APIs, cloud storage and databases. Experience with the Azure cloud data ...

code. Effectively coordinate the infrastructure/service needs working with the architecture and infrastructure teams. Implement overall solution comprising of Glue ETL PySpark jobs including SQL queries, Lambda, Step Functions, REST API calls. Develop and integrate custom developed software solutions to leverage automated deployment technologies. Use industry leading DevOps … based development. Working experience with development using Amazon Web Services (AWS) and primarily on AWS Glue studio. Experience working with programming language PySpark/Python Experience working with relational database like PostgreSQL. Experience working in a fast-paced development environment with drive to completion. Well versed with using version ...

Python Data Engineer Azure & PySpark - SC Cleared Contract £400-£458pd (Inside IR35) SC Clearance is Essential Summary Were looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing click apply for full job details ...

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable, and test … logic Azure Key Vault for secure credential management Azure Blob Storage for data lake operations What We're Looking For Proven experience in Python, PySpark, and Delta Lake. SC Cleared Strong knowledge of Behave for test-driven development. Experience with Docker and containerised deployments. Familiarity with Azure cloud services ...

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable, and test … logic Azure Key Vault for secure credential management Azure Blob Storage for data lake operations What We're Looking For Proven experience in Python, PySpark, and Delta Lake. SC Cleared Strong knowledge of Behave for test-driven development. Experience with Docker and containerised deployments. Familiarity with Azure cloud services ...