101 to 125 of 137 Permanent PySpark Jobs

team members on a regular basis • Adhere to all established policies regarding security, completion of mandatory training courses and timesheet submission Required Experience: • Experience developing in Python • Experience with PySpark application programming interface (API) to process large datasets in a distributed cluster • Ability to efficiently balance simultaneous, high-profile projects and deliver those projects in an effective manner is More ❯

etc. Knows when and why to use specific algorithms depending on the business use case. Can fine-tune/train models and evaluate performance. Technical Skills: Proficient in Python, PySpark, and working with MongoDB JSON data. Must be comfortable working directly with raw/complex datasets and drawing insights. Experience in model selection based on data exploration. Bonus (Not More ❯

technical concepts to a range of audiences. Able to provide coaching and training to less experienced members of the team. Essential skills: Programming Languages such as Spark, Java, Python, PySpark, Scala (minimum of 2). Extensive Data Engineering hands-on experience (coding, configuration, automation, delivery, monitoring, security). ETL Tools such as Azure Data Fabric (ADF) and Databricks or … UK, and you MUST have the Right to Work in the UK long-term without the need for Company Sponsorship. KEYWORDS Senior Data Engineer, Coding Skills, Spark, Java, Python, PySpark, Scala, ETL Tools, Azure Data Fabric (ADF), Databricks, HDFS, Hadoop, Big Data, Cloudera, Data Lakes, Azure Data, Delta Lake, Data Lake, Databricks Lakehouse, Data Analytics, SQL, Geospatial Data, FME More ❯

mathematic models, methods, and/or techniques (e.g. algorithm or development) to study issues and solve problems, engineering (electrical or computer), and/or high performance computing Preferred Python & pySpark experience More ❯

role for you. Key Responsibilities: Adapt and deploy a cutting-edge platform to meet customer needs Design scalable generative AI workflows (e.g., using Palantir) Execute complex data integrations using PySpark and similar tools Collaborate directly with clients to understand their priorities and deliver impact Why Join? Be part of a mission-driven startup redefining how industrial companies operate Work More ❯

role for you. Key Responsibilities: Adapt and deploy a cutting-edge platform to meet customer needs Design scalable generative AI workflows (e.g., using Palantir) Execute complex data integrations using PySpark and similar tools Collaborate directly with clients to understand their priorities and deliver impact Why Join? Be part of a mission-driven startup redefining how industrial companies operate Work More ❯

and maintain efficient ETL/ELT data pipelines using Microsoft Fabric and its components, including Fabric Pipelines and Spark jobs. Notebook-based Transformation: Write, test, and optimize Python notebooks (PySpark) within the Fabric/Synapse environment to perform complex data ingestion, cleansing, and transformation tasks. Data Warehouse and Lakehouse Management: Implement and manage data storage solutions, leveraging Azure Data … Microsoft Azure ecosystem. Strong hands-on experience with Microsoft Fabric, specifically its data engineering workloads (Notebooks, Pipelines, and Lakehouses). Demonstrable expertise in Azure Synapse Analytics, including Synapse Notebooks (PySpark) and SQL pool resources. Proficiency in Python for data manipulation, transformation, and automation. Experience in building and maintaining ETL/ELT pipelines using Azure Data Factory and Fabric Pipelines. More ❯

technical concepts to a range of audiences. Able to provide coaching and training to less experienced members of the team. Essential Skills: Programming Languages such as Spark, Java, Python, PySpark, Scala or similar (minimum of 2). Extensive Big Data hands-on experience across coding/configuration/automation/monitoring/security is necessary. Significant AWS or Azure … the Right to Work in the UK long-term as our client is NOT offering sponsorship for this role. KEYWORDS Lead Data Engineer, Senior Data Engineer, Spark, Java, Python, PySpark, Scala, Big Data, AWS, Azure, Cloud, On-Prem, ETL, Azure Data Fabric, ADF, Hadoop , HDFS , Azure Data, Delta Lake, Data Lake Please note that due to a high level More ❯

Lead Data Engineer (Databricks) - Leeds (Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer, Python, PySpark, SQL, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI/CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer) Our … modern data platform using cutting-edge technologies, architecting big data solutions and developing complex enterprise data ETL and ML pipelines and projections. The successful candidate will have strong Python, PySpark and SQL experience, possess a clear understanding of databricks, as well as a passion for Data Science (R, Machine Learning and AI). Database experience with SQL and No … Benefits To apply for this position please send your CV to Nathan Warner at Noir Consulting. (Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer, Python, PySpark, SQL, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI/CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Lead Data More ❯

role for you. Key Responsibilities: Adapt and deploy a cutting-edge platform to meet customer needs Design scalable generative AI workflows (e.g., using Palantir) Execute complex data integrations using PySpark and similar tools Collaborate directly with clients to understand their priorities and deliver impact Why Join? Be part of a mission-driven startup redefining how industrial companies operate Work More ❯

in designing and developing interactive dashboards and reports using BI tools like Oracle Business Intelligence Enterprise Edition (OBIEE) and Oracle Analytics Cloud (OAC).Proficiency in programming languages such as Pyspark and PL/SQL for data processing and automation.Have a strong understanding of Azure cloud platform and its services for deploying and managing cloud-based solutions.Experience with Cloud services … such as Azure Data Services, ADLS and AKSExperience with Python and PySpark for distributed data processing, along with proficiency in Numpy, Pandas and other data manipulation libraries.Experience in optimizing big data architectures for high availability and performance.Strong problem-solving skills, analytical mindset, and ability to work in fast-paced environments.Experience with the Incorta platform, including its Direct Data Mapping … support related documents and maintain & upgrade the existing ones.Discuss with business users for new requirements and prepare Functional & Technical Specification Documents.Build, optimize and manage large scale data pipelines leveraging pyspark and spark clusters.Deploy, Orchestrate and manage containerized workloads using Azure Kubernetes services (AKS) for distributed data processing and analytics.Design and manage Azure IaaS resources for high performance data processing. More ❯

engineering practices. Experience with ETL development in data lake and data warehouse (preferably Snowflake) Exhibited hands-on experience in building data pipelines and reusable components using AWS Services and PySpark, and Snowflake within the last two years Exposure to Databricks Unity Catalog and UniForm is a plus The ability to deliver work at a steady, predictable pace to achieve More ❯

engineering practices. Experience with ETL development in data lake and data warehouse (preferably Snowflake) Exhibited hands-on experience in building data pipelines and reusable components using AWS Services and PySpark, and Snowflake within the last two years Exposure to Databricks Unity Catalog and UniForm is a plus The ability to deliver work at a steady, predictable pace to achieve More ❯

/long-term Interview Criteria: Telephonic + Zoom Direct Client Requirement Role: AWS Data Engineer We are seeking a skilled AWS Data Engineer who has experience working with Python, PySpark, lambda, Airflow, and Snowflake.Responsibilities: Design, build, and optimize ETLs using Python, PySpark, lambda, Airflow and other AWS services. Create SQL queries to segment, manipulate, and formatdata. Build automations … and maintain ETL/ELT pipelines to ingest data into Amazon Redshift for analytics and reporting. Requirements: Minimum 5 years of experience as Data Engineer. 3+ years of Python, PySpark, and Lambda. Must have experience with Airflow and Snowflake. Advanced SQL query development proficiency Understanding of data modelling principles and techniques. Knowledge of data security best practices and compliance More ❯

specialists, contributing to the development and maintenance of advanced data pipelines and supporting various analytical initiatives. Responsibilities: • Assist in the development and maintenance of data pipelines using Spark, Scala, PySpark, and Python. • Support the deployment and management of AWS services including EC2, S3, and IAM. • Work with the team to implement and optimize big data processing frameworks such as … equivalent practical experience. • Basic knowledge of Spark and Hadoop distributed processing frameworks. • Familiarity with AWS services, particularly EC2, S3, and IAM. • Some experience with programming languages such as Scala, PySpark, Python, and SQL. • Understanding of data pipeline development and maintenance. • Strong problem-solving skills and the ability to work collaboratively in a team environment. • Eagerness to learn and grow More ❯

scalable data pipelines and services. (Hybrid 2 days office - Dublin) Responsibilities Architecture leadership: Define roadmaps, set standards, choose tools, review designs. Hands-on engineering: Still coding in Python/PySpark, designing pipelines, embedding CI/CD. Mentorship: Guide and review other engineers' work. Security ownership: Especially around AWS IAM roles and least privilege access. Governance: Data models, metadata, documentation … Collaboration: Work closely with global colleagues; communicate designs clearly. Requirements AWS data stack, hands on experience: Glue, Airflow/MWAA, Athena, Redshift, RDS, S3. Strong coding skills in Python & PySpark/Pandas & SQL. CI/CD automation: AWS CodePipeline, CloudFormation (infrastructure as code). Architect-level experience: At least 5 years in a senior/lead role. Security-first More ❯

Technology Architect - Data Engineering (Hybrid - London) Contract: 6 months Work Mode: Hybrid (12 days WFO/month)We are looking for an experienced Technology Architect with deep expertise in PySpark, ADF, and Databricks to lead and design data engineering solutions for our client. What You'll Do: Lead technical design using Medallion architecture and Azure ServicesCreate conceptual diagrams, source … pipelines Collaborate effectively with team members and stakeholdersOptional: Work with Log Analytics and KQL queries Must-Have Skills: 10+ years of experience in Data Engineering Hands-on experience with PySpark, ADF, Databricks, SQL Strong understanding of dimensional modeling, normalization, schema design, and data harmonization Experience with Erwin and data modeling toolsExcellent communication, problem-solving, and client-facing skills Why More ❯

Our financial services client is seeking a Database Developer for an exciting direct hire opportunity. DUTIES:Design and develop ETL processes to transform a variety of raw data, flat files and excel spreadsheets into SQL databases. Develop and optimize queries More ❯

role: Adapt and deploy a powerful data platform to solve complex business problems Design scalable generative AI workflows using modern platforms like Palantir AIP Execute advanced data integration using PySpark and distributed technologies Collaborate directly with clients to understand priorities and deliver outcomes What We're Looking For: Strong skills in PySpark, Python, and SQL Ability to translate More ❯

role: Adapt and deploy a powerful data platform to solve complex business problems Design scalable generative AI workflows using modern platforms like Palantir AIP Execute advanced data integration using PySpark and distributed technologies Collaborate directly with clients to understand priorities and deliver outcomes What We're Looking For: Strong skills in PySpark, Python, and SQL Ability to translate More ❯

equivalent combination of education, technical training, or work/military experience Over 5 years of hands-on Pentaho experience Over 2 years of software development experience in JavaScript, Python, PySpark, or other object-oriented development languages Required Skills: 1) Min 5 years of solid Pentaho-specific ETL development experience 2) Min 2 years of Python and PySpark (Pandas More ❯

leaders, working at the intersection of cutting-edge technology and real-world impact. As part of this role, you will be responsible for: Executing complex data integration projects using PySpark and distributed technologies Designing and implementing scalable generative AI workflows using modern AI infrastructure Collaborating with cross-functional teams to ensure successful delivery and adoption Driving continuous improvement and … innovation across client engagements To be successful in this role, you will have: Experience working in data engineering or data integration Strong technical skills in Python or PySpark Exposure to generative AI platforms or interest in building AI-powered workflows Ability to work closely with clients and lead delivery in fast-paced environments Exposure to Airflow, Databricks or DBT More ❯

leaders, working at the intersection of cutting-edge technology and real-world impact. As part of this role, you will be responsible for: Executing complex data integration projects using PySpark and distributed technologies Designing and implementing scalable generative AI workflows using modern AI infrastructure Collaborating with cross-functional teams to ensure successful delivery and adoption Driving continuous improvement and … innovation across client engagements To be successful in this role, you will have: Experience working in data engineering or data integration Strong technical skills in Python or PySpark Exposure to generative AI platforms or interest in building AI-powered workflows Ability to work closely with clients and lead delivery in fast-paced environments Exposure to Airflow, Databricks or DBT More ❯

About The Role We are looking for an experienced Test Manager with a strong background in data platform testing, particularly across Microsoft Fabric, Azure services, PySpark notebooks, and automated testing frameworks. You will play a pivotal role in ensuring quality and governance across data pipelines, lakehouses, and reporting layers. This role requires a hands-on leader who can define … Microsoft Fabric workloads (Lakehouses, Pipelines, Notebooks, Power BI). Lead and manage the QA effort across multiple agile teams. Drive the development and maintenance of automated testing for: 1-PySpark notebooks in Fabric/Databricks 2-Data pipelines and transformations 3-Delta tables and lakehouse validation Embed testing into CI/CD pipelines (Azure DevOps or GitHub Actions). More ❯

role: Adapt and deploy a powerful data platform to solve complex business problems Design scalable generative AI workflows using modern platforms like Palantir AIP Execute advanced data integration using PySpark and distributed technologies Collaborate directly with clients to understand priorities and deliver outcomes What We're Looking For: Strong skills in PySpark, Python, and SQL Ability to translate More ❯