51 to 75 of 165 PySpark Jobs

robust in-house team, this role could be the next step for you. Data Engineer Responsibilities: Design, build, and optimise data pipelines in Python, PySpark, SparkSQL, and Databricks. Ingest, transform, and enrich structured, semi structured, and unstructured data. Operate and support production grade data systems with strong observability ...

Required Skills & Experience Proven experience in MLOps or DevOps roles within machine learning environments Strong programming skills in Python, with hands-on experience in PySpark and SQL Deep understanding of ML lifecycle management and CI/CD best practices Familiarity with cloud-native ML platforms and scalable deployment strategies ...

years commercial software engineering experience, with a strong focus on data-engineering at scale Strong experience building robust & scalable data pipelines using Python and PySpark A deep knowledge of Schema Design, SQL and database optimizations Experience of being a bar-raiser and helping improve engineering best practices Experience ...

product teams who'll challenge them in the best ways The ideal candidate will: Know their way around AWS data tools (Glue/PySpark, Athena) and Microsoft Fabric Be able to write clean Python and SQL in their sleep Have battle scars from integrating CRMs (HubSpot, Salesforce) via APIs ...

tools such as JUnit, Spock or Clojure.test. Scripting skills in Bash, Ruby, or Python. Some technologies and processes we use: Clojure, Go, Python Databricks, PySPark Kafka, Kafka Streams Relational DB, Document Stores (e.g. ElasticSearch) GCP, AWS Kubernetes, Cloud Native CI/CD, CircleCI, GitOps, FluxCD ...

Responsibilities Design, develop, and optimize end-to-end data pipelines using Azure services Build and maintain scalable ETL/ELT workflows using Azure Databricks (PySpark/SQL) Develop and manage data warehouses and analytics solutions using Azure Synapse Analytics Ingest data from multiple sources (APIs, databases, files, streaming sources … pipelines, and operational processes Required Qualifications 3+ years of experience as a Data Engineer or in a similar role Strong experience with Azure Databricks (PySpark, Spark SQL) Hands-on experience with Azure Synapse Analytics (dedicated and/or serverless pools) Solid understanding of data warehousing concepts and dimensional modeling ...

Responsibilities Design, develop, and optimize end-to-end data pipelines using Azure services Build and maintain scalable ETL/ELT workflows using Azure Databricks (PySpark/SQL) Develop and manage data warehouses and analytics solutions using Azure Synapse Analytics Ingest data from multiple sources (APIs, databases, files, streaming sources … pipelines, and operational processes Required Qualifications 3+ years of experience as a Data Engineer or in a similar role Strong experience with Azure Databricks (PySpark, Spark SQL) Hands-on experience with Azure Synapse Analytics (dedicated and/or serverless pools) Solid understanding of data warehousing concepts and dimensional modeling ...

build, and operationalize data and ML pipelines using AWS Glue, Step Functions, Lambda, EMR, and SageMaker. Write robust AWS Glue ETL jobs (Python/PySpark) with exception handling, logging, and CloudWatch monitoring. Implement monitoring and dashboards in CloudWatch for ETL processes, data pipelines, and ML endpoints. Work with PostgreSQL ...

data stack, including Databricks and Data Factory, to help us build and optimise cutting-edge solutions. You'll bring strong programming skills in Python, PySpark, and SQL, along with proven experience in designing data models, developing pipelines, and implementing modern data architecture. Your ability to translate complex technical concepts ...

services, data modeling, ETL/ELT development, and collaborative engineering practices. Key Responsibilities * Design, develop, and maintain scalable data pipelines using Azure Databricks (Python, PySpark, SQL). * Build and optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services … standards and best practices. Required Skills and Experience * Proven experience as a Data Engineer working with Azure cloud services. * Strong proficiency in Databricks, including PySpark, Spark SQL, notebooks, Delta Lake, and job orchestration. * Strong SQL and data modeling skills (e.g., dimensional modeling, data vault). * Experience with Azure Data ...

enabling advanced analytics and data-driven decision making across the business. Key Responsibilities Design, develop, and maintain ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. Build and optimise data lakehouse architectures on Azure Data Lake Storage ( ADLS ) . Develop high-performance data solutions using Azure Synapse … Actions, or Azure DevOps Required Skills & Experience 3+ years' experience as a Data Engineer working in Azure environments. Strong hands-on experience with Databricks (PySpark, Delta Lake, cluster optimisation, job scheduling). Solid knowledge of Azure cloud services including: Azure Data Lake Storage Azure Data Factory Azure Synapse/ ...

training, inference, monitoring, and iteration. Strong background of ML/DL/LLM algorithms, model architectures, and training techniques. Proficiency in Python, SQL, Spark, PySpark, TensorFlow or other analytical/model-building programming languages. Proficiency with tools and LLMs. Ability to work independently and collaboratively within a team. Preferred ...

Active Security Clearance (SC) Strong Python skills with modular, test-driven design Experience with Behave for unit and BDD testing (mocking, patching) Proficiency in PySpark and distributed Data processing Solid understanding of Delta Lake (design and maintenance) Hands-on with Docker for development and deployment Familiarity with Azure services ...

Active Security Clearance (SC) Strong Python skills with modular, test-driven design Experience with Behave for unit and BDD testing (mocking, patching) Proficiency in PySpark and distributed Data processing Solid understanding of Delta Lake (design and maintenance) Hands-on with Docker for development and deployment Familiarity with Azure services ...

trading and risk management. Document system architecture, data flows, and technical processes for transparency and reproducibility. Requirements Strong proficiency in Python (pandas, NumPy, PySpark) and ETL development. Hands-on experience with AWS services (S3, Glue, Lambda) and Databricks. Solid understanding of financial market data, particularly time-series. Knowledge ...

issues Contribute to new Finance Engineering initiatives What We’re Looking For Excellent programming skills in Python and strong working knowledge of Pandas and PySpark libraries Excellent knowledge of relational databases and SQL Experience with integration of multiple platforms Experience with ETL processes Experience providing first-line support ...

databases and AWS cloud environments. Strong hands-on experience with ETL processes and tools (Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi) or scripting using Python, PySpark, and SQL. Solid understanding of data warehousing and modelling techniques (Star Schema, Snowflake Schema). Familiarity with security frameworks such as GDPR, HIPAA ...

Strong knowledge of AWS data services (Glue, S3, Lambda, Redshift, etc.) Solid understanding of ETL processes and data pipeline management Proficiency in Python and PySpark Experience working with SQL-based platforms Previous involvement in migrating on-premise solutions to cloud is highly desirable Excellent collaboration skills and ability ...

governance , schema validation, and master data management. Strong grasp of real-time vs batch processing trade-offs . Beneficial Experience CockroachDB deployment and management. PySpark or similar for large-scale processing. SAP data structures and PIM systems. E-commerce and B2B data integration patterns. Why Apply? Fully remote contract ...

governance , schema validation, and master data management. Strong grasp of real-time vs batch processing trade-offs . Beneficial Experience CockroachDB deployment and management. PySpark or similar for large-scale processing. SAP data structures and PIM systems. E-commerce and B2B data integration patterns. Why Apply? Fully remote contract ...

mentoring experience, with a genuine commitment to team growth and wellbeing. Strong hands-on skills in: AWS (or equivalent cloud platforms) Python/PySpark for data engineering and automation TypeScript, Node.js, React.js for full-stack development Solid grasp of distributed systems design, secure coding, and data privacy principles. Familiarity ...

data governance, security, and compliance. Key Skills & Experience* Proven experience as an Azure Data Engineer.* Strong hands-on expertise with Databricks - 5+ years experience (PySpark, notebooks, clusters, Delta Lake).* Solid knowledge of Azure services (Data Lake, Synapse, Data Factory, Event Hub).* Experience working with DevOps teams ...

Databricks-based data warehousing solution Designing and developing data models, ETL pipelines, and data integration processes Large scale data processing using PySpark Monitoring, tuning, and optimising data platforms for reliability and performance Upskilling the wider team in Databricks best practices, including modern architecture patterns Location: Atherstone (Hybrid – 3 days ...

client requirement) Job Summary We are seeking a highly skilled Senior Data Engineer with strong expertise in AWS cloud services, Python, and PySpark to join our team in Columbus, OH. The ideal candidate will design, build, and optimize scalable data pipelines, ensuring high data quality, performance, and reliability. This … Data Engineer. Strong hands-on expertise in AWS Data Services: Glue, EMR, Redshift, Athena, S3, Lambda, Step Functions, CloudWatch. Proficiency in Python and PySpark for ETL, transformation, and automation. Strong experience in SQL and data modeling (star schema, snowflake schema, dimensional models Experience building scalable, high-performance data pipelines ...

scalable solutions. Proficiency in ETL between different RDBMSs (e.g., Oracle) to Databricks and NoSQL databases Experience working with Mongo DB Working knowledge in Python, PySpark, and Scala Design and implement scalable data processing solutions using Azure Databricks Collaborate with data scientists and data analysts to understand data needs Optimize ...