1 to 25 of 325 PySpark Jobs

years of experience in Data Engineering , with a focus on cloud platforms (AWS, Azure, GCP); You have a proven track record working with Databricks (PySpark, SQL, Delta Lake, Unity Catalog); You have extensive experience in ETL/ELT development and data pipeline orchestration (e.g., Databricks Workflows, DLT, Airflow, ADF More ❯

/CD) automation, rigorous code reviews, documentation as communication. Preferred Qualifications Familiar with data manipulation and experience with Python libraries like Flask, FastAPI, Pandas, PySpark, PyTorch, to name a few. Proficiency in statistics and/or machine learning libraries like NumPy, matplotlib, seaborn, scikit-learn, etc. Experience in building More ❯

storage, data pipelines to ingest and transform data, and querying & reporting of analytical data. You've worked with technologies such as Python, Spark, SQL, Pyspark, PowerBI etc. You're a problem-solver, pragmatically exploring options and finding effective solutions. An understanding of how to design and build well-structured More ❯

Proven experience of ETL/ELT, including Lakehouse, Pipeline Design, Batch/Stream processing. Strong working knowledge of programming languages, including Python, SQL, PowerShell, PySpark, Spark SQL. Good working knowledge of data warehouse and data mart architectures. Good experience in Data Governance, including Unity Catalog, Metadata Management, Data Lineage More ❯

code development practices. Knowledge of Apache Spark and similar programming to support streaming data. Experience with some of the following Python libraries: NumPy, Pandas, PySpark, Dask, Apache Airflow, Luigi, SQLAlchemy, Great Expectations, Petl, Boto3, matplotlib, dbutils, koalas, OpenPyXL, XlsxWriter. Experience with monitoring and logging tools (e.g., Prometheus, Grafana, ELK More ❯

various departments to gather requirements and ensure data solutions reflect real business needs. Key Experience Required: Deep expertise in SQL, Python, and Spark (particularly PySpark) for building and testing end-to-end pipelines in that process both structured and semi-structured datasets. Experience mentoring peers and supporting team growth More ❯

FOSSA. • 3+ years of experience with data engineering tools and technologies, such as, Kubernetes, Container-as-a-Service (CaaS) platforms, OpenShift, DataProc, Spark (with PySpark) or Airflow. • Experience with CI/CD practices and tools, including Tekton or Terraform, as well as containerization technologies like Docker or Kubernetes. • Excellent More ❯

and support architectural decisions as a recognised Databricks expert. Essential Skills & Experience: Demonstrable expertise with Databricks and Apache Spark in production environments. Proficiency in PySpark, SQL, and working within one or more cloud platforms (Azure, AWS, or GCP). In-depth understanding of Lakehouse concepts, medallion architecture, and modern More ❯

of experience in Data Engineering, with a focus on cloud platforms (Azure, AWS, GCP). You have a proven track record working with Databricks (PySpark, SQL, Delta Lake, Unity Catalog). You have extensive experience in ETL/ELT development and data pipeline orchestration (Databricks Workflows, DLT, Airflow, ADF More ❯

platforms for your clients. Work with us to use big data for good. Qualifications You Have: 3+ years of experience using Python, SQL, and PySpark 3+ years of experience utilizing Databricks or Apache Spark Experience designing and maintaining Data Lakes or Data Lakehouses Experience with big data tools such More ❯

Delta Lake/Databricks), PL/SQL, Java/J2EE, React, CI/CD pipeline, and release management. Strong experience in Python, Scala/PySpark, PERL/scripting. Experience as a Data Engineer for Cloud Data Lake activities, especially in high-volume data processing frameworks, ETL development using distributed More ❯

Delta Lake/Databricks), PL/SQL, Java/J2EE, React, CI/CD pipeline, and release management. Strong experience in Python, Scala/PySpark, PERL/scripting. Experience as a Data Engineer for Cloud Data Lake activities, especially in high-volume data processing frameworks, ETL development using distributed More ❯

experience as a Senior Data Engineer, with some experience mentoring others Excellent Python and SQL skills, with hands-on experience building pipelines in Spark (PySpark preferred) Experience with cloud platforms (AWS/Azure) Solid understanding of data architecture, modelling, and ETL/ELT pipelines Experience using tools like Databricks More ❯

experience as a Senior Data Engineer, with some experience mentoring others Excellent Python and SQL skills, with hands-on experience building pipelines in Spark (PySpark preferred) Experience with cloud platforms (AWS/Azure) Solid understanding of data architecture, modelling, and ETL/ELT pipelines Experience using tools like Databricks More ❯

Azure Databricks Azure Function Apps & Logic Apps Azure Stream Analytics Azure Resource Manager tools: Terraform, Azure Portal, Azure CLI, and Azure PowerShell Proficient in PySpark, Delta Lake, Unity Catalog, and Python Ability to write unit and integration tests using unittest, pytest, etc. Solid understanding of software engineering principles, including More ❯

Azure Databricks Azure Function Apps & Logic Apps Azure Stream Analytics Azure Resource Manager tools: Terraform, Azure Portal, Azure CLI, and Azure PowerShell Proficient in PySpark, Delta Lake, Unity Catalog, and Python Ability to write unit and integration tests using unittest, pytest, etc. Solid understanding of software engineering principles, including More ❯

Knowledge, Skills and Experience: Essentia l Strong expertise in Databricks and Apache Spark for data engineering and analytics. Proficient in SQL and Python/PySpark for data transformation and analysis. Experience in data lakehouse development and Delta Lake optimisation. Experience with ETL/ELT processes for integrating diverse data More ❯

to store and process data. Document workflows, pipelines, and transformation logic for transparency. Key Skills & Experience: Strong hands-on experience in Python (Pandas, NumPy, PySpark). Experience building ETL/ELT processes. Familiarity with cloud platforms (AWS, Azure, GCP) and big data technologies (e.g., Snowflake, Databricks). Understanding of More ❯

to Octopus offices across Europe and the US. Our Data Stack: SQL-based pipelines built with dbt on Databricks Analysis via Python Jupyter notebooks Pyspark in Databricks workflows for heavy lifting Streamlit and Python for dashboarding Airflow DAGs with Python for ETL running on Kubernetes and Docker Django for More ❯

data analysis and writing and developing complex queries using SQL. Experience in designing Big Data/Cloud Solution Designs and data models. Knowledge of Pyspark, Shell scripting, SQL, Python, and some of the standard data science packages (Pandas, Numpy, etc.). Strong verbal and business communication skills. Experience in More ❯

scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place in Chantilly, VA, McLean, VA More ❯

scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place in Chantilly, VA, McLean, VA More ❯

scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place in Chantilly, VA, McLean, VA More ❯

validation, enrichment, deduplication and lineage. Experience in using a range of tools and languages to access and retrieve data from the Data Lake. Python, PySpark, SQL. Experience in implementing security and compliance with the Data Lakehouse to avoid unauthorized access, modification and leakage. Additional Information At Version 1, we More ❯

validation, enrichment, deduplication and lineage. Experience in using a range of tools and languages to access and retrieve data from the Data Lake. Python, PySpark, SQL. Experience in implementing security and compliance with the Data Lakehouse to avoid unauthorized access, modification and leakage. Additional Information At Version 1, we More ❯