51 to 75 of 223 Permanent PySpark Jobs

languages eg. Python, R, Scala, etc.; (Python preferred). Proficiency in database technologies eg. SQL, ETL, No-SQL, DW, and Big Data technologies e.g. pySpark, Hive, etc. Experienced working with structured and also unstructured data eg. Text, PDFs, jpgs, call recordings, video, etc. Knowledge of machine learning modelling techniques More ❯

s or PhD in a relevant field (e.g., Computer Science, Data Science, Engineering, Applied Mathematics, Statistics, etc.). Proficiency in Python, SQL, AWS, Airflow, PySpark, PyTorch, NumPy, and related data technologies. Experience with cloud infrastructure, data pipelines, and machine learning model deployment. Proven experience leading diverse teams of data More ❯

s or PhD in a relevant field (e.g., Computer Science, Data Science, Engineering, Applied Mathematics, Statistics, etc.). Proficiency in Python, SQL, AWS, Airflow, PySpark, PyTorch, NumPy, and related data technologies. Experience with cloud infrastructure, data pipelines, and machine learning model deployment. Proven experience leading diverse teams of data More ❯

Experience with regression, linear/non-linear optimization, classification and segmentation problems • Strong Python skills required, especially with Pandas, common machine learning libraries and PySpark • Strong SQL skills • Solid analytical skills, ability to derive conclusions and questions from data patterns • Efficient with git and Github. • Familiarity with Google Cloud More ❯

commercial experience is a plus Strong understanding of mathematical background, focusing on statistics and linear algebra Highly proficient in Python (Pandas, Scikit-Learn, PyTorch, PySpark) and SQL Experience with Snowflake (function & procedure) and Snowpark is a plus Experience with unit and integration tests Strong understanding of machine learning algorithms More ❯

Hue, Zookeeper, HCatalog, Solr, Avro, Parquet, Iceberg, Hudi). Experience developing software and data engineering code in one or more programming languages (Java, Python, PySpark, Node, etc). AWS and other Data and AI aligned Certifications. Ability to think strategically about business, product, and technical challenges in an enterprise More ❯

experience with data warehousing, ETL/ELT, integration tools, and BI solutions. Expertise with Synapse (notebooks and data flows) is essential. Working knowledge of PySpark is essential. Experience with Azure Data Lake Storage is essential. Understanding of medallion data lakehouse architecture is essential. Strong problem-solving skills for complex More ❯

pipeline. Requirements Minimum Qualifications: - 7+ years of professional experience in Python development and data processing. - Deep expertise with Python data processing libraries such as PySpark, Pandas, and NumPy. - Strong experience with API development using FastAPI or similar frameworks. - Proficiency in test-driven development using PyTest and mocking libraries. - Advanced More ❯

will be at the forefront of processing ETL on new and varied data sets and working with Big Data at scale, using Spark/PySpark to handle petabytes of data. Key Responsibilities: • Customer Liaison: Develop a deep understanding of the customer's current and upcoming mission needs, generating requirements More ❯

this position. Qualifications: To be considered for this role, you must have: Active Secret clearance or higher (required). Strong proficiency in Python, SQL, PySpark, and Java. At least 5 years of hands-on experience in pipeline engineering or data engineering roles. Demonstrated ability to perform root cause analysis More ❯

Chantilly (both locations are in close proximity to exits of Rt 28). Required Skills: Extract, Transform and Load (ETL) tools and processes Python, Pyspark, Pytorch AWS SQL APIs Linux Geospatial tools/data Desired Skills: Agile experience delivering on agile teams (Participates in scrum and PI Planning) Docker More ❯

National Geospatial-Intelligence Agency (NGA) to process ETL on new and varied data sets, and work with Big Data at scale, using Spark/PySpark to handle petabytes of data. As an immediate liaison with the customer, you will have a deep understanding of their current and upcoming mission More ❯

required. A master's degree in a related field and 10+ years' relevant experience. Data Architecture experience. Well-versed in Python and Spark/PySpark Experience building and managing data pipelines and using data analytics tools such as Databricks, Qlik, Power BI Experience working via the terminal/command More ❯

does not apply) At least 1 year of experience in big data technologies Preferred Qualifications: 5+ years of experience in application development including Python, Pyspark or Scala 2+ years of experience with AWS 3+ years experience with Distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi More ❯

experience in coding languages e.g. Python, C++, etc.; (Python preferred). Proficiency in database technologies e.g. SQL, No-SQL and Big Data technologies e.g. pySpark, Hive, etc. Experience working with structured and unstructured data e.g. Text, PDFs, jpgs, call recordings, video, etc. Knowledge of machine learning modelling techniques and More ❯

or a related quantitative field - Experience with scripting languages (e.g., Python, Java, R) and big data technologies/languages (e.g. Spark, Hive, Hadoop, PyTorch, PySpark) PREFERRED QUALIFICATIONS - Master's degree, or Advanced technical degree - Knowledge of data modeling and data pipeline design - Experience with statistical analysis, co-relation analysis More ❯

using Azure services including Azure Databricks, Azure Data Factory, Delta Lake, Azure Data Lake (ADLS), Power BI. Solid hands-on experience with Azure Databricks - Pyspark coding and Spark SQL coding - Must have. Very good knowledge of data warehousing skills including dimensional modeling, slowly changing dimension patterns, and time travel. More ❯

for seamless data workflows. Collaborate with cross-functional teams to ensure data integrity, security, and accessibility. Key Skills & Experience: Strong programming skills in Python (PySpark). Hands-on experience with Azure Data Services (Azure Data Factory, Databricks, Synapse, Data Lakes, etc.). Experience with CI/CD pipelines for More ❯

transformation logic, and build of all Power BI dashboards - including testing, optimization & integration to data sources Good exposure to all elements of data engineering (PySpark, Lakehouses, Kafka, etc.) Experience building reports from streaming data Strong understanding of CI/CD pipeline Financial and/or trading exposure, particularly energy More ❯

transformation logic, and build of all Power BI dashboards - including testing, optimization & integration to data sources Good exposure to all elements of data engineering (PySpark, Lakehouses, Kafka, etc.) Experience building reports from streaming data Strong understanding of CI/CD pipeline Financial and/or trading exposure, particularly energy More ❯

ML teams. Strong experience with generative models (e.g. GPT, LLaMA, Diffusion models) applied to real-world use cases. Proficiency in Python and SQL (bonus: PySpark , NoSQL ). Experience with RAG , prompt engineering , fine-tuning , and embeddings . Familiarity with GenAI platforms like OpenAI , Hugging Face , and Google Vertex AI More ❯

experience or PhD with 5-7 years of relevant experience Experience in research Data Engineering Experience (DE Path) ETL Big Data Experience and Tooling (PySpark, Databricks) Python Testing Frameworks Data Validation and Data Quality Frameworks Data Handling (SQL & NoSQL) Feature Engineering Chunking Document Ingestion Graph Data Structures (Neo4j) CI More ❯

least 8 years of experience within the field. The Role: Technical Skills: The candidate should be a very strong developer with extensive experience in Pyspark, Python, DBX, and Azure DE stack (ADF, Logic apps, DevOps - CI/CD). Certification in Databricks or Microsoft (advanced level) is highly preferable. More ❯

Ensure compliance with GDPR and other data regulations when handling sensitive information. Support the stability and performance of enterprise data platforms. Requirements: Proficient with PySpark, Delta Lake, Unity Catalog and Python (including unit and integration testing). Deep understanding of software development principles (SOLID, testing, CI/CD, version More ❯

external data engineers and data scientists, who may not be familiar with the datasets, to accelerate development. Our technology stack consists of: Python and Pyspark AWS glue jobs assembled into Step functions Pydeequ for data quality testing Circle CI for deployment Amazon Athena for querying data Hosted on AWS More ❯