101 to 125 of 222 PySpark Jobs

complex data led organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design ...

complex data led organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design ...

solution design, workshops and Proof of Concepts. What you'll bring: Proven experience designing enterprise data platforms using Microsoft Fabric. Strong knowledge of Spark, PySpark, Delta Lake, SQL and modern data engineering. Expertise with Microsoft Azure Excellent stakeholder management and consultancy experience. What's in it for you? 📍Remote ...

science team. What we're looking for Proven experience delivering end-to-end data science solutions in a commercial environment. Strong Python, SQL and PySpark skills. Experience working with structured and unstructured data, applying NLP and GenAI techniques. Experience deploying models into the cloud (Azure preferred) with CI/ ...

Azure Data Platform and Databricks ecosystem. The ideal candidate should have strong hands-on experience with Structured Streaming, Delta Live Tables (DLT), PySpark architecture, Python, and SQL, and a proven track record of delivering large-scale enterprise data platforms. Youll be responsible for: Leading the design, development, and implementation ...

Skills – Able to extract meaningful insights from complex datasets and demonstrate clear business impact. Technical Expertise – Strong experience with SQL, Power BI, Python/PySpark; Databricks is a plus but not essential. Stakeholder Management – Confident in engaging with senior stakeholders, translating data into compelling narratives that drive business decisions. ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation.Experience with Databricks and PySpark for large-scale data processing.Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring.Experience implementing responsible AI practices ...

datasets for modelling and analysis. Strong background in NLP and GenAI techniques, including text classification, summarization, and embeddings. Advanced proficiency in SQL, Python, and PySpark, with a focus on software development best practises. Experience deploying data science models in a cloud environment (Azure preferred). Familiarity with database management ...

working partners.Technical skills preferred in any or all of the following: Microsoft Excel, Google Suite, Atlassian Jira/Confluence, SQL, Python (Jupyter Notebook, Pandas, PySpark, etc.), Databricks, Bloomberg, Pitchbook, or other common financial services systems and applications.Our Values Act Like an Owner - Think and operate with intention, purpose ...

also contribute to engineering excellence through best practices, mentoring, and thoughtful technical direction.Job responsibilitiesDesign scalable, reusable data processing and data quality frameworks using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow … data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence and reporting workloadsWrite production-grade Python/PySpark code with disciplined testing, performance tuning, and maintainable object-oriented designImplement infrastructure-as-code (e.g., Terraform) to provision and manage cloud-based data platform ...

responsibilitiesDesign and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators (e.g., Workflows, Airflow, ADF).Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency.Build and maintain curated data models (bronze/silver/gold), data quality checks … realized by automationRequired qualifications, skills and capabilitiesExperience with building data pipelines on Databricks and/or Apache Spark in production.Strong coding skills in Python (PySpark) and SQL (Scala a plus).Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM).Experience ...

Senior Data Engineer Who we areArtefact is a new generation of data service provider, specialising in data consulting and data-driven digital marketing, dedicated to transforming data into business impact across the entire value chain ...

communicate future challenges, ensuring operational effectiveness. Skills and Experience: Essential: Full-stack Python Software Developer skills, specialising in numerical processing using Pandas, Numpy, Polars, PySpark, Coroutines, async tasks and Unix Building scalable applications, with intensive risk-based processing. Package software for distribution (eg Docker, Poetry, python-wheels, PyInstaller, Databricks ...

continuous integration/continuous delivery, application resiliency, and securityDemonstrated knowledge of software applications and technical processes within a technical discipline (e.g., cloud, Databricks, PySpark, artificial intelligence, machine learning, mobile, etc.)Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, testing ...

Data Manager/Lead Location : London (Hybrid) Comp : £115,000 P/A (FTC) Duration: 12 Months Skills: Python (Programming Language), SQL, Apache Spark, PySpark, Azure Databricks, Data Lakes, Data Warehouse Architecture, Leadership, Software Development, Management The role We're partnering with a large, data-driven organisation seeking ...

excellence. Champion AI and data architecture as strategic enablers of business transformation—linking technology decisions directly to measurable client outcomes. Technologies & Concepts Programming & APIs: PySpark , Python, FastAPI , FastMCP , SQL, OAuth2 AI Architectures: Retrieval-Augmented Generation, In-Context Learning, Agent Design, Function Calling, LLM Fine-Tuning & Adaptation Strategies Data Analytics ...

experience using Pandas. Experience working with SQL and cloud-based data warehousing technologies. Knowledge of Google BigQuery would be highly beneficial. Exposure to PySpark and distributed data processing environments. Ability to analyse, transform, and optimise complex datasets. Strong problem-solving and stakeholder communication skills. How to Apply ...

passion for solving complex data challengesWhile we advocate using the right tech for the right task, we often leverage the following technologies: Python, PySpark, the PyData stack, SQL, Airflow, Databricks, Kedro (our open-source data pipelining framework), Dask/RAPIDS, Docker, Kubernetes, and cloud solutions such ...

They are also a key driver in translating Vision and Data Strategy into IT solutions and delivering on them.Strong hands-on expertise in Databricks, PySpark, Python and SQL is central to this role.Key Skills:Azure StackELT ExperienceData ModelingData Integration & IngestionData Manipulation and ProcessingGitHub, GitHub Actions, Azure DevOpsData Factory, Databricks ...

workflow orchestration, and machine learning operations.Excellent communication skills with the ability to influence cross-functional teams and stakeholders.Experience with Python, SQL, Java, HQL, Hive, Pyspark, or similar programming languages preferred.Well-versed in design thinking principles.Great problem-solving and analytical skills. The total cash range for this position in Seattle ...

machine learning operations. Excellent communication skills with the ability to influence cross-functional teams and stakeholders. Experience with Python, SQL, Java, HQL, Hive, Pyspark, or similar programming languages preferred. Well-versed in design thinking principles. Great problem-solving and analytical skills. The total cash range for this position ...

efficient, and user-friendly reports that serve a demanding user base. * Creating and automating analytical pipelines using Power BI (DAX, Power Query), SQL, and PySpark to optimise data processing and reporting. * Integrating diverse data sources into a central environment, providing real-time insights into customer, financial, and operational metrics. ...

DESIRABLE Understanding of SQL and relational datasets Experience building dashboards in SaaS (software as a service) products Knowledge of DAX, Power Query Knowledge of Pyspark Experience supporting live dashboards and reporting environments Experience automating reporting solutions Experience of other reporting and dashboard tools (Databricks, Qlik, Tableau, Looker, Amazon ...

implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency. Build and maintain curated data models (bronze/silver/gold), data quality checks … Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM ...

implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency. Build and maintain curated data models (bronze/silver/gold), data quality checks … Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM ...