76 to 100 of 150 Remote/Hybrid PySpark Jobs

Microsoft Azure. You will build production-grade data pipelines with Delta Lake, Unity Catalog, and Databricks tools. The role requires hands-on expertise in PySpark, Python, SQL, and Spark architecture, plus experience with GitLab-based CI/CD, Databricks Asset Bundles, and Azure Data Lake Storage. #J-18808-Ljbffr ...

troubleshooting of Azure DatabricksDatabricks Serverless configuration and workload optimisationDatabricks SQL and Delta Lake developmentCluster management, compute policies, autoscaling, and workload monitoringData EngineeringStrong Python, PySpark, and SQL development skillsDesign and build ingestion and transformation pipelinesFull and incremental data loadingCDC (Change Data Capture) implementationData quality controls, schema evolution, reconciliation, and error ...

understanding of DAG (Directed Acyclic Graphs) and their role in pipeline orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic ...

experience with Azure AI Search or other vector/hybrid search solutions, including vector search, hybrid search, and retrieval evaluation.Experience with Databricks and PySpark for large-scale data processing.Experience designing evaluation frameworks for LLM systems, including offline metrics, hallucination detection, and production monitoring.Experience implementing responsible AI practices ...

Delta Lake, with governance, security, and real-time processing at scale. The role sits in the Enterprise Data Team and requires strong SQL and PySpark, plus experience with cloud-native data platforms. Hybrid working available, with 2 days in the office and 3 days from home. #J-18808-Ljbffr ...

functional teams to deliver high-quality technical solutions. Support the ongoing maintenance and optimisation of platform services. Essential Skills & Experience Hands-on experience with PySpark for data transformation and processing. Experience working with AWS data services , including: AWS Glue Amazon S3 Experience using Terraform for Infrastructure as Code. Experience ...

Declarative Pipelines with Expectations, including failure handling and observability Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns Production-level PySpark and SQL development skills Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM Experience with Databricks Workflows, orchestration ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

will be doing Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into … into engineered solutions What we need from you 8+ years in data engineering, including 3+ years hands on with Azure Databricks Strong Python and PySpark, backed by advanced SQL Deep grounding in data warehousing, data modelling and integration patterns Unity Catalog and Delta Lake in a production environment Data ...

Engineer to join the Data & Analytics team. The role is hybrid, focusing on building scalable data pipelines on Databricks and AWS, with Python/PySpark expertise. You will design data architectures, work with data warehousing concepts and contribute to analytics and AI initiatives, while collaborating with stakeholders ...

with a leading cloud-based software innovator to shape data analytics, AI-driven automation and scalable data pipelines. You will craft Python, SQL, and PySpark solutions, design analytics architectures, and validate AI-assisted outputs to guide business decisions. The role blends data engineering, analytics, and responsible AI practices ...

turn data into revenue. The role combines hands-on development with architecture decisions in a hybrid London setting. You will work with Python (pyspark), AWS, Databricks and dbt/Fivetran to deliver scalable data pipelines, apply governance, and drive #J-18808-Ljbffr ...

Kubernetes Experience in creating production grade coding and SOLID programming principles, including test-driven development (TDD) approaches Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Experience in source-control software, e.g., GitHub Proficient at communicating results in a concise manner both verbally and written Experience in data ...

production settings Experience with multimodal modelling (e.g. text + image) Familiarity with human-in-the-loop systems or moderation workflows Experience with Databricks, PySpark, or large-scale data processing systems Additional Information Health + Mental Wellbeing PMI and cash plan healthcare access with Bupa Subsidised counselling and coaching with ...

scalable, high-quality data platform from the ground up. The Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into … Microsoft Fabric (Lakehouse, Warehouse, pipelines). Azure Data Factory experience will also be considered if no working experience with fabric Advanced SQL and strong PySpark experience Proficiency in Python for data processing and API integration Experience building Power BI models and dashboards for business reporting Strong understanding ...

solutions, ensuring alignment with business priorities and best practice Implementing robust ETL/ELT pipelines using Azure Data Factory, Synapse Pipelines and Azure Databricks (PySpark/Spark SQL) to support batch and streaming data processing Designing and delivering real-time and batch data ingestion solutions using Azure Event Hubs … Synapse Analytics, Azure SQL, Azure Data Lake, Datra bricks, Purview and Azure App Service is essential Proficiency in Python/SQL/SparkSQL/PySpark is essential Strong experience of Azure delivery experience including analysis/development/configuration/deployments essential Expert level technical skill ...

Phoenix enables digital transformation across the UK public sector, empowering organisations to innovate with cloud and hybrid infrastructures, data, AI, security and collaboration technologies. We are now hiring a Technical Consultant - Data & AI to join ...

Data Engineer in the United Kingdom to design, build, and maintain data pipelines on a modern AWS-native stack. You will use Spark/PySpark for distributed processing, Airflow for orchestration, and Terraform for infrastructure as code in a hybrid work arrangement with weekly travel to Newcastle office. ...

delivering insights for senior stakeholders. The role requires 5+ years of UK-based analytical experience with DAX/Power BI, SQL and Python/PySpark, plus strong communication, initiative and attention to detail. Three days per week in Hungerford offices. #J-18808-Ljbffr ...

technical content in commercial terms for non-technical stakeholders, such as customers, sales and product teams Desirable B2B product experience Experience with Spark SQL, PySpark, databricks Experience with CausalAI modelling frameworks (e.g. dowhy, econml) Diversity & Inclusion Intent HQ is an equal opportunities employer with an ethos of commitment ...

growing Data & Analytics team in Belfast. This is a fantastic opportunity for a Data Engineer with experience in Databricks, AWS, Python and/or PySpark to join an organisation investing heavily in its data engineering, analytics and AI capabilities. The successful Data Engineer will play a key role … Spark, while contributing to a wider programme of data transformation and continuous improvement. Key requirements include 2+ years' Data Engineering experience, strong Python/PySpark and/or Databricks experience, and a good understanding of data warehousing and big data concepts. Right to Work Applicants must have the existing ...

hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets … data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable ...

Arbuthnot Latham is seeking a Senior Data Engineer to develop and maintain complex data systems that support the Bank's operations and reporting needs. You will work with the Technical Lead for Data Engineering to ...

with 8+ years of experience, focusing on scalable data pipelines, governance, and analytics platforms. The role requires deep AWS data services expertise, Python/PySpark coding, and collaboration with globally distributed teams to deliver impact for Fortune 1000 clients. #J-18808-Ljbffr ...

well as familiarity with software engineering best practices (such as version control, unit testing, CI/CD) and big data and cloud technologies (PySpark preferred). Experience training and deploying deep learning models in cloud environments (including GPU‐enabled workflows) is a strong advantage. What ...