1 to 25 of 42 Remote/Hybrid Permanent Apache Spark Jobs

will initially be for 6 months. It is also fully remote. Key Responsibilities • Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark) • Build and optimize ETL/ELT workflows for large-scale structured and unstructured data • Implement Delta Lake architectures (Bronze/Silver/… Gold layers) • Integrate data from multiple sources (databases, APIs, event streams, files) • Optimize Spark jobs for performance, scalability, and cost • Manage data quality, validation, and monitoring • Collaborate with analytics and ML teams to support reporting and model development • Implement CI/CD, version control, and automated testing for data ...

platform, features, and best practices with Strong understanding of data modeling principles and techniques A solid understanding of big data technologies such as Apache Spark, and knowledge of Hadoop ecosystems. Knowledge of programming languages such as Python, R, or Java is beneficial. Knowledge of ETL/ELT processes ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

Charity Volunteer Days Cycle to work scheme And More.. DATA ENGINEER – ESSTENTIAL SKILLS Proven experience building data pipelines using Databricks . Strong understanding of Apache Spark (PySpark or Scala) and Structured Streaming . Experience working with Kafka (MSK) and handling real-time data . Good knowledge of Delta ...

experience and familiarity with frameworks (e.g. Flask, Django) Experience building and orchestrating ETL pipelines (e.g. Airflow, Luigi, Argo) Exposure to big data technologies (e.g. Spark, Kafka, Hadoop) Experience working in Agile environments Solid understanding of data architecture and data quality best practices ...

experience and familiarity with frameworks (e.g. Flask, Django) Experience building and orchestrating ETL pipelines (e.g. Airflow, Luigi, Argo) Exposure to big data technologies (e.g. Spark, Kafka, Hadoop) Experience working in Agile environments Solid understanding of data architecture and data quality best practices ...

platforms such as AWS, Azure, or GCP. Exposure to modern data tools such as Snowflake, Databricks, or BigQuery. Familiarity with streaming technologies (e.g., Kafka, Spark Streaming, Flink) is an advantage. Experience with orchestration and infrastructure tools such as Airflow, dbt, Prefect, CI/CD pipelines, and Terraform. What ...

plus) Experience with orchestration and workflow tools (Airflow, Cloud Composer, Dagster, etc.) Familiarity with modern data stack components (BigQuery, dbt, Kafka/PubSub, Spark, etc.) Knowledge of data security, access control, and best practices for handling sensitive data Experience collaborating across engineering, analytics, and product teams Strong communication skills ...

transformation. Proficiency in Python and PySpark for advanced data engineering, machine learning, and large-scale distributed processing within Databricks. Hands-on experience with Spark optimizations (Adaptive Query Execution, caching, partitioning, broadcast joins) to deliver performant solutions for big data workloads. Advanced knowledge of BI and visualization tools such ...

sources, analytics, or trading workflows. Exposure to cloud platforms (Azure, AWS, or GCP). Experience hiring, mentoring, or leading small engineering teams. Familiarity with Apache Spark, Kafka, Delta Lake, or similar data technologies. Apply Now to avoid disappointment! To find out more about Huxley, please visit ...

model lifecycle management Experience with NLP, deep learning, or computer vision Knowledge of data visualisation tools (Power BI, Tableau) Exposure to big‐data technologies (Spark, Hadoop) AWS certifications Experience supporting government or highly regulated environments Clearance Requirements: Clearance to Start BPSS Clearance for Role SC Applicants must have ...

TypeScript, JavaScript Machine Learning: TensorFlow, PyTorch, Scikit-learn GenAI/LLMs: OpenAI API, Hugging Face Transformers, LangChain, Langraph, RAG pipelines Data & Analytics: Pandas, NumPy, Spark, Databricks, Hadoop, Vector Databases, Snowflake Model Deployment: MLflow, FastAPI, Flask, Docker, Kubernetes Cloud & Infrastructure: AWS, Azure, GCP CI/CD pipelines (GitLab, Jenkins) Desirable ...

solutions.• Pipeline Development: Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data factory and engineering tools like Apache Spark to ensure seamless data flow.• Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery ...

Azure, utilizing managed services like SageMaker or Vertex AI. Data Foundations: Strong understanding of data modeling, feature stores, and distributed computing (e.g., Spark, Dask). Qualifications & Experience Experience: 5+ years in a software or data engineering role, with at least 3 years specifically focused on the deployment and scaling ...

organisation Supporting a move towards a product-aligned, hub-based operating model Technology Stack Cloud-first environment with Microsoft Azure Fabric Python, PyTorch SQL Apache Spark Azure Data Factory (ADF) Git What We’re Looking For 3–4 years’ experience in AI/ML engineering, data engineering ...

Data Analytics Sales Executive - Data Sales, Talend, Snowflake, Apache Spark, Kafka, Dataops, AWS, Azure, Data visualisation, SQL, ETL, Pre-Sales, Account Manager, SaaS Sales, Cloud Solutions, Digital Transformation, Business Development, Sales Consultant, CXO Engagement, Strategic Accounts, Deal Negotiation, Remote Working, UK-wide, Permanent Role, Sales Targets, New Business ...

engineers, setting technical standards and driving best practices across the organisation. The role demands hands-on expertise with Azure, big data technologies like Spark and Hadoop, and programming languages including Python. You will also oversee database architecture, data modelling, and ETL processes to ensure seamless integration across multiple platforms ...

data platform. You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions are scalable ...

from classical statistics and regression to deep learning and NLP. Data Engineering: Proficiency in SQL and experience working with modern data stacks (e.g., Snowflake, Spark, or cloud-native tools on AWS/GCP/Azure). Scientific Rigor: A strong foundation in the scientific methodforming hypotheses, designing experiments ...

Azure based Production Data & Analytics ETL/integration solutions (Fabric, Databricks, Synapse, Data Factory, Function Apps, Power BI etc) against SLAs Python, PySpark, Scala, Spark SQL Ability to identify and investigate complex problems and implement resolutions Direct contact with customers/end-users on the phone and by email ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Azure Services Understanding of software architecture, design patterns, and agile delivery Experience with geospatial data, REST APIs, JSON and spatial query engines (e.g. Apache Spark) Experience with Generative AI technologies including Large Language Models (LLMs), Retrieval Augmented Generation (RAG) and Prompt/Context Engineering Passion for continuous improvement ...

skills; experience with deep learning frameworks (TensorFlow, PyTorch) Solid understanding of ML algorithms, statistics, and data modelling Experience with large-scale data tools (Hadoop, Spark, Kafka) Cloud & MLOps experience (AWS, Azure, or GCP) NLP experience highly beneficial Strong leadership, communication, and stakeholder management ...

deep learning techniques. Experience deploying ML models to production using cloud platforms (AWS, GCP or Azure). Familiarity with big-data or distributed tools (Spark, Kafka or similar). A track record of leading complex ML projects and influencing technical decisions. Excellent communication skills across distributed, cross-functional teams. ...