22 of 22 Apache Spark Jobs in the South East

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

help design and deliver cutting-edge data solutions for customers operating in some of the UK's most complex and challenging environments. Using Databricks, Apache Spark and cloud-native technologies, you’ll develop scalable Lakehouse architectures and production-grade data pipelines that enable organisations to trust, share … opportunity to grow your career while helping customers unlock the power of their data. Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark, PySpark and Delta Lake. Develop and implement modern Lakehouse architectures, applying Medallion (Bronze, Silver, Gold) design principles to support trusted and reusable ...

business. Key Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platformDevelop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta LakeContribute to platform architecture, engineering standards and best practices across the data estateWork with Product, Analytics and Data Science teams … contribute to continuous improvement across the engineering teamJob Requirements: Significant experience building and supporting scalable, production-grade data platformsStrong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQLGood understanding of cloud-based data engineering, ideally within Microsoft Azure environmentsExperience with Infrastructure as Code tools such ...

techniques, including machine learning and their application in financial services. Proficiency in data tools, programming languages, and database technologies, in particular in Azure Databricks, Spark, Spark SQL and Python/pyspark. Understanding of data governance, security, and data quality management principles. Experience working with large, complex datasets … producing audit evidence Essential Skills: Data pipeline development and optimisation Data modelling and architecture (in Azure Databricks) Programming and scripting for data processing (Spark, Spark SQL, Python, pyspark) Data science and machine learning Data governance and quality assurance Problem-solving and analytical thinkingStakeholder engagement and technical communication, including ...

need to obtain and maintain this clearance as a condition of employment. Nice to have Experience with distributed data processing or streaming tools (e.g. Spark, Kafka) Workflow orchestration (e.g. Airflow, Dagster or similar) Working with LLMs, vector databases, embeddings or RAG patterns MLOps practices: model versioning, deployment and monitoring ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

learning or data science. Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials. Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow ...

schemas for both structured and unstructured data with a snowflake environment Enhance cloud data capabilities, working with AWS S3, Azure Data Lake, and Apache Spark Collaborate with cross-functional teams to ensure data availability for advanced analytics and reporting Automate workflows, implement CI/CD pipelines, and maintain ...

experience working with Data A strong understanding of data integration techniques and solution/performance optimisation Excellent understanding and knowledge of SQL, Python or Spark Apply relevant data modelling techniques (e.g. STAR, Snowflake) Excellent Communication and Teamwork Skills Coaching and mentoring experience Technologies & Experience Desired: Git and JIRA software ...

GenAI infrastructure and workflows Skills and experience: Passionfor building scalable, resilient cloud data platforms Strongexperience with Databricks or Snowflake on AWS ProvenPython skills, including Spark and Airflow expertise AdvancedSQL skills and end‐to‐end data modelling experience Experiencebuilding batch and real-time data integrations Hands-onCICD skills with ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

edge and implementing secure configuration distribution Building streaming ingestion, event pipelines, batch/stream processing workloads using both distributed event streaming technologies and Apache Flink and/or Apache Spark with clear SLA’s and repeatable deployment Implementing metadata, lineage and catalogue capabilities and implementing policy-based ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

Scale processing across distributed cloud warehouses/storage via container orchestration (e.g. Kubernetes), High-Performance GPU Compute (e.g. Slurm), or distributed compute frameworks (e.g. Spark, Ray). Contribute to an engineering culture that values maintainability, testing, robust system design, and deep collaboration, but allows flexibility for rapid prototyping ...

model validation. Experiment tracking and reproducibility tools, for example MLflow, Weights & Biases or equivalent. Data wrangling and analysis using Pandas, NumPy, SQL; familiarity with Spark or similar is a plus. Model optimisation and research-to-prototype deployment fundamentals, including ONNX, TorchScript, FastAPI/gRPC and GPU acceleration, with CUDA ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...

Envitia is seeking a Senior Data Engineer to design and deliver cutting-edge data solutions for customers in the UK. Using Databricks, Apache Spark and cloud-native technologies, you’ll build scalable Lakehouse architectures and production-grade pipelines that enable organisations to trust and share data. ...

engineering practices to multi-disciplinary teams. Desirable Skills, Qualifications & Experience: Experience leveraging Kubernetes for application deployment, and familiarity with distributed computing frameworks (e.g., Ray, Spark), or specialised batch schedulers/resource managers (e.g., Slurm, Volcano, Kueue). Demonstratedexpertisewith specialised serving engines (e.g.,vLLM,SGLang) or techniques for deploying models ...

deploying scalable data platforms end-to-end on AWS and/or Databricks. Production Coding: Strong, hands-on production experience with Python, SQL, Spark, and Databricks or AWS data tools. Consulting/Client Delivery: Experience working in a consultancy, systems integrator, or client-facing setting with direct accountability …/or AWS certifications at Associate or Professional level. Streaming Architectures: Practical experience with real-time or event-driven streaming (Kafka/MSK, Kinesis, Spark Structured Streaming). Advanced Data Architectures: Exposure to Data Mesh concepts, data products, or federated data governance models. AI/ML Infrastructure: Exposure ...

Senior Data Scientist - Python/Machine Learning Permanent Location: UK - Remote Salary: £65,000 - £70,000 (+ benefits) Skills: Python, Machine Learning, Statistical Modelling, AWS, Data Engineering An exciting opportunity has arisen for an experienced ...

Software Engineer - Space Domain Awareness | Job DetailsAre you looking for an innovative and consolidated place for your professional development? GMV is an experienced technology group and we are looking for a highly motivated Software Engineer ...