1 to 25 of 137 Apache Spark Jobs in the UK

/ELT) to support analytics and downstream use cases Developing scalable data solutions using Python , with a focus on maintainability and performance Working with Apache Spark/PySpark to process and transform large datasets Supporting the ingestion, transformation and validation of complex financial data Improving the performance, reliability … That Will Help You Succeed Essential Commercial experience as a Data Engineer at a mid‐level Strong Python development skills Hands‐on experience with Apache Spark/PySpark Solid experience building ETL/ELT pipelines Background within the financial services industry (investment management experience desirable) Comfortable working with ...

urgently require a Data Platform Engineer with core strengths in Building and Operating Data Platforms, and hands on expertise in Azure Services, Synapse Spark Architecture, ADF and Terraform (IaC), to assist in designing and delivering complex cloud solutions. Key Requirements: Proven, hands-on experience of Building and Operating Data … Advanced knowledge of Azure Services Demonstrable experience with Azure Data Factory (ADF), including network configurations and integration runtime options Strong expertise in Synapse Analytics Spark, Azure Key Vault, and Data Lake Storage Accounts Ability to showcase DevOps and CI/CD capabilities Working knowledge of designing and deploying infrastructure ...

deliver reliable, high-performance data solutions using Databricks and modern cloud technologies. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Delta Lake Build and optimize ETL/ELT workflows for batch and streaming data Develop data models to support analytics, reporting … data pipelines, architectures, and best practices Required Qualifications 3+ years of experience in data engineering or a similar role Strong experience with Databricks and Apache Spark Proficiency in Python and SQL Experience with Delta Lake, data modeling, and performance tuning Hands-on experience with cloud platforms (AWS, Azure ...

innovate, grow, and succeed. Join our team and be part of a dynamic, growth-oriented organization that values talent, creativity, and results. Role : Spark-Scala Developer Location : London, UK (Hybrid 3 days/week) Contract (InsideIR35) Must have skills: -Spark & Scala Nice to have skills: -Spark Streaming … fundamental Data Structures and their usage • At least 8+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. • Having expertise in Spark Core, Spark SQL and Spark Streaming. • Experience with Hadoop, HDFS, Hive and other ...

Global Recruitment specialist that provides support to the clients across EMEA, APAC, US and Canada. We have an excellent job opportunity for you. Role: Spark-Scala Developer Location: London Working Mode: Hybrid (Weekly 3 days onsite) Contract Type: Inside IR35 Duration: 12 months Rate: £440 per day Inside IR35 … Must have skills: -Spark & Scala Nice to have skills: -Spark Streaming, Hadoop, -Hive, -SQL, -Sqoop, - Impala Detailed Job Description: At least 8+ years of experience and strong knowledge in Scala programming language. Able to write clean, maintainable and efficient Scala code following best practices. Good knowledge ...

will initially be for 6 months. It is also fully remote. Key Responsibilities • Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark) • Build and optimize ETL/ELT workflows for large-scale structured and unstructured data • Implement Delta Lake architectures (Bronze/Silver/… Gold layers) • Integrate data from multiple sources (databases, APIs, event streams, files) • Optimize Spark jobs for performance, scalability, and cost • Manage data quality, validation, and monitoring • Collaborate with analytics and ML teams to support reporting and model development • Implement CI/CD, version control, and automated testing for data ...

native services, infrastructure automation, and end to end ML workflow operations. The ideal candidate will have deep experience with CloudFormation, SageMaker, Glue, Lambda, Spark, and Python, along with solid understanding of machine learning concepts, data pipelines, model monitoring, and production deployment. Key Responsibilities • Design, automate, and maintain scalable … services. • Build, deploy, and manage ML models on Amazon SageMaker (training, tuning, hosting, endpoints, pipelines). • Develop and optimize distributed data processing workflows using Apache Spark, AWS Glue, and related ETL frameworks. • Build serverless automation and integration logic using AWS Lambda and Python-based microservices. • Implement MLOps best ...

prem data platform aligned to the metals strategy while developing and maintaining scalable data pipelines and analytics infrastructure. Using Hadoop, Big Data, and Spark technologies, you will ensure data quality through automated validation, monitoring, and testing. You will also enable seamless integration across data warehouses and data lakes, contributing … detailed technical design. Hands-on experience configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

highly regulated financial settings. Key Technologies & Tools Core ML: scikit-learn, PyTorch, TensorFlow, XGBoost, LightGBM Data Manipulation: Pandas, NumPy, LangGraph MLOps & Orchestration: MLflow, Apache Airflow Databases: Neo4j, MongoDB API Development: FastAPI Big Data (Beneficial): Apache Hadoop, Apache Spark (PySpark) What We Need From ...

Java and/or Scala Frontend development using Angular or a similar modern JavaScript framework Experience with distributed compute concepts and tools such as: Apache Spark Databricks Apache Beam (or similar) Solid experience with Azure cloud technologies Database experience with cloud and relational databases, e.g. Snowflake, Sybase ...

development Backend development using Java and/or Scala Frontend development with Angular or similar JavaScript frameworks Experience with distributed compute technologies, such as: Apache Spark Databricks Apache Beam (or similar) Solid experience with Azure cloud technologies Database experience with cloud and relational databases, e.g. Snowflake, Sybase ...

development Backend development using Java and/or Scala Frontend development with Angular or similar JavaScript frameworks Experience with distributed compute technologies, such as: Apache Spark Databricks Apache Beam (or similar) Solid experience with Azure cloud technologies Database experience with cloud and relational databases, e.g. Snowflake, Sybase ...

experience in cloud-based data services. Proficiency in SQL and Python for data manipulation and transformation. Experience with modern data engineering tools, including Apache Spark, Kafka, and Airflow. Strong understanding of data modelling, schema design, and data warehousing concepts. Familiarity with data governance, privacy, and compliance frameworks (e.g. ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

requirements Excellent communication, problem-solving and documentation skills Desirable: Experience with cloud data platforms (AWS, Azure or GCP) Exposure to big data technologies (Hadoop, Spark, Kafka) Familiarity with ETL/orchestration tools (e.g. Informatica, Talend, Airflow) Experience with API integration and microservices data flows Familiarity with AI/ ...

Experience building and deploying cloud data platforms Infrastructure-as-Code and CI/CD practices (GitHub, CodeBuild, pipelines) Strong coding experience in Python (Pandas), Apache Spark, and/or Apache Iceberg Expertise in SQL and advanced data manipulation and analysis Experience with data warehouse design and data ...

while working on complex, real-world use cases. Key Responsibilities Design, develop, and optimise data ingestion pipelines and transformations within the Quantexa platform using Spark and Scala Configure and implement Quantexa components, including Entity Resolution, Scoring, and Network Generation, to meet specific use cases Collaborate with Solution Architects … Experience Quantexa Technical Certification (mandatory) 6+ years' experience in data engineering roles, including 2+ years delivering Quantexa solutions Strong hands-on experience with Scala, Apache Spark, and Big Data frameworks (eg Hadoop, HDFS) Proven experience with core Quantexa capabilities: Data Ingestion Pipelines Entity Resolution configuration Scoring logic implementation ...

platform, features, and best practices with Strong understanding of data modeling principles and techniques A solid understanding of big data technologies such as Apache Spark, and knowledge of Hadoop ecosystems. Knowledge of programming languages such as Python, R, or Java is beneficial. Knowledge of ETL/ELT processes ...

platform, features, and best practices with Strong understanding of data modeling principles and techniques A solid understanding of big data technologies such as Apache Spark, and knowledge of Hadoop ecosystems. Knowledge of programming languages such as Python, R, or Java is beneficial. Knowledge of ETL/ELT processes ...

share knowledge with team members. Required Qualifications Proven experience in designing and building data pipelines (ETL/ELT) using modern technologies (Python, SQL, Spark, Airflow, etc.). Strong analytical skills with the ability to interpret complex data and deliver business value. Experience integrating data from multiple sources and systems. ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

Knowledge of machine learning frameworks (e.g., PySpark, PyTorch) and model deployment tools (e.g., MLflow, TensorFlow Serving). Strong experience with data processing frameworks (e.g., Apache Spark, Flink). Expertise in SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra). Hands-on experience with cloud platforms (e.g. ...

individuals’ desires and capabilities. Essential Skills Proven experience as a Data Architect (or similar/related role) Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience with Big Data Experience with Enterprise level Data sets Experience of using programming languages such ...

pragmatic, impact-focused way What You Bring Senior level Data Engineering experience Excellent Python and SQL skills Experience in big data environments (e.g., Spark, distributed systems, cloud data platforms) Solid understanding of data modeling, ETL/ELT, and pipeline architecture Willingness to work onsite in Edinburgh or London ...