1 to 25 of 99 Apache Spark Jobs in the UK

deliver reliable, high-performance data solutions using Databricks and modern cloud technologies. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Delta Lake Build and optimize ETL/ELT workflows for batch and streaming data Develop data models to support analytics, reporting … data pipelines, architectures, and best practices Required Qualifications 3+ years of experience in data engineering or a similar role Strong experience with Databricks and Apache Spark Proficiency in Python and SQL Experience with Delta Lake, data modeling, and performance tuning Hands-on experience with cloud platforms (AWS, Azure ...

prem data platform aligned to the metals strategy while developing and maintaining scalable data pipelines and analytics infrastructure. Using Hadoop, Big Data, and Spark technologies, you will ensure data quality through automated validation, monitoring, and testing. You will also enable seamless integration across data warehouses and data lakes, contributing … detailed technical design. Hands-on experience configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue, EMR, or Spark. Ensure high … data Billing & revenue assurance Build models and pipelines to support network analytics, customer 360, churn prediction, fraud detection , etc. 7. Performance Optimization & Monitoring Tune Spark/Glue jobs for performance and cost. Monitor Redshift/Athena/S3 efficiency and implement best practices. Perform data quality checks and validation ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

requirements Excellent communication, problem-solving and documentation skills Desirable: Experience with cloud data platforms (AWS, Azure or GCP) Exposure to big data technologies (Hadoop, Spark, Kafka) Familiarity with ETL/orchestration tools (e.g. Informatica, Talend, Airflow) Experience with API integration and microservices data flows Familiarity with AI/ ...

share knowledge with team members. Required Qualifications Proven experience in designing and building data pipelines (ETL/ELT) using modern technologies (Python, SQL, Spark, Airflow, etc.). Strong analytical skills with the ability to interpret complex data and deliver business value. Experience integrating data from multiple sources and systems. ...

/CD pipelines (e.g. Azure DevOps) Excellent communication skills and the ability to engage confidently with senior and executive stakeholders Desirable Experience Agentic AI, Apache Spark, or enterprise architecture frameworks Enterprise data integration and Microsoft BI technologies Experience with AWS or GCP and additional CI/CD tooling ...

/CD pipelines (e.g. Azure DevOps) Excellent communication skills and the ability to engage confidently with senior and executive stakeholders Desirable Experience Agentic AI, Apache Spark, or enterprise architecture frameworks Enterprise data integration and Microsoft BI technologies Experience with AWS or GCP and additional CI/CD tooling ...

practices, including ETL/ELT, data warehousing and data modelling. - Expertise in cloud data platforms, ideally Azure services such as Data Factory, Synapse, Databricks, Spark or Snowflake. - Proficiency in Python, SQL and other relevant languages for building pipelines and enabling automation. - Experience building and managing production data services, integrating ...

Strong full-stack experience with Python , Java (Spring Boot) , React , SQL, and relational databases. Hands-on experience with modern data engineering tools (e.g. Airflow, Spark, Kafka, dbt, Snowflake or similar). Experience with cloud platforms , containerisation (Docker/Kubernetes) , and CI/CD pipelines . Solid understanding of data ...

leading and developing data engineering platforms. The ideal candidate will possess hands-on expertise in the following areas: Extensive enterprise experience with Hadoop, Spark , and Splunk. Proficiency in object-oriented and functional scripting, particularly in Python . Skilled in handling raw, structured, semi-structured, and unstructured data ...

Data Engineer (Python Spark SQL) *Newcastle Onsite* to £120k Do you have a first class education combined with Data Engineering skills? You could be progressing your career at a start-up Investment Management firm that have secure backing, an established Hedge Fund client as a partner and massive growth … scientific discipline, backed by minimum A A B grades at A-level You have commercial Data Engineering experience working with technologies such as SQL, Apache Spark and Python including PySpark and Pandas You have a good understanding of modern data engineering best practices Ideally you will also have ...

/or Computer Vision. Strong knowledge of Machine Learning algorithms and statistics. Experience with SQL/NoSQL databases and big data tools (Spark, Hadoop preferred). Experience with MLOps tools such as Docker, Kubernetes, CI/CD pipelines. Preferred Skills:- Experience with LLMs/Generative AI (OpenAI, Hugging Face ...

Seaborn) Ability to communicate complex analytical findings to diverse audiences Preferred Qualifications Experience with cloud platforms (AWS, GCP, Azure) Familiarity with big data technologies (Spark, Hadoop) Experience deploying models into production environments Knowledge of deep learning frameworks (TensorFlow, PyTorch) Experience working in Agile or cross-functional product teams ...

deliver high-quality, consumable datasets while contributing to best practices across the wider platform. You'll be working heavily with Python , SQL and Spark , using tools such as Databricks , Airflow , dbt and Kafka within AWS . Experience with modern data stacks is important, along with a solid understanding ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

discussions and provide guidance on data engineering best practices. Ability to troubleshoot and optimize data workflows. Required Knowledge and Skills: Experience with Databricks or Spark-based environments, including optimization strategies. Familiarity with data governance tools and frameworks (e.g., Unity Catalog). Experience with JavaScript Full-Stack development environments including ...

sectors. Strong knowledge of data fabric architectures, metadata management, and data governance principles. Experience with SQL, ETL tools, and big data technologies (e.g., Hadoop, Spark, Kafka). Familiarity with cloud platforms (AWS, Azure, or Google Cloud) and data integration frameworks. Strong problem-solving skills and ability to work ...

sectors. Strong knowledge of data fabric architectures, metadata management, and data governance principles. Experience with SQL, ETL tools, and big data technologies (e.g., Hadoop, Spark, Kafka). Familiarity with cloud platforms (AWS, Azure, or Google Cloud) and data integration frameworks. Strong problem-solving skills and ability to work ...

sectors. Strong knowledge of data fabric architectures, metadata management, and data governance principles. Experience with SQL, ETL tools, and big data technologies (e.g., Hadoop, Spark, Kafka). Familiarity with cloud platforms (AWS, Azure, or Google Cloud) and data integration frameworks. Strong problem-solving skills and ability to work ...

sectors. Strong knowledge of data fabric architectures, metadata management, and data governance principles. Experience with SQL, ETL tools, and big data technologies (e.g., Hadoop, Spark, Kafka). Familiarity with cloud platforms (AWS, Azure, or Google Cloud) and data integration frameworks. Strong problem-solving skills and ability to work ...