1 to 25 of 128 Apache Spark Jobs in the UK

British Citizenship, or ILR (Indefinite Leave to Remain), or UK Settlement Visa. We do not provide visa sponsorship. Key Requirements: Must have skills: -Spark & Scala Nice to have skills: -Spark Streaming, Hadoop, -Hive, -SQL, -Sqoop, - Impala Detailed Job Description: At least 8+ years of experience and strong knowledge … fundamental Data Structures and their usage At least 8+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. Having expertise in Spark Core, Spark SQL and Spark Streaming. Experience with Hadoop, HDFS, Hive and other ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark Very useful - Python, SQL, JSON, LakeFS. ...

prem data platform aligned to the metals strategy while developing and maintaining scalable data pipelines and analytics infrastructure. Using Hadoop, Big Data, and Spark technologies, you will ensure data quality through automated validation, monitoring, and testing. You will also enable seamless integration across data warehouses and data lakes, contributing … detailed technical design. Hands-on experience configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

prem data platform aligned to the metals strategy while developing and maintaining scalable data pipelines and analytics infrastructure. Using Hadoop, Big Data, and Spark technologies, you will ensure data quality through automated validation, monitoring, and testing. You will also enable seamless integration across data warehouses and data lakes, contributing … detailed technical design. Hands-on experience configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

prem data platform aligned to the metals strategy while developing and maintaining scalable data pipelines and analytics infrastructure. Using Hadoop, Big Data, and Spark technologies, you will ensure data quality through automated validation, monitoring, and testing. You will also enable seamless integration across data warehouses and data lakes, contributing … detailed technical design. Hands-on experience configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

team members, building technical expertise and capability Key Responsibilities Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes Implement modern ...

Skills & Experience Strong experience with Python and/or SQL in a production environment. Hands-on experience with modern data engineering frameworks such as: Apache Spark, Databricks Airflow/Prefect/Dagster Kafka/EventHub/Kinesis Experience building data solutions on at least one major cloud platform ...

keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue, EMR, or Spark. Ensure high … data Billing & revenue assurance Build models and pipelines to support network analytics, customer 360, churn prediction, fraud detection , etc. 7. Performance Optimization & Monitoring Tune Spark/Glue jobs for performance and cost. Monitor Redshift/Athena/S3 efficiency and implement best practices. Perform data quality checks and validation ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

requirements Excellent communication, problem-solving and documentation skills Desirable: Experience with cloud data platforms (AWS, Azure or GCP) Exposure to big data technologies (Hadoop, Spark, Kafka) Familiarity with ETL/orchestration tools (e.g. Informatica, Talend, Airflow) Experience with API integration and microservices data flows Familiarity with AI/ ...

requirements Excellent communication, problem-solving and documentation skills Desirable: Experience with cloud data platforms (AWS, Azure or GCP) Exposure to big data technologies (Hadoop, Spark, Kafka) Familiarity with ETL/orchestration tools (e.g. Informatica, Talend, Airflow) Experience with API integration and microservices data flows Familiarity with AI/ ...

downstream data services across the organisation. Key Responsibilities Design, build, and maintain data pipelines using Databricks Develop ETL/ELT processes with PySpark and Spark SQL Transform and model structured and semi-structured datasets Improve performance, reliability, and cost efficiency of data workloads Ensure compliance with data governance, security … practice routes Required Skills & Experience Active SC Clearance - mandatory and non-negotiable Strong hands-on experience with Databricks in production environments Expert knowledge of Apache Spark , including PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure ...

evolves. • Is versatile, displays leadership qualities and is enthusiastic to take on new challenges across the data stack. Job Requirements • Experience with Databricks, Spark and working knowledge of DBT for data processing and transformation. • Working understanding of Data Warehousing concepts of Data Vault, Dimensional modelling, OLAP design. • Hands … experience building data pipelines using technology such as Azure Data Factory, AWS Glue, GCP Dataflow and Apache Spark (preferably in Databricks). • Extensive experience with CI/CD platforms such as GitLab CI, GitHub Actions, AWS CodeBuild, Azure Pipelines, GCP Cloud Build, and Jenkins. • Skilled in scripting languages ...

senior technical interface with Palantir Essential Experience Proven commercial experience implementing and scaling Palantir Foundry Strong experience in Python, SQL and Spark (PySpark preferred) Experience building data models and ontologies Strong understanding of ETL/ELT design patterns Cloud-native architecture experience (AWS/Azure/GCP) Experience with … DevOps and Infrastructure-as-Code principles (eg Terraform) Strong understanding of data governance and security controls Desirable Skills Experience with big data tooling (eg Apache Spark, Kafka) Experience integrating Foundry with ERP/CRM platforms Knowledge of data science and ML workflows Experience in regulated or security-sensitive ...

/CD pipelines (e.g. Azure DevOps) Excellent communication skills and the ability to engage confidently with senior and executive stakeholders Desirable Experience Agentic AI, Apache Spark, or enterprise architecture frameworks Enterprise data integration and Microsoft BI technologies Experience with AWS or GCP and additional CI/CD tooling ...

/CD pipelines (e.g. Azure DevOps) Excellent communication skills and the ability to engage confidently with senior and executive stakeholders Desirable Experience Agentic AI, Apache Spark, or enterprise architecture frameworks Enterprise data integration and Microsoft BI technologies Experience with AWS or GCP and additional CI/CD tooling ...

practices, including ETL/ELT, data warehousing and data modelling. - Expertise in cloud data platforms, ideally Azure services such as Data Factory, Synapse, Databricks, Spark or Snowflake. - Proficiency in Python, SQL and other relevant languages for building pipelines and enabling automation. - Experience building and managing production data services, integrating ...

Strong full-stack experience with Python , Java (Spring Boot) , React , SQL, and relational databases. Hands-on experience with modern data engineering tools (e.g. Airflow, Spark, Kafka, dbt, Snowflake or similar). Experience with cloud platforms , containerisation (Docker/Kubernetes) , and CI/CD pipelines . Solid understanding of data ...

Ecosystem (Preferred/Good to Have) Azure data services experience in one or more: Azure Synapse Analytics/Dedicated SQL Pools Azure Databricks/Spark ADLS Gen2, Azure SQL DB, Managed Instance Event Hub/Kafka, Stream Analytics (if real-time involved) Monitoring & observability: Azure Monitor, Log Analytics, Application ...

GenAI workloads. Strong knowledge of data modelling, ETL processes, SQL, and scripting languages such as Python. Experience with Big Data technologies (e.g., Hadoop, Spark) and data visualisation tools (e.g., Tableau, Power BI). Familiarity with AI/ML frameworks (TensorFlow, PyTorch, Hugging Face, etc.). Experience in implementing security ...

Data Engineer (Python Spark SQL) *Newcastle Onsite* to £120k Do you have a first class education combined with Data Engineering skills? You could be progressing your career at a start-up Investment Management firm that have secure backing, an established Hedge Fund client as a partner and massive growth … scientific discipline, backed by minimum A A B grades at A-level You have commercial Data Engineering experience working with technologies such as SQL, Apache Spark and Python including PySpark and Pandas You have a good understanding of modern data engineering best practices Ideally you will also have ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...