1 to 25 of 50 Apache Spark Jobs in the UK excluding London

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc Experience developing end‐to‐end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam. Expert understanding of when to use a range of data storage technologies including relational/non‐relational, document ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

design and deliver scalable, enterprise-grade data solutions. This role will focus on building and optimising modern data platforms using Python, Databricks and Spark, enabling advanced analytics, reporting and data-driven innovation across the business. Client Details Our client is a leading international financial institution with a long-established … complex global environment. Description Design, build and maintain scalable data pipelines using Python and Databricks Develop and optimise ETL/ELT processes leveraging Spark and Delta Lake Create robust data models and data architectures within modern Lakehouse environments Implement and manage data workflows across the Databricks ecosystem Ensure data ...

seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives. This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement … similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced ...

/CD, and Infrastructure as Code. Desirable: Experience within UK Defence, Intelligence, or National Security sectors. Knowledge of big data tools such as Kafka, Spark, or Hadoop. Support for machine learning or advanced analytics workloads. Experience working with classified datasets and Elastic Stack (ELK). If you're ready ...

understanding of Quantexa components such as: Entity Resolution (ER), Scoring Framework, Contextual Network Generation, UI Configuration. Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch. Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

Rust Data Platforms: BigQuery, Snowflake, Redshift, Databricks, PostgreSQL, MySQL, MongoDB, or other SQL/NoSQL technologies Data Tooling: Airflow, dbt, AWS Glue, Step Functions, Spark, Hadoop, or similar tooling Infrastructure: AWS, Terraform, Docker, Kubernetes, or other cloud/platform technologies CI/CD: GitHub Actions, Jenkins, ArgoCD, or similar ...

Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/or Amazon Athena ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

strategic thinking. Essential Skills & Experience Strong Microsoft Fabric expertise (Lakehouse, Warehouse, Pipelines, Notebooks) OneLake architecture and management Advanced SQL/T-SQL PySpark and Spark optimisation Data warehousing, modelling and ETL/ELT development Power BI semantic modelling Experience delivering enterprise cloud data platforms Experience integrating multiple business systems ...

Snowflake, Databricks). Experience with data integration (ETL/ELT, APIs). Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) is a plus. Domain Knowledge Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations. In-depth knowledge ...

need as Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD ...

large-scale data platforms and end-to-end technical solutions within enterprise environments. Strong hands-on experience with open-source data technologies such as Apache Spark, Trino, Airflow, Apache Superset , or similar platforms. Demonstrable security architecture expertise, including threat modelling, IAM (RBAC/ABAC), encryption, secrets management ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

range of projects that include internal and external suppliers. Essential Skills Proven experience as a Data Engineer Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience with Microsoft Fabric Experience of using programming languages such as Python, Scala and SQL. Demonstrable ...

work closely with stakeholders and translate business requirements into effective AI solutions Desirable Generative AI experience Large Language Models (LLMs) LangChain or similar frameworks Spark, Hadoop or other big data technologies Experience with distributed computing, streaming platforms or workflow orchestration tools Experience monitoring and optimising model performance in production ...

platforms for performance and reliability Collaborate with Agile teams to deliver secure, high-quality solutions Write clean, tested, maintainable code Required Skills: Experience with Apache Spark, Hadoop or similar big data technologies AWS Apache NiFi Python or Java Experience building data pipelines and ETL processes Desirable ...

platforms for performance and reliability Collaborate with Agile teams to deliver secure, high-quality solutions Write clean, tested, maintainable code Required Skills: Experience with Apache Spark, Hadoop or similar big data technologies AWS Apache NiFi Python or Java Experience building data pipelines and ETL processes Desirable ...

reviews and continuous improvement Skills & Experience * Commercial experience in Data Engineering * Strong Python or Java development skills * Experience with AWS cloud services * Knowledge of Apache Spark, Hadoop, NiFi or similar big data technologies * Experience building ETL/ELT pipelines and processing large datasets * Understanding of software engineering best ...

Architecture.- Strong programming skills in Python and SQL.- Familiarity with cloud platforms such as AWS, Azure, or GCP.- Knowledge of Snowflake and experience with Spark/PySpark.- Experience in designing and implementing modern data platform engineering solutions.If you are ready to make an impact within a dynamic team ...

multidisciplinary team to support and enhance over 70 active BI products. What You'll Bring Essential Skills & Experience: Core Technologies: Azure Databricks (or similar Spark-based platforms) and Azure Data Factory (ADF). Languages: SQL and Python. Data Platforms: Azure Data Lake and SQL Server RDBMS. ETL Tools ...

Work with container technologies Core Data Engineering Strong programming proficiency in Java or Python. Hands-on experience with at least one of: Kafka, Flink, Spark Solid understanding of stream processing concepts Understanding of software engineering best practices: testing, design patterns, CI/CD, Git. Experience building ETL/ ...