26 to 50 of 200 Apache Spark Jobs in the UK excluding London

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

continuous learning to understand new data management systems Strong proficiency in hands-on development using industry-standard tools and languages (e.g., SQL, Python, Spark, cloud platforms), with a commitment to code quality and delivery excellence Preferred qualifications, capabilities, and skills Hands-on experience with Databricks for scalable data engineering … analytics, and collaborative development, as well as proficiency in managing large datasets using Apache Iceberg for efficient, reliable table management and data lake optimization Experience architecting and implementing enterprise-scale data warehouses on cloud platforms (e.g., AWS, Azure, GCP), including migration from legacy systems Advanced proficiency in designing ...

processes. Technical Skills Programming: Proficiency in Python, Java, Scala, or similar languages. Big Data Technologies: Hands-on experience with big data tools e.g. (Databricks, Apache Spark, Hadoop). Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. Expertise in relational databases (e.g. … postgres, sql server) Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment. Experience ...

fields of data analytics and software Proficiency in Python programming language and common data science packages. Experience with data engineering tools (e.g. Apache Spark, AWS Glue, Kinesis, Snowflake, Apache Airflow) Experience in building data pipelines and data streaming applications. Experience with tools to support the software ...

more data-driven culture. Description Design, develop, and maintain scalable data pipelines using Python and Databricks. Build and optimise ETL/ELT workflows using Spark and Delta Lake. Develop and maintain data models and lakehouse architectures. Ensure high levels of data quality, reliability, governance, and security. Optimise the performance … Data Engineering. Strong commercial experience with Python. Advanced hands-on knowledge of Databricks, including notebooks, workflows, jobs, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and maintaining scalable ETL pipelines. Strong SQL and data modelling expertise. Experience working within AWS, Azure ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

with Snowflake and Databricks native solutions. Experience on Snowflake Cortex is must. Proficiency in programming skills in Python. Experience with data processing frameworks like Apache Spark or Hadoop. Knowledge of database systems (SQL and NoSQL). Familiarity with cloud platforms (AWS, Azure) and their data services. Understanding ...

stakeholder/data consumer management. Strong programming skills in languages such as Python, familiarity with large scale, distributed data processing engines such as Apache Spark, with a solid understanding of data structures and algorithms. Proficiency in SQL and experience with relational databases (e.g., PostgreSQL, MySQL) or NoSQL databases ...

/CD, and Infrastructure as Code. Desirable: Experience within UK Defence, Intelligence, or National Security sectors. Knowledge of big data tools such as Kafka, Spark, or Hadoop. Support for machine learning or advanced analytics workloads. Experience working with classified datasets and Elastic Stack (ELK). If you're ready ...

Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience in Data ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

Factory, Databricks and Azure Data Lake. Advanced SQL and Python (PySpark) skills for data transformation, automation and optimisation. Hands-on experience with Delta Lake, Spark, Databricks Workflows and Structured Streaming. Knowledge of data warehousing, lakehouse architectures and dimensional modelling. Experience delivering scalable, high-quality data solutions to support analytics ...

modern data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python. Experience with data engineering and pipeline technologies such as Apache Airflow and Spark. Experience working with cloud platforms, particularly AWS and Azure. Knowledge of big data technologies and distributed data processing. Understanding of data ...

have Programming: Proficiency in Python, Java, Scala, or similar languages for data processing. Big Data Technologies: Hands-on experience with tools such as Databricks, Apache Spark, and Hadoop. Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. Database Management: Expertise in relational … databases (e.g. Postgres, SQL Server). Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

control systems such as Git. Knowledge of cloud-based data platforms and modern data engineering practices. Desirable Skills Experience with Scala development. Experience with Apache Spark or other distributed data processing frameworks. Experience with Kafka, Event Hubs, or other streaming technologies. Exposure to cloud platforms such as Azure ...

data processing. Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning. Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar). Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.). Strong understanding of data quality, security ...

data processing. Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning. Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar). Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.). Strong understanding of data quality, security ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

solutions. Pipeline Development : Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data factory and engineering tools like Apache Spark to ensure seamless data flow. Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery … using SQL to query and manipulate data. Proficiency in cloud data tooling such as Synapse Analytics, Microsoft Fabric, Azure Data Factory or AWS equivalents. Spark/Pyspark or Python skills. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills. MS Fabric interest, skills or experience ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills ...

Strong experience in IaC and guidance on deploying infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc.) with Apache Spark, Databricks or Hadoop. Good understanding of modern data system design architectures (Data Warehouse, Data Lakes, Data Meshes). Ability to create data ...

maintain SC clearance. Desirable AWS certification at Associate or Professional level Experience with streaming and event‐driven data technologies Experience with technologies such as Spark, Kafka or Databricks Experience delivering data platforms within public sector, defence or other regulated environments Experience working within secure environments and understanding associated security ...