26 to 50 of 372 Apache Spark Jobs in England

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

stores). Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop). Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis). End-to-end exposure … data pipeline lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background ...

software engineering experience using Python and/or other modern programming languages. Experience developing and orchestrating data pipelines using technologies such as DBT, Apache Spark, Airflow, Dagster, Kafka, Flink, or equivalent tooling. Solid understanding of data lakehouse, warehouse, and medallion architecture patterns. Experience applying DevOps and DataOps best ...

across the team and support the continued growth of Unifeye. What You'll Do Design and build scalable batch and streaming data pipelines using Spark and Databricks. Develop modern Lakehouse architectures using Delta Lake and Medallion (Bronze, Silver and Gold) design principles. Build robust ingestion frameworks from APIs, files … pipelines, deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with ...

associated Azure technologies. Build scalable ETL/ELT pipelines to ingest, transform, and load data from multiple enterprise systems. Develop data transformations using PySpark, Spark SQL, SQL, and Python. Create and maintain curated datasets that support business intelligence, reporting, and analytics. Implement data quality, reconciliation, validation, and monitoring processes. … engineering or business intelligence solutions. Strong proven commercial experience with Databricks. Strong experience developing ETL/ELT pipelines and data integration solutions. Experience using Apache Spark, PySpark, and Spark SQL. Strong SQL development and optimisation skills. Experience working with large and complex datasets from multiple source systems. ...

field. Proven experience designing and building scalable data pipelines (ETL/ELT) for batch and streaming use cases using modern technologies (e.g., Python, SQL, Spark, Flink, Airflow) and advanced knowledge on the workings of these. Hands-on experience in building and operating workloads Databricks. Hands-on experience creating data ...

need to obtain and maintain this clearance as a condition of employment. Nice to have Experience with distributed data processing or streaming tools (e.g. Spark, Kafka) Workflow orchestration (e.g. Airflow, Dagster or similar) Working with LLMs, vector databases, embeddings or RAG patterns MLOps practices: model versioning, deployment and monitoring ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

across the team and support the continued growth of Unifeye. What You'll Do Design and build scalable batch and streaming data pipelines using Spark and Databricks. Develop modern Lakehouse architectures using Delta Lake and Medallion (Bronze, Silver and Gold) design principles. Build robust ingestion frameworks from APIs, files … pipelines, deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns … Transformation and quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

Programming-related field Experience influencing technical decisions and supporting capability growth. Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Exposure to infrastructure scaling e.g. Kubernetes Exposure to Ansible and Terraform Understanding of ML algorithms and deployment Experience with SQL Server high-availability ...

operating ETL/ELT pipelines and large-scale data architectures. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

delivery across a range of projects, including data analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases ...

solutions are secure, scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines to ensure trusted ...

environments. Technical Skills: Experience building and operating ETL/ELT pipelines. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We … version control.* **Data Engineering Stack** Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL.* **Orchestration** Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect.* **Data Quality and Lineage** Familiarity with tools ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud-native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

engineering Site Reliability Engineering principles Data & Analytics Experience with technologies such as: Databricks Microsoft Fabric Azure Data Factory Synapse Analytics Snowflake BigQuery Redshift Kafka Apache Spark Knowledge of: Data modelling Data governance Data integration Data migration Data warehousing Master Data Management Applications & Integration Experience designing solutions using ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...