1 to 25 of 69 Parquet Jobs in England

dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Exposure to data governance concepts: critical data elements (CDEs), lineage, data quality dimensions, audit frameworks. Experience running delivery (intake, triage, prioritization ...

data integration components and services, including Java-based microservices where required Prepare, transform and validate structured and semi-structured data, including formats such as parquet, avro and JSON Write efficient SQL to query, validate and optimise data across cloud-based analytical platforms such as Athena, Redshift and Azure Synapse ...

dashboards (data validation, extract refresh checks, reconciliation to source). Nice-to-Have Experience with tools such as PySpark, Spark SQL, Hive, Impala, HDFS, Parquet, and Oracle databases. Hands-on exposure to DCRM tooling and operational exception management processes. Experience with governance/catalog tools and lineage documentation methods ...

Python for data engineering and automation. Hands‐on experience with PySpark and Spark SQL in production environments. Solid knowledge of Hive, Impala, HDFS, and Parquet . Advanced SQL skills; experience with Oracle databases . Experience designing and supporting ETL/ELT pipelines and data quality frameworks . Working knowledge ...

work across comparable technologies. Examples Of Technologies In Scope Include Data processing and logic: ANSI SQL, Apache Spark, Kafka Data formats: JSON, Avro, Parquet Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised or Kubernetes-based deployment approaches ...

data warehousing (Redshift, dimensional modeling) Proficiency in Python or Scala for data engineering Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet) Experience migrating data pipelines from on-prem or other cloud platforms Understanding of data governance, security, and access control on AWS Experience with ...

Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet DevOps and DataOps Experience of IaC tools, particularly Terraform Understanding of implementing logging, monitoring and alerting for enterprise level pipelines Understanding of access controls, policies ...

Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet DevOps and DataOps Experience of IaC tools, particularly Terraform Understanding of implementing logging, monitoring and alerting for enterprise level pipelines Understanding of access controls, policies ...

based environment Advanced experience working and understanding the tradeoffs of at least one of the following Data Lake table/file formats: Delta Lake, Parquet, Iceberg, Hudi Experience working with lakehouse/medallion architectures in Databricks Experience with experimentation support tooling (such as Optimizely) A genuine curiosity about ...

environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with a modern and evolving data stack. Candidates are not expected to have deep expertise ...

including Kafka, Azure Functions or Azure Service Bus. Strong understanding of Spark processing and performance tuning. Experience working with file formats and partitioning, including Parquet, JSON, XML and CSV. Experience with Azure DevOps/GitHub and CI/CD. Strong programming experience in Python, with knowledge of other programming ...

environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with a modern and evolving data stack. Candidates are not expected to have deep expertise ...

required• Event streaming or real-time processing with Kafka, Kinesis, Flink, Spark Structured Streaming, or similar technologies.• Distributed processing with Spark and familiarity with Parquet, Avro, JSON, and open table formats.• Search or retrieval systems such as Elasticsearch/OpenSearch, vector databases, or embedding pipelines.• Feature stores, ML data ...

Fabric, Medallion and Data Products. Hands on mindset : Fluent in the language of data, whether SQL, Python, PySpark, R or Scala. Open formats including Parquet, Iceberg and JSON. Agile Ways of Working Run and contribute to agile ceremonies, helping to shape and improve the agile process Work with ...

Automation Python and SQL PostgreSQL and Amazon RDS Power Automate Power BI REST API and enterprise integration architecture JSON, XML, XSD, CSV, and Parquet OCR and unstructured-document processing Data modeling and metadata management IAM, role-based security, and secrets management Git, CI/CD, version control, and deployment ...

data orchestration across multiple sources and systems. Experience designing data models, schemas, data contracts and integration frameworks. Strong understanding of REST API integration, Parquet data formats and S3-compatible object storage. Experience deploying or supporting data platforms within OpenShift or Kubernetes environments. Knowledge of GitLab CI/ ...

production data pipelines in a collaborative engineering environment Experience with distributed data processing frameworks such as Apache Spark Working knowledge of JSON, Avro and Parquet Nice to have: technical design ownership across multiple datasets or pipeline domains, experience guiding implementation standards and engineering practices, ability to lead delivery ...

invalidation, persistence strategy, data consistency MongoDB TimescaleDB NATS JetStream AWS S3/Glue Data lake staging, archival, batch ingestion into Snowflake Partitioning, file format (Parquet/ORC), Lake Formation governance Finance Domain Knowledge Candidates should have hands‐on data architecture experience in one or more of the following financial ...

Experience with Delta Lake and Lakehouse architectures Good grasp of data modelling, orchestration and workflow automation Comfortable working with structured and semi-structured data (Parquet, JSON, CSV) Solid understanding of CI/CD and Git-based workflows Experience with data quality checks, testing and monitoring Why Apply? Genuine input ...

building lean processes Experience with distributed data processing Experience with monitoring, testing, and data quality Experience building robust pipelines from diverse sources e.g. parquet, SQL & no-SQL databases, API endpoints It would be helpful to have experience/expertise/knowledge in the following: Python AWS Kubernetes Spark & distributed ...

building lean processes Experience with distributed data processing Experience with monitoring, testing, and data quality Experience building robust pipelines from diverse sources e.g. parquet, SQL & no-SQL databases, API endpoints It would be helpful to have experience/expertise/knowledge in the following: Python AWS Kubernetes Spark & distributed ...

Role Description The Technical Analyst – Data Lineage will support a Data Governance, Controls, and Reporting program for a top-tier banking client. Responsible for establishing and validating end-to-end lineage across critical datasets used ...

Orchestra for task orchestration Terraform for infrastructure Lightdash for data applications Airflow and Astronomer for job scheduling and tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here ...

Modern Tech Stack: Experience is helpful, but not essential: Languages: Python, SQL, Terragrunt, JavaScript Cloud: AWS (primary), some GCP Warehouse & Storage: Snowflake, S3/Parquet Data & ETL: dbt, Fivetran Platform & Infra: Kubernetes, Kafka, RabbitMQ, Argo, GitHub Actions, HashiCorp Vault Observability: Datadog, Grafana Dashboarding: Preset Other: Claude WHAT ...

modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic modelling, summarisation ...