1 to 25 of 43 Parquet Jobs in England

/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture Hybrid cloud environments Enterprise integration patterns Modern data formats including Parquet, Avro, and ORC DevOps & Platform Engineering Experience implementing CI/CD pipelines and DevOps practices. Experience with infrastructure automation and environment management. Familiarity with ...

data warehousing (Redshift, dimensional modeling) Proficiency in Python or Scala for data engineering Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet) Experience migrating data pipelines from on-prem or other cloud platforms Understanding of data governance, security, and access control on AWS Experience with ...

environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with a modern and evolving data stack. Candidates are not expected to have deep expertise ...

environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design across multiple datasets or pipeline domains. Experience guiding implementation standards, code quality and engineering ...

Automation Python and SQL PostgreSQL and Amazon RDS Power Automate Power BI REST API and enterprise integration architecture JSON, XML, XSD, CSV, and Parquet OCR and unstructured-document processing Data modeling and metadata management IAM, role-based security, and secrets management Git, CI/CD, version control, and deployment ...

production data pipelines in a collaborative engineering environment Experience with distributed data processing frameworks such as Apache Spark Working knowledge of JSON, Avro and Parquet Nice to have: technical design ownership across multiple datasets or pipeline domains, experience guiding implementation standards and engineering practices, ability to lead delivery ...

invalidation, persistence strategy, data consistency MongoDB TimescaleDB NATS JetStream AWS S3/Glue Data lake staging, archival, batch ingestion into Snowflake Partitioning, file format (Parquet/ORC), Lake Formation governance Finance Domain Knowledge Candidates should have hands‐on data architecture experience in one or more of the following financial ...

Experience with Delta Lake and Lakehouse architectures Good grasp of data modelling, orchestration and workflow automation Comfortable working with structured and semi-structured data (Parquet, JSON, CSV) Solid understanding of CI/CD and Git-based workflows Experience with data quality checks, testing and monitoring WhyApply? Genuine input into ...

building lean processes Experience with distributed data processing Experience with monitoring, testing, and data quality Experience building robust pipelines from diverse sources e.g. parquet, SQL & no-SQL databases, API endpoints It would be helpful to have experience/expertise/knowledge in the following: Python AWS Kubernetes Spark & distributed ...

unifying and normalising data into a relational store with proper versioning and lineage; polling an external API and landing it as a Hive-partitioned Parquet dataset; storing document blobs and their metadata via a document-management approach. There will be many more shapes; your job is to choose ...

Orchestra for task orchestration Terraform for infrastructure Lightdash for data applications Airflow and Astronomer for job scheduling and tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here ...

evolve the platform. Experience is helpful, but not essential: Languages: Python, SQL, Terragrunt, JavaScript Cloud: AWS (primary), some GCP Warehouse & Storage: Snowflake, S3/Parquet Data & ETL: dbt, Fivetran Platform & Infra: Kubernetes, Kafka, RabbitMQ, Argo, GitHub Actions, HashiCorp Vault Observability: Datadog, Grafana Dashboarding: Preset Other: Claude What ...

ways of working, and presenting these to the community Here's a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT What we're looking for We'd love to hear from ...

experience with Apache Spark and Apache Flink. Strong understanding of Apache HTTP, Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data formats. Knowledge of Iceberg, Databricks, and Hadoop ecosystems. Skills that will help Follows TDD, ATDD and Clean Coding practices and used to setting ...

sourced systems.Hands-on experience with Apache Spark and Apache Flink.Strong understanding of Apache HTTP, Envoy, and REST API design.Experience working with Protobuf, Avro, and Parquet data formats.Knowledge of Iceberg, Databricks, and Hadoop ecosystems.Skills that will help:Follows TDD, ATDD and Clean Coding practices and used to setting ...

Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications• Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications• Advanced Data Management – Strong SQL design, query optimization ...

Requirements: 6+ years building production software in Python, with strong engineering fundamentals (testing, performance, clean design). Solid data engineering: SQL, columnar formats (e.g. Parquet), pipeline design, and handling datasets large enough that naive approaches don’t scale. Hands-on experience with at least one analytical or query engine ...

definition Kubernetes for data services and task orchestration Argo CD for application deployments Airflow purely for job scheduling and tracking CircleCI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Postgres/Aurora for our relational databases Spark for data processing dbt for data ...

market data, or other high volume time series data.Familiarity with SQL and at least one columnar/time‐series store (e.g., kdb+, ClickHouse, InfluxDB, Parquet/Arrow).Proficiency with Linux development, Git, containers, and CI/CD workflows.Excellent problem‐solving abilities, attention to detail, and clear communication skillsNice ...

real downstream consumers. Work with economists and engineers to turn modelling decisions into reliable production data. Essential Strong production Python. Datasets in pandas and Parquet/Arrow, plus an analytical engine, e.g. DuckDB, or a warehouse such as Snowflake. Orchestrated batch pipelines you've operated, not just written: Dagster ...

SOAP services, file transfers, batch ingestion, legacy adapters and bespoke interfaces. Experience with structured, semi-structured and unstructured data formats including CSV, XML, JSON, Parquet, Avro and proprietary formats. Experience designing solutions that operate across multiple security classifications and secure domains. Strong knowledge of data governance, lineage, metadata management ...

governance Experience designing REST, SOAP, file, batch, legacy and bespoke integrations Knowledge of structured, semi-structured and unstructured data, including CSV, XML, JSON, Parquet and Avro Experience designing solutions across multiple security classifications Strong knowledge of data governance, lineage, metadata, data quality, access control and auditability Strong stakeholder engagement ...

work effectively with remote teams"**Experience with the Following is Desirable:*** Go* Pinot or other time-series/OLAP-style database* Iceberg* Kubernetes* Jenkins* Parquet* Protocol Buffers/GRPC#LI-MZ1**Benefits of Working at CrowdStrike:*** Market leader in compensation and equity awards* Comprehensive physical and mental wellness programs* Competitive ...

Glue Amazon S3 AWS Lake Formation Amazon Athena Experience building Lakehouse architectures and modern data platforms. Experience working with data formats such as Parquet and Iceberg tables. Strong experience with Terraform for infrastructure automation. #J-18808-Ljbffr ...

PostgreSQL, and cloud databases Proven track record with complex data migration projects (terabyte+ datasets, multiple legacy source systems, structures and unstructured data) Proficiency with Parquet/Delta Lake or other modern data storage formats Experience with streaming architectures using Kafka, Event Hubs, or Kinesis for real-time data processing ...