1 to 25 of 46 Permanent Parquet Jobs in the UK

machine learning features and predictive modellingConsume data from a variety of sources (RDBMS, APIs, FTPs and other cloud storage) & formats (Excel, CSV, XML, JSON, Parquet, Unstructured)Write advanced/complex SQL with performance tuning and optimizationBuild AI based solutions for solving business needs, process automations and improving operational efficiencyIdentify ...

Airflow, Step Functions, etc.). Databricks or AWS certifications. Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning). Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats. #J-18808-Ljbffr ...

data orchestration tools (Airflow, Step Functions, etc.).Databricks or AWS certifications.Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.J.P. Morgan is a global leader in financial services, providing strategic advice and products to the world ...

orchestration tools (Airflow, Step Functions, etc.).Databricks or AWS certifications. Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.J.P. Morgan is a global leader in financial services, providing strategic advice and products to the world ...

data warehousing (Redshift, dimensional modeling) Proficiency in Python or Scala for data engineering Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet) Experience migrating data pipelines from on-prem or other cloud platforms Understanding of data governance, security, and access control on AWS Experience with ...

/Azure DevOps pipelines Experience validating large-scale datasets and data transformations Knowledge of data warehousing concepts and dimensional modeling Understanding of Delta Lake, Parquet, and Lakehouse architectures Experience with CI/CD and automated deployment validation Familiarity with Agile/Scrum delivery methodologies Microsoft Fabric Components Knowledge Experience ...

environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with a modern and evolving data stack. Candidates are not expected to have deep expertise ...

environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . For More Experienced Candidates Stronger ownership of technical design across multiple datasets or pipeline domains. Experience guiding implementation standards, code quality and engineering ...

Automation Python and SQL PostgreSQL and Amazon RDS Power Automate Power BI REST API and enterprise integration architecture JSON, XML, XSD, CSV, and Parquet OCR and unstructured-document processing Data modeling and metadata management IAM, role-based security, and secrets management Git, CI/CD, version control, and deployment ...

production data pipelines in a collaborative engineering environment Experience with distributed data processing frameworks such as Apache Spark Working knowledge of JSON, Avro and Parquet Nice to have: technical design ownership across multiple datasets or pipeline domains, experience guiding implementation standards and engineering practices, ability to lead delivery ...

invalidation, persistence strategy, data consistency MongoDB TimescaleDB NATS JetStream AWS S3/Glue Data lake staging, archival, batch ingestion into Snowflake Partitioning, file format (Parquet/ORC), Lake Formation governance Finance Domain Knowledge Candidates should have hands‐on data architecture experience in one or more of the following financial ...

Experience with Delta Lake and Lakehouse architectures Good grasp of data modelling, orchestration and workflow automation Comfortable working with structured and semi-structured data (Parquet, JSON, CSV) Solid understanding of CI/CD and Git-based workflows Experience with data quality checks, testing and monitoring WhyApply? Genuine input into ...

building lean processes Experience with distributed data processing Experience with monitoring, testing, and data quality Experience building robust pipelines from diverse sources e.g. parquet, SQL & no-SQL databases, API endpoints It would be helpful to have experience/expertise/knowledge in the following: Python AWS Kubernetes Spark & distributed ...

unifying and normalising data into a relational store with proper versioning and lineage; polling an external API and landing it as a Hive-partitioned Parquet dataset; storing document blobs and their metadata via a document-management approach. There will be many more shapes; your job is to choose ...

engineering, or distributed systems — with at least 3 years managing engineering teamsExperience and knowledge of building data-lakes in AWS (i.e. Spark, Athena, Iceberg, Parquet, Presto), including data modeling, data quality best practices, and self-service tooling.Strong expertise in building and operating real-time data at scale including Kafka ...

Orchestra for task orchestration Terraform for infrastructure Lightdash for data applications Airflow and Astronomer for job scheduling and tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing SparkSQL for analytics Why else you'll love it here ...

evolve the platform. Experience is helpful, but not essential: Languages: Python, SQL, Terragrunt, JavaScript Cloud: AWS (primary), some GCP Warehouse & Storage: Snowflake, S3/Parquet Data & ETL: dbt, Fivetran Platform & Infra: Kubernetes, Kafka, RabbitMQ, Argo, GitHub Actions, HashiCorp Vault Observability: Datadog, Grafana Dashboarding: Preset Other: Claude What ...

ways of working, and presenting these to the community Here's a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT What we're looking for We'd love to hear from ...

experience with Apache Spark and Apache Flink. Strong understanding of Apache HTTP, Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data formats. Knowledge of Iceberg, Databricks, and Hadoop ecosystems. Skills that will help Follows TDD, ATDD and Clean Coding practices and used to setting ...

sourced systems.Hands-on experience with Apache Spark and Apache Flink.Strong understanding of Apache HTTP, Envoy, and REST API design.Experience working with Protobuf, Avro, and Parquet data formats.Knowledge of Iceberg, Databricks, and Hadoop ecosystems.Skills that will help:Follows TDD, ATDD and Clean Coding practices and used to setting ...

Learning – Understanding of AI/ML concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications• Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications• Advanced Data Management – Strong SQL design, query optimization ...

Requirements: 6+ years building production software in Python, with strong engineering fundamentals (testing, performance, clean design). Solid data engineering: SQL, columnar formats (e.g. Parquet), pipeline design, and handling datasets large enough that naive approaches don’t scale. Hands-on experience with at least one analytical or query engine ...

definition Kubernetes for data services and task orchestration Argo CD for application deployments Airflow purely for job scheduling and tracking CircleCI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Postgres/Aurora for our relational databases Spark for data processing dbt for data ...

market data, or other high volume time series data.Familiarity with SQL and at least one columnar/time‐series store (e.g., kdb+, ClickHouse, InfluxDB, Parquet/Arrow).Proficiency with Linux development, Git, containers, and CI/CD workflows.Excellent problem‐solving abilities, attention to detail, and clear communication skillsNice ...

real downstream consumers. Work with economists and engineers to turn modelling decisions into reliable production data. Essential Strong production Python. Datasets in pandas and Parquet/Arrow, plus an analytical engine, e.g. DuckDB, or a warehouse such as Snowflake. Orchestrated batch pipelines you've operated, not just written: Dagster ...