IAM, S3, RDS, EMR, ECS and more Advanced experience working and understanding the tradeoffs of at least one of the following Data Lake table/file formats: Delta Lake, Parquet, Iceberg, Hudi Previous h ands-on expertise with Spark Experience working with containerisation technologies - Docker, Kubernetes Streaming Knowledge: Experience with Kafka/Flink or other streaming ecosystems, with a More ❯
Agile software development framework. Be familiar with Data Build Tool (DBT) for building data models, tests, and transformations. Have a thorough understanding of distributed file and table formats like Parquet, Delta, Iceberg, Hudi. Preferred Experience with Infrastructure as Code (IaC) solutions such as Terraform or Pulumi. Experience with modern CI/CD DevOps frameworks. Experience developing data visualizations using More ❯
self-teach new tools and technologies; an ability to educate others Familiarity with the Linux command line and shell scripting Understanding of basic data concepts - file formats (e.g. CSV, Parquet), data pipelines, and storage layers Exposure to containers and Podman or Docker Comfortable using Git; an awareness of CI/CD practices and tools such as GitHub Actions or More ❯
file drops, message queues (SQS, Kafka), and 3rd party SaaS integrations, with idempotency and error handling. Storage & Query Engines: Strong with RDBMS (PostgreSQL, MySQL), NoSQL (DynamoDB, Cassandra), data lakes (Parquet, ORC), and warehouse paradigms. Observability & Quality: Deep familiarity with metrics, logging, tracing, and data quality tools (e.g., Great Expectations, Monte Carlo, custom validation/test suites). Security & Governance More ❯
Data modelling (building optimised and efficient data marts and warehouses in the cloud) Work with Infrastructure as code (Terraform) and containerising applications (Docker) Work with AWS, S3, SQS, Iceberg, Parquet, Glue and EMR for our Data Lake Experience developing CI/CD pipelines More information: Enjoy fantastic perks like private healthcare & dental insurance, a generous work from abroad policy More ❯
Birmingham, West Midlands, United Kingdom Hybrid / WFH Options
Amtis Professional Ltd
continuously improve data infrastructure Explore AI-driven enhancements to boost data accuracy and productivity Requirements: Strong experience with: Azure Databricks, Data Factory, Blob Storage Python/PySpark SQL Server, Parquet, Delta Lake Deep understanding of: ETL/ELT, CDC, stream processing Lakehouse architecture and data warehousing Scalable pipeline design and database optimisation A proactive mindset, strong problem-solving skills More ❯
at least two major cloud environments (AWS, GCP, Azure). ? In-depth knowledge of AWS Glue, including job authoring, triggers, and cost-aware configuration. ? Familiarity with distributed data formats (Parquet, Avro), data lakes (Iceberg, Delta Lake), and cloud storage systems (S3, GCS, Azure Blob). ? Hands-on experience with Docker, Kubernetes, and CI/CD pipelines. ? Strong documentation and More ❯
Data Lake and create secure, efficient, and scalable environments for our data platforms. Leveraging cloud-native technologies and AWS tools such as AWS S3, EKS, Glue, Airflow, Trino, and Parquet, you will prepare to adopt Apache Iceberg for greater performance and flexibility. You'll address high-performance data workloads, ensuring seamless execution of massive queries, including 600+ billion-row More ❯
Birmingham, West Midlands, United Kingdom Hybrid / WFH Options
MYO Talent
role Large and complex datasets Azure, Azure Databricks Microsoft SQL Server Lakehouse, Delta Lake Data Warehousing ETL CDC Stream Processing Database Design ML Python/PySpark Azure Blob Storage Parquet Azure Data Factory Desirable: Any exposure working in a software house, consultancy, retail or retail automotive sector would be beneficial but not essential. More ❯
EMR. Experience with RDBMS. Experience with Jenkins CI/CD pipeline. Bachelor's degree in a technical discipline. Plus: Knowledge of Hadoop/Spark and various data formats like Parquet, CSV, etc. Additional Information Benefits/Perks: Great compensation package and bonus plan Core benefits including medical, dental, vision, and matching 401K Flexible work environment, ability to work remote More ❯
and Unix skills. Highly proficient working with cloud environments (ideally Azure), distributed computing and optimising workflows and pipelines. Experience working with common data transformation and storage formats, e.g. ApacheParquet, Delta tables. Strong experience working with containerisation (e.g. Docker) and deployment (e.g. Kubernetes). Experience with Spark, Databricks, data lakes. Highly proficient in working with version control and Git More ❯
Cloud infrastructure: GCP. Qualifications: - At least 4+ year's work experience including hands-on as a Data engineer on modern cloud data platforms - Experience in different data formats (Avro, Parquet) - Experience in data query languages (SQL or similar) - Experience in data centric programming using one of more programming languages Python, Java/or Scala. - Good understanding of different data More ❯
and RAG-based solutions Proficiency in Python and modern AI/ML libraries (e.g. HuggingFace, LangChain, TensorFlow, PyTorch) Experience with data exchange and storage frameworks (e.g. APIs, SQL, NoSQL, Parquet) Track record of delivering technical solutions in Agile environments Excellent communication skills and a collaborative mindset Beneficial, but not essential: Experience with containerisation (Docker) Awareness of secure data handling More ❯
with a keen understanding of LLM and RAG technologies. Strong development capabilities, particularly in Python. Experience with data exchange, processing, and storage frameworks (ETL, ESB, API, SQL, NoSQL, and Parquet). Comfort with Agile development methodologies. Excellent teamwork and communication skills, with a talent for translating technical concepts into actionable insights for non-specialists. Ability to influence company decision More ❯
our datalake platform Kubernetes for data services and task orchestration Terraform for infrastructure Streamlit for data applications Airflow purely for job scheduling and tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data processing DBT for data modelling SparkSQL for analytics Why else you'll love it here Wondering More ❯
an expert BS degree in Computer Science or meaningful relevant work experience Preferred Qualifications Experience with large scale data platform infrastructure such as Spark, Flink, HDFS, AWS/S3, Parquet, Kubernetes is a plus More ❯
time for your personal development What you'll be working with: •Backend: Distributed, event-driven core Java (90% of the code-base), MySQL, Kafka •Data analytics: Python & Jupyter notebooks, Parquet, Docker •Testing: JUnit, JMH, JCStress, Jenkins, Selenium, many in-house tools •OS: Linux (Fedora for development, Rocky in production) The LMAX way is to use the right tool for More ❯
role Prior experience developing data quality control processes to detect data gaps or inaccuracies is a plus Familiarity with compressed/optimized file formats for data storage such as Parquet and HDF5 Prior experience providing technical guidance to junior data analysts Prior experience mapping security identifier data or working with a security master database would be beneficial Prior experience More ❯
time for your personal development What you'll be working with: •Backend: Distributed, event-driven core Java (90% of the code-base), MySQL, Kafka •Data analytics: Python & Jupyter notebooks, Parquet, Docker •Testing: JUnit, JMH, JCStress, Jenkins, Selenium, many in-house tools •OS: Linux (Fedora for development, Rocky in production) The LMAX way is to use the right tool for More ❯
About the role Taktile is a high-growth, post product-market-fit start-up, on a fast trajectory to becoming market leader in the field of automated decisioning. We are looking for a Full-stack Engineer to join the Decide More ❯
tech ML startup, unblocking a high standard of execution across the company. Architecting and building our core data infrastructure for managing large-scale ML training datasets (e.g., Apache Iceberg, Parquet). Develop cloud-based data processing pipelines , that ingest and compute auxiliary metadata signals on image, video and 3D data (PySpark, Airflow). Develop a data serving strategy for … Qualifications: 3 years full-time professional experience, committing code to a production environment. Proficiency in large-scale data processing (e.g. Spark, Cloud SQL ), and large-scale data systems (Iceberg, Parquet, ) Proficiency in cloud platforms (e.g. AWS, GCP, Azure). Proficiency in the Python ecosystem and its best practices Experience in CI/CD (e.g. CircleCI). Preferred Qualifications Familiarity More ❯
Location London, United Kingdom Employment Type Full time Location Type Remote Department R&D Investigations The engineering team at Chainalysis is inspired by solving the hardest technical challenges and creating products that build trust in cryptocurrencies. We're a global More ❯
Python & C# Developer - Front Office Risk (VP) Commodities Tech Overview CITIGROUP is one of the largest players in the financial industry, employing more than 200,000 people across the globe. What We do/The Team The Commodities trading business More ❯
