76 to 100 of 207 Apache Jobs

Databricks-based Lakehouse that drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast, trusted … looking for:We’re looking for an experienced Data Engineer with proven expertise in building pipelines in Databricks and a strong grasp of Apache Spark, whether in PySpark or Scala, including Structured Streaming. You should have experience with Kafka (MSK) and real-time data ingestion, as well ...

configuration management and an interest in migrating to Ansible and cloud native tooling. Experience managing Kubernetes platforms and deploying containerised applications. Knowledge of GitLab, Apache NiFi, HashiCorp Vault, Trend Micro Deep Security Manager, Ivanti device control, and Ansible Tower/AWX. Windows Infrastructure Experience with Windows Server administration, deployment ...

configuration management and an interest in migrating to Ansible and cloud native tooling. Experience managing Kubernetes platforms and deploying containerised applications. Knowledge of GitLab, Apache NiFi, HashiCorp Vault, Trend Micro Deep Security Manager, Ivanti device control, and Ansible Tower/AWX. Windows Infrastructure Experience with Windows Server administration, deployment ...

methodologies and a strong philosophy around automated test coverageKnowledge of financial products equity, swaps, futures & fxExperience working with NoSQL database Mongo DBExperience working with Apache FlinkSummaryLocation: London (82)Type: Full time ...

stakeholdersWillingness to pick up and learn new technologies and frameworksNice to have:Knowledge of databases, SQLFamiliarity with Boost ASIOFamiliarity with data serialization formats such Apache Arrow/Parquet, Google Protocol Buffers, FlatbuffersExperience with gRPC, http/REST and Websocket protocolsExperience with Google Cloud/AWS and/or containerization ...

Databases: PostgreSQL, Graph Databases ( Neo4j , Kuzu ), and data mesh/metadata management platforms. Modern Tech Stack: Next.js 14, FastAPI (Python), Casbin RBAC, Pipeline Orchestration, Apache Iceberg, MCP AI Server Integration, and ODCS v2 Data Contracts layer. DevOps & Integration: Azure DevOps, CI/CD, Data Transformation/Quality frameworks, ITSM ...

tools, e.g. NetBoxLog ingestion: OpenSearch/Elasticsearch, Logstash, Kibana, Filebeat, SyslogContainerisation: Docker and/or KubernetesVirtualisation experienceCloud: AWSDatabases: PostgreSQL and/or MySQLApplication Servers: Apache or nginx, WSGIDeep understanding of key service health metricsKnowledge of current security best practices including encryption and systems hardeningExperience implementing technologies with a focus ...

Months contract 2 Days a month onsite - Expenses to Newcastle depends on their hub. Start 3 from offer Experience required AWS Services Terraform Apache Spark Python Airflow Docker Nice to haves Gitlab Security Scanning Agents (eg Trivy, Trend Micro, Wiz etc) Jupyter Notebooks Integration with DWP Integration Team Products ...

expertise with AWS services supporting modern data platforms Proficiency in SQL and Python for data engineering and automation Experience with orchestration tools such as Apache Airflow or MWAA Understanding of data warehousing concepts, dimensional modeling, and modern data architecture patterns Familiarity with CI/CD practices, data quality validation ...

Expopsure to data governance frameworks, metadata management, and regulatory complianc It would be a bonus if you have: Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, real-time/streaming architectures, or data observability tooling Experience leading architecture workshops with both technical and non-technical stakeholders ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

warehousing architecturesExpopsure to data governance frameworks, metadata management, and regulatory compliancIt would be a bonus if you have:Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, real-time/streaming architectures, or data observability toolingExperience leading architecture workshops with both technical and non-technical stakeholdersA background providing ...

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...

Ability to build and deploy scalable applications or libraries Strong debugging, problem-solving, and performance optimisation skills Understanding of open-source licensing (MIT, GPL, Apache) Experience with cloud platforms such as AWS or Azure Ability to create clear, comprehensive technical documentation Experience working within development teams on both minor ...

showing” rather than “telling”, preferring to inspire stakeholders through built examples.Technical Expertise: Deep experience in core data & AI technologies, including languages (Python, SQL), frameworks (Apache Spark) and platforms (Databricks, etc.)Collaboration: You can orchestrate diverse teams to achieve a common goal and explain complex tech to non-technical leaders.Travel ...

building, and operating ETL/ELT pipelines and large-scale data architectures.Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential).Ability ...

Document stores).Hands-on experience building, scaling, and shipping large-scale data platforms in production.Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop).Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis).End-to-end exposure to data pipeline lifecycle … development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices (testing ...

database, storage, AI, IaC/CI/CD tooling, gaming or mediastreaming tech stacks.Demonstrate experience and hands-on skills in open-source software like Apache Web Server, MySQL/PostgreSQL, Redis, Nginx, Terraform.Communicate complex technical content effectively through writing, speaking, and presenting to diverse technical and business audiences.About usAt ...

database, storage, AI, IaC/CI/CD tooling, gaming or mediastreaming tech stacks.Demonstrate experience and hands-on skills in open-source software like Apache Web Server, MySQL/PostgreSQL, Redis, Nginx, Terraform.Communicate complex technical content effectively through writing, speaking, and presenting to diverse technical and business audiences.About usAt ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

Complex modelling and performance-aware DML and DDL across Snowflake/Clickhouse• Kubernetes and Docker• Kafka• Strong understanding of Git, code review, CI, TDD• Apache Airflow - desirable, but not essentialResponsibilities• Design, build and own bespoke low latency pipelines for our most demanding data sources.• On-call rota, owning detection ...

implementing distributed data platforms at scaleProven track record architecting real-time, low-latency streaming platforms and event-driven architecturesStrong hands-on experience with Apache Spark and Databricks, including performance tuning at scaleExperience with Cassandra or similar distributed databases, including partitioning, consistency models, and optimisationSolid experience with OLTP databases such ...

throughput, low-latency software systemsExperience with pipeline orchestration frameworks (e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experienceFor more information about ...

databasesExperience with one of Azure/AWS/GCPETL tools like Informatica, Alteryx and Business Glossary tools like Collibra, Azure Purview or Apache Atlas.TDD/BDD, Experience working in an Agile development environmentIdentity management and synchronisation will be an add on for the resourcesExperience with Data Dictionaries, Business glossaries ...

infrastructure. Well versed and proven experience in programming languages such as Python, Java, or C++. Experience with data pipeline and workflow management tools (e.g., Apache Airflow, Kubeflow). Strong problem-solving skills and ability to work in a fast-paced environment. Excellent communication and collaboration skills. Longstanding experience ...