1 to 25 of 216 Apache Jobs

orchestration tools (e.g., Azure Data Factory, AWS Data Pipeline) Experience with cloud-based data platforms (Azure, Databricks, Synapse, Snowflake) Knowledge of data lake architecture, Apache Spark, and large-scale data processing frameworks Familiarity with DevOps tools and practices (Git, Azure DevOps, CI/CD pipelines) Understanding of Agile methodologies ...

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with Apache Airflow, data modelling, and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit your ...

Azure Database) You are comfortable configuring compute resources, backups, and geolocation requirements Data orchestration: Experience building resilient pipelines using frameworks such as Dagster or Apache Airflow You have a track record of maintaining data health for both real-time streaming and batch processing Systems thinking: A strong understanding ...

warehousing concepts Familiarity with version control systems, particularly Git Desirable Skills: Experience with infrastructure as code tools such as Terraform or CloudFormation Exposure to Apache Spark for distributed data processing Familiarity with workflow orchestration tools such as Airflow or AWS Step Functions Understanding of containerisation using Docker Experience with ...

Azure Database) You are comfortable configuring compute resources, backups, and geolocation requirements Experience building resilient pipelines using frameworks such as Dagster or Apache Airflow You have a track record of maintaining data health for both real-time streaming and batch processing A strong understanding of how data infrastructure integrates ...

systems. • Ability to obtain UK Security Clearance (SC). We'd love it if you also have: • Experience with data visualisation tools such as Apache Superset, Jupytr or Power BI • Familiarity with visualisation libraries like D3.js, Chart.js or Plotly for building interactive dashboards. • Exposure to DevOps practices including ...

stream data processing. Cloud Pub/Sub: For real-time messaging and event ingestion. Cloud Storage: For scalable and durable object storage. Cloud Composer (Apache Airflow): For workflow orchestration and pipeline management. Cloud Dataproc: For managed Hadoop and Spark clusters (good to have). Cloud SQL/Cloud Spanner ...

documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building and maintaining ...

preferred, AWS experience also valued Familiarity with modern data platforms such as Databricks, Snowflake, Microsoft Fabric, or Redshift Understanding of orchestration and workflow management (Apache Airflow, Databricks Workflows, Temporal, or similar) Experience with data quality frameworks and testing strategies for pipelines Familiarity with infrastructure as code tooling (Terraform ...

nodes, _segments, _recovery, _stats), logs, JVM GC analysis, thread-pool diagnostics, and OS-level metrics. Highly Desired Skills and Demonstrated Experience: Demonstrated experience with Apache NiFi for flow-based ingestion and provenance-driven data movement. Demonstrated experience with schema evolution strategies, mapping conflicts, versioning, and automated reindex pipelines. Demonstrated ...

data estate. Deep expertise in Microsoft Azure cloud services - ADF, ADLS, Synapse, Purview. Databricks - Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large-scale distributed computing and performance tuning. Microsoft Fabric - OneLake and DirectLake integration. Azure Synapse Analytics for enterprise-scale data warehousing. Infrastructure-as-Code ...

systems. • Ability to obtain UK Security Clearance (SC). We’d love it if you also have: • Experience with data visualisation tools such as Apache Superset, Jupytr or Power BI • Familiarity with visualisation libraries like D3.js, Chart.js or Plotly for building interactive dashboards. • Exposure to DevOps practices including ...

best practices. AWS and Databricks certifications are a significant plus. Exceptional Python programming skills for data engineering and automation. Experience with Databricks or Apache Spark for big data processing. Experience with at least one major cloud ML platform, eg AWS Sagemaker Advanced SQL capabilities for data manipulation, querying ...

pipelines (Jenkins, GitLab CI, GitHub Actions, etc.) Data Engineering & Big Data Technologies Experience working with large-scale datasets and distributed processing frameworks such as: Apache Spark, Hadoop, or Flink Familiarity with data streaming technologies (Kafka, Kinesis) Experience with databases: Relational: PostgreSQL, MySQL NoSQL: MongoDB, Elasticsearch, DynamoDB Cloud & MLOps Hands ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google Cloud ...

experience working with data warehouses. Solid understanding of ETL/ELT design and data pipeline optimisation. Experience with big data frameworks such as Apache Spark, and exposure to streaming technologies like Kafka. Strong problem-solving skills and the ability to collaborate effectively across teams. What's on offer: Opportunity ...

development. Experience with OAuth2, JWT, and AWS Cognito for authentication and authorization. Familiarity with Maven for build and dependency management. Experience with JSP and Apache Tiles for server-side rendering. Exposure to Ionic/Capacitor for mobile development. Experience with Azure DevOps Pipelines or Jenkins for CI/CD. ...

support Workflow orchestration (Airflow or Dagster) AWS data services (Athena, S3, MWAA, Iceberg) Data quality monitoring & incident management Nice to Have: Airbyte DBT Apache Spark Grafana FTP/data distribution platforms Financial data experience ...

Delta Lake Familiarity with Azure services such as Functions, App Services, Data Lake, and event-driven components Experience with big data technologies such as Apache Spark Strong system design and architecture skills with ability to explain technical decisions clearly Experience with version control (Git, GitHub, Azure DevOps) Exposure ...

frameworks Develop robust backend services, APIs, and microservices architectures Big Data & Data Processing Work with large-scale data processing frameworks such as Hadoop, Apache Spark, and Kafka Design and implement data pipelines to ingest, transform, and process structured and unstructured data Optimize data workflows for performance, scalability, and reliability ...

with Git-based workflows, branching strategies, and version control best practices What Would Be Nice to Have Experience deploying and managing data platform components: Apache Spark, Tri no, Apache Ranger, Apache Iceberg, Apache Superset, data catalogs Familiarity with data lakehouse architectures and supporting infrastructure Experience with ...

databricks Proficiency in programming languages such as Python, Spark, SQL. Strong experience with SQL databases. Expertise in data pipeline and workflow management tools (e.g., Apache Airflow, ADF). Experience with cloud platforms (Azure preferred) and related data services. Knowledge of big data technologies (e.g., Hadoop, Spark, Kafka). Experience ...

Containerization and orchestration technologies (Docker, Kubernetes) Cloud platforms (AWS GovCloud, SC2S, C2S) Object-oriented design and development Data streaming and integration technologies such as Apache Nifi Infrastructure as Code (Terraform) GraphQL Microservices architecture Performance optimization and scalability design Cloud Certification Benefits Generous cost sharing for medical insurance ...

DevSecOps practices and tools Performance optimization and scalability design Security engineering principles Frontend frameworks (Angular, React, Svelte) Data streaming and integration technologies such as Apache Nifi Infrastructure as Code (Terraform) GraphQL Microservices architecture SAFe Agile methodologies Working in classified environments Strong understanding of forensic and investigative system requirements Demonstrated ...

Agile Development using SCRUM. Experience of Oracle/Relational Databases and/or Mongo. Experience of GitLab CI/CD Pipelines. Experience of Apache NiFi. Experience in the use of Atlassian Suite - Bitbucket, Jira, Confluence. Experience of Elasticsearch and Kibana. Experience of Hibernate. Experience of AWS Services such ...