1 to 25 of 73 Permanent Apache Jobs

patterns (YAML, PySpark, SQL-driven generation from mapping documents) Databricks & Lakehouse Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred) Strong Apache Spark skills (PySpark/Spark SQL) Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse Ability to re-implement ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands-on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud-native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal experience ...

understanding of data modeling. Familiarity with data governance, lineage, and quality frameworks. Fluency with AI tools and ADLC Working knowledge of Databricks Experience with Apache Spark Preferred: Hands-on experience with AWS data services (e.g., S3, Glue, Lambda, MSK). Proficient in CI/CD pipelines, automated testing, code ...

collaboration skills. Desirable Experience Experience with AWS services such as S3, Glue, Lambda, Step Functions, Redshift or related data engineering technologies. Familiarity with Apache Spark, Airflow, distributed processing, data lakes and data warehouses. Experience with infrastructure as code, Docker or automated deployment pipelines. Background in government, defence, national security ...

Lake. Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security. Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL). Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage, and fine-grained ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance practices ...

modern data platforms (Azure Databricks, Delta Lake, Microsoft Fabric) Experience building data pipelines, dashboards, automation solutions and reporting (Power BI) Knowledge of Apache Spark, enterprise integrations, REST APIs, SDKs and software delivery practices (testing, version control, CI/CD) Experience with SAP data structures (SAP S/4HANA); familiarity ...

with monitoring/logging tools (ELK stack, Prometheus, Splunk) Familiarity with service mesh and modern networking (Istio, Envoy) Exposure to data engineering and integration (Apache ecosystem, SQL, Python) Recognised technical certifications (AWS/Azure Architect, Kubernetes CKA/CKAD, Terraform) Experience in influencing technical standards, best practices, or contributing ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data Meshes ...

Directory, Group Policy, DNS, DHCP PowerShell, Bash, Python VMware vSphere, Hyper-V Git, Azure DevOps, GitLab, Jenkins, GitHub Actions Terraform, Ansible Docker, Kubernetes IIS, Apache, Nginx Monitoring and logging platforms Backup and disaster recovery solutions $145,000 - $160,000 annually. Compensation will be determined based on the selected candidate ...

production problems and taking responsibility for critical infrastructure. Startup Mindset: Proactive, resourceful and comfortable taking ownership in a fast-moving engineering environment. Bonus : with Apache Airflow, real-time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial-market systems. Benefits An Ambitious, Collaborative Culture: Join ...

containerisation and orchestration technologies including Kubernetes, Docker, ECS, EKS, and AKS . Experience working with distributed event-streaming technologies such as Kafka and Apache Flink . Strong troubleshooting skills with the ability to resolve complex technical and operational issues. Proven ability to lead, mentor, and develop engineers within Agile ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications • Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications • Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise • Observability – Experience ...

real-time data processing. Lead the architecture of scalable data pipelines integrating multiple enterprise source systems. Define data integration and processing patterns using Apache Spark, PySpark, Databricks, Cloudera/Hadoop, Kafka, and cloud-native services. Establish best practices for data ingestion, transformation, orchestration, processing, and delivery. Optimize data pipelines … HDFS, Hive, Spark, YARN Data Lakes & Data Warehouses Delta Lake Data Mesh/Data Fabric Data Engineering ETL/ELT Architecture Enterprise Data Pipelines Apache Spark/PySpark Kafka Batch & Real-Time Processing Streaming Architecture Data Quality & Observability Cloud Platforms AWS Microsoft Azure Google Cloud Platform (GCP) Cloud Security ...

Expopsure to data governance frameworks, metadata management, and regulatory complianc It would be a bonus if you have: Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, real-time/streaming architectures, or data observability tooling Experience leading architecture workshops with both technical and non-technical stakeholders ...

effectively with technical teams including Data Scientists and MLOps Engineers. Desirable Skills Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/ ...

build elegant solutions Understanding of system administration tasks on some of the following; Operating systems (e.g. Linux, CoreOS and Windows) Application stacks (e.g. nginx, Apache, Tomcat, JBoss, Websphere, Hybris, IIS, .Net, Node.js, Solr) RDBMS (e.g. PostgreSQL, MySQL, Oracle, MS SQL) NoSQL (e.g. MongoDB, Cassandra, Riak) Containerisation and microservices (e.g. ...

technical stakeholders. Desirable Experience AWS services, including RDS, S3 and CloudWatch. CI/CD pipelines using GitHub Actions, GitLab CI or similar. Nginx or Apache administration. Experience within eCommerce, financial or operational business environments. Leading technical projects or workstreams. If you're an experienced PHP Developer who enjoys solving ...

experience with cloud platforms (AWS, Azure, GCP) Experience in Python, with experience in frameworks like Tensorflow, PyTorch, and tools for building ETL pipelines (e.g. Apache NiFi, Airflow) Practical experience with NLP and/or Search techniques Prompt engineering, and LLMs for enterprise-scale applications. You have team lead experience ...