1 to 25 of 178 Apache Jobs

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling (e.g. ...

skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure as Code. Git/GitLab and modern software development practices. API integration ...

Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/GCP) Experience ...

business stakeholders AWS & Data Engineering : Very strong hands on experience with AWS services (S3, Lambda, Glue, Athena, Redshift, EMR, etc.) and proficiency with Apache Spark for large scale data processing Backend Development : Strong experience with Python for building data processing services and APIs, plus expert level SQL for data ...

Engineer, Principal Data Engineer or Technical Lead within a large-scale, data-driven organisation Strong hands-on software engineering expertise with Python, PySpark and Apache Airflow Deep experience designing and building distributed data processing systems and large-scale data platforms on AWS Strong knowledge of modern AWS data technologies ...

years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

scripting and data engineering. Core Tech Stack: Deep proficiency with Python, Postgres SQL, Azure Databricks, AWS (S3), Git, Azure DevOps CI/CD, and Apache Airflow. Python Ecosystem: Extensive, recent hands-on experience utilizing NumPy, Pandas, Polars, BeautifulSoup, Selenium, and Requests . Data Ecosystems: Strong background in building native ...

based solutions. Experience working with Cosmos DB SDKs in technologies such as .NET, Java, Python, or Node.js. Strong understanding of distributed computing concepts and Apache Spark architecture. Hands-on experience developing data pipelines using PySpark. Experience with Pytest and automated testing frameworks. Strong understanding of partitioning, indexing strategies ...

Mentor data engineers and other team members who want to grow towards architecture roles. Required technical skills & knowledge: Azure databricks, Microsoft Fabric, Power BI, Apache Spark, Azure Synapse, Azure Data Lake, Azure Data Factory, Python, PySpark, SQL Your profile You don't need to tick every ...

/CD pipelines and deployment automation. • Familiarity with monitoring, logging, and observability tools. • Awareness of application platforms and middleware such as Java, Tomcat, Apache, and NodeJS. • Experience using Git and collaborative development workflows. • Basic scripting skills using Shell or Python. • Understanding of containerisation concepts using Docker. • Strong problem-solving ...

Mentor data engineers and other team members who want to grow towards architecture roles. Required technical skills & knowledge: Azure databricks, Microsoft Fabric, Power BI, Apache Spark, Azure Synapse, Azure Data Lake, Azure Data Factory, Python, PySpark, SQL Your profile You don't need to tick every ...

principles Experience working within an SDLC with an integrated development team using CI/CD tooling including Git Desirable Skills Hands-on experience with Apache NiFi for data pipeline development and orchestration AWS experience including familiarity with EMR, S3 and SQS Awareness of domain-driven design approaches and architecture ...

others. Active SC Clearance, or eligibility to obtain. Desirable: SAS Viya for data analytics. Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). Denodo for data virtualisation. Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). ALM Tooling (Jira, Confluence). ...

platform role Strong hands-on experience with AWS (EC2, S3, RDS, Lambda, Redshift, Glue, Athena) Experience with real-time and batch data processing - Apache Flink, Kafka, Spark, or similar Proficiency in one or more of: Python, Java, Scala Experience with data modelling , ETL processes, and distributed systems Comfortable working ...

skills and attention to detail ✅ A proactive approach with the ability to own and deliver tasks Desirable: Experience with Azure Data Factory, Databricks or Apache Spark Familiarity with tools such as Fivetran Understanding of modern data architectures (e.g. medallion architecture) Exposure to CI/CD, Azure DevOps or Jira ...

engineers and support code reviews, design reviews, and engineering best practices. Your Skills: Strong hands-on experience with Databricks on AWS. Strong experience with Apache Spark/PySpark. Excellent programming skills in Python. Strong SQL skills including complex queries, joins, CTEs, window functions, and query optimization. Experience with Spark ...

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes Location: London (2 days per week onsite) Rate: £700–£800 per day (Outside IR35) Duration: Initial 6-month contract with likely extension Overview We're looking for an experienced Lead Backend Software Engineer … data services capable of securely querying and processing data across multiple environments. Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks. Design and implement Kubernetes-based services and containerised workloads. Build secure APIs and backend services ...

developing others. Desirable to have skills or experience in: - SAS Viya for data analytics. - Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). - Denodo for data virtualisation. - Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). - ALM Tooling (Jira, Confluence). ...

optimising ETL/ELT pipelines. Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures. Experience with Azure Data Factory, Azure Synapse, and Apache Spark. Knowledge of data modelling, semantic models, and enterprise reporting solutions. Experience implementing data governance, security, and quality frameworks. Strong understanding of modern data engineering ...

Core Java Spring Boot frameworks Proven experience building distributed backend systems Hands-on experience with Kafka (or similar streaming platforms) Experience with Scala and Apache Spark for data processing Integration withbackend services such as: PostgreSQL Redis S3 (or equivalent object storage) Expertise in modern engineering practices, including: Continuous delivery ...

experience with containerization (Docker, Podman) and orchestration using Kubernetes, including infrastructure provisioning and operationalization in cloud environments. Experience with workflow orchestration tools such as Apache Airflow and/or Argo Workflows. Strong experience building and maintaining REST APIs, ideally for serving ML models and AI services. Experience working with ...

production. Practical experience with cloud platforms (AWS and/or GCP) and their core data services. Hands on experience with workflow orchestration tools (Apache Airflow, dbt, Prefect, or equivalent). Strong familiarity with data warehousing and lakehouse technologies, including Snowflake. Solid understanding of data privacy regulations (GDPR, CCPA ...

production. Practical experience with cloud platforms (AWS and/or GCP) and their core data services. Hands-on experience with workflow orchestration tools (Apache Airflow, dbt, Prefect, or equivalent). Strong familiarity with data warehousing and lakehouse technologies, including Snowflake. Solid understanding of data privacy regulations (GDPR, CCPA ...

tools such as dbt, Matillion, Informatica, Fivetran, or similar. Strong Python programming skills for automation and data engineering. Experience with orchestration tools such as Apache Airflow or similar. Experience implementing CI/CD pipelines using Git, Azure DevOps, GitHub Actions, or Jenkins. Good understanding of dimensional modelling, data warehousing ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly to stakeholders ...