1 to 25 of 232 Apache Jobs

design, scalability, and clean coding practices. Familiarity with application monitoring solutions Data Engineering 5+ years building scalable and reliable data pipelines using Databricks and Apache Spark. We need both. Experience with cloud data warehouse/data lake architectures and tooling. Experience building solutions using cloud providers, preferably Azure ...

Platform & Application Testing • Test and validate distributed data engineering applications across Cloudera Manager components: Kafka, Spark, HBase, Hive, Solr, NiFi, Hue, Livy, Unravel. • Test Apache NiFi pipelines including processors, flow configurations, data lineage, provenance, NiFi registry flows, and integration with downstream systems. • Validate ingestion pipelines, streaming jobs, batch ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

governance tools (AWS Glue Catalog, Lake Formation). Knowledge of data warehouse design patterns and best practices. Experience with data orchestration tools (e.g., Apache Airflow, Step Functions). Working knowledge of Java is a plus. Education B.S. in Computer Science, MIS or related degree and a minimum of five ...

experience in cloud-based data services. Proficiency in SQL and Python for data manipulation and transformation. Experience with modern data engineering tools, including Apache Spark, Kafka, and Airflow. Strong understanding of data modelling, schema design, and data warehousing concepts. Familiarity with data governance, privacy, and compliance frameworks (e.g., GDPR ...

software development experience with Python and experience with modern software development and release engineering practices (e.g. TDD, CI/CD). Experience with Apache Spark or any other distributed data programming frameworks. Comfortable writing efficient SQL and debugging on cloud warehouses like Databricks SQL or Snowflake. Experience with cloud ...

working in a cloud environment such as Google Cloud Platform or AWS Hands on programming experience of the following (or similar) technologies: Kubernetes, Docker Apache Beam, Apache Flink, Apache Spark Google BigQuery, Snowflake Google BigTable Google Pub/Sub, Kafka Apache Airflow Bachelors’ degree with ...

native solutions that support analytics, automation, and digital transformation. Data Engineering, Full Stack & Platform Development Design and implement scalable data pipelines using tools like Apache Spark, Airflow, or dbt.Build and maintain data lakes, warehouses, and real-time streaming solutions. Develop APIs and microservices to expose data securely and efficiently. ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI ...

Charity Volunteer Days Cycle to work scheme And More.. DATA ENGINEER - ESSTENTIAL SKILLS Proven experience building data pipelines using Databricks . Strong understanding of Apache Spark (PySpark or Scala) and Structured Streaming . Experience working with Kafka (MSK) and handling real-time data . Good knowledge of Delta Lake ...

/or Azure (Data Lake, Synapse). Programming & Scripting:Proficiency in Python, SQL, PySpark etc. ETL/ELT & Streaming:Expertise in technologies like Apache Airflow, Glue, Kafka, Informatica, EventBridge etc. Industrial Data Integration:Familiarity with OT data schema originating from OSIsoft PI, SCADA, MES, and Historian systems. Information Modeling ...

performance optimisation. Proven experience with cloud platforms (AWS, Azure, or GCP) and their data services. Hands-on experience with big data frameworks (e.g. Apache Spark, Hadoop). Strong knowledge of data governance, security, and compliance. Ability to lead technical projects and mentor junior engineers. Excellent problem-solving skills ...

Java and Spring or are willing to learn on the job. Have strong familiarity with SQL Experience with the following tools and frameworks: Apache Airflow and or data pipelines. Celery FastAPI and/or Flask Postgres (AWS Aurora Ideally) DynamoDB Have experience working in the FinTech industry, particularly ...

equivalent in enterprise data architecture, data engineering, or data platform strategy. • Deep expertise in building and modernizing distributed data platforms using technologies such as Apache Spark, Kafka, Flink, NiFi, and Cloudera/Hadoop. • Hands-on proficiency with modern data platforms and tools including Databricks, Snowflake, Delta Lake, and Redshift. ...

equivalent in enterprise data architecture, data engineering, or data platform strategy. • Deep expertise in building and modernizing distributed data platforms using technologies such as Apache Spark, Kafka, Flink, NiFi, and Cloudera/Hadoop. • Hands-on proficiency with modern data platforms and tools including Databricks, Snowflake, Delta Lake, and Redshift. ...

equivalent in enterprise data architecture, data engineering, or data platform strategy. • Deep expertise in building and modernizing distributed data platforms using technologies such as Apache Spark, Kafka, Flink, NiFi, and Cloudera/Hadoop. • Hands-on proficiency with modern data platforms and tools including Databricks, Snowflake, Delta Lake, and Redshift. ...

equivalent in enterprise data architecture, data engineering, or data platform strategy. • Deep expertise in building and modernizing distributed data platforms using technologies such as Apache Spark, Kafka, Flink, NiFi, and Cloudera/Hadoop. • Hands-on proficiency with modern data platforms and tools including Databricks, Snowflake, Delta Lake, and Redshift. ...

learning solutions. With published authors and award-winning data scientists who contribute some of the major machine learning and distributed data technologies, such as Apache Spark, we are a friendly, passionate group of engineers making a career out of building great software for our customers. The Adthena data team ...

automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client projects … languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like Apache Airflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud Dataflow. Proficiency ...

have strong Python backend software engineer skills You have experience working with large data sets You have experience of using PySpark and ideally also Apache Spark You believe in automating wherever possible You're a collaborative problem solver with great communication skills Other technology in the stack includes: FastAPI ...