51 to 75 of 335 Apache Jobs

project stakeholders; Strong problem-solving skills; Ability to refine problem statements and propose solutions considering effort, impact, and scalability trade-offs; Nice to have: Apache Spark, Iceberg, Kafka, dbt, Scikit-Learn, XGBoost, PyTorch, MLFlow, GraphFrames, Ray, AWS S3, EMR, SageMaker, Lakeformation, Terraform, Docker, GitHub CI/CD, Knowledge Graphs ...

scraping pipelines to ensure data accuracy and reliability.Required Technical SkillsStrong proficiency in Python.Experience with monitoring and observability tools.Hands-on experience with workflow orchestration tools - Apache Airflow.Familiarity with data pipelines and ETL processes.Proficiency with containerization and orchestration technologies - Docker, KubernetesStrong understanding of CI/CD pipelines and toolsStrong problem-solving ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

pipelines.Excellent knowledge of PostgreSQL (or similar relational databases) and NoSQL databases.Experience with vector databases such as Qdrant, Milvus or pgvector.Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars.Strong understanding of data modelling, storage architecture and performance optimisation.DesirableExperience working on AI or Machine Learning ...

build out technology roadmapsCodebase ownershipTechnical Application ownershipQualificationsStrong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , HarnessStrong experience of developing on Apache Spark/HadoopStrong experience with AWS cloud services including AWS Glue, S3, StepStrong experience building data pipelines and data lakeExperience of the full development ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

with the ability to work fluently across scientific and technical domains.Essential skills and experience:Proven data engineering experience with Databricks (Delta Lake, Unity Catalog, Apache Spark) in cloud-based data platforms and environments (AWS).Strong programming skills in Python and SQL, with experience in software version control and deployment ...

Ideally will be a bonus if you have the following... Experience or knowledge of FluxCD, GitOps workflow, Appgate, MongoDB, Airflow (MWAA), Snowflake Experience in Apache Airflow (MWAA), AWS Backup and S3 versioning for business continuity. Programming experience in C#, Python and/or SQL Experience in Keycloak or other ...

time streaming infrastructureOwn the architecture and roadmap for high-volume low-frequency data systems, with focus on data stack like Spark, Kafka, Iceberg, RisingWave, Apache FlinkDesign and operate scalable data architecture that serve trading, risk, compliance, analytics and many product teams.Drive adoption of AI automation and intelligent workflows — automating ...

digital landscape. This role requires strong expertise in building and managing data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. The successful candidate will design, implement, and maintain scalable, secure data solutions, ensuring compliance with strict security standards and regulations.This is a UK based onsite role … will do as a Senior Data Engineer:Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi.Implement data ingestion, transformation, and integration processes, ensuring data quality and security.Collaborate with data architects and security teams to ensure compliance with security policies ...

reviews and continuous improvement Skills & Experience * Commercial experience in Data Engineering * Strong Python or Java development skills * Experience with AWS cloud services * Knowledge of Apache Spark, Hadoop, NiFi or similar big data technologies * Experience building ETL/ELT pipelines and processing large datasets * Understanding of software engineering best practices ...

innovation-led working environment Key Experience Proven experience as a Data Architect, Databricks Architect, Lead Data Engineer or Solutions Architect Strong expertise in Databricks, Apache Spark, PySpark and SQL Experience designing and implementing modern data lake or lakehouse architectures Hands-on experience with Kafka Strong understanding of AWS data ...

production, providing subject matter expertise on the .Net stack and contributing to technical design discussions. You'll use a range of technology including Apache Flink with Java for large scale data processing and will be able to assess and recommend new and emerging technologies, using the best tool ...

Frequency) will be nice. Knowledge of Development Practices/Pipeline/Methodologies will be needed. Nice to Have Knowledge of workflow toolkits such as Apache Airflow, Dagster, or similar. Active use of modern LLMs (Claude, GitHub Copilot) to accelerate development and pioneer AI-assisted coding practices within the team. ...

Codebase ownership Technical Application ownership Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins, Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

with vector databases utilised in modern AI applications.Strong Python development skills.Data & Infrastructure:Experience with some of the following technologies is highly desirable:Frameworks & Infrastructure:Apache SparkApache AirflowKafkaElasticsearch/OpenSearchDatabases:PostgreSQLMongoDBVector databases such as Qdrant, Milvus, or pgvectorPython Ecosystem:PandasPolarsEngineering Skills:Experience building highly scalable backend systems.Strong understanding of data ...

stakeholders, and business leaders.Strong presentation, demo, and storytelling skills, with the ability to explain complex technical concepts clearly and credibly.Familiarity with SQL, Python, R, Apache Spark, or similar technologies is a plus.Experience with Alteryx or similar analytics platforms is highly desirable.Ability to work independently in a fast-paced environment ...

preferred knowledge of AI infrastructure. Proficiency in programming languages such as Python, Java, or C++. Experience with data pipeline and workflow management tools (e.g., Apache Airflow, Kubeflow). Strong problem-solving skills and ability to work in a fast-paced environment. Excellent communication and collaboration skills. Proven experience ...

concepts, experience with LLMs, prompt engineering, or integrating AI capabilities into applications• Data Technologies – Experience with Kafka, PySpark, Parquet and Avro data formats, Apache Airflow for workflow management, or Streamlit for building interactive data applications• Advanced Data Management – Strong SQL design, query optimization, and database architecture expertise• Observability – Experience ...

Github integration and automation). Experience with scripting languages such as Python or R. Working knowledge of message queuing and stream processing. Experience with Apache Spark or Similar Technologies. Experience with Agile and Scrum Technologies. Familiarity with dbt and Airflow is an advantage. Experience working in a start ...

complex pipelines and data models in a cloud environment Strong proficiency in SQL; solid Python and/or PySpark capabilityDemonstrable experience with Databricks, Apache Spark, and Delta Lake architectureProven ability to design and own enterprise-grade canonical data models across multiple upstream source systemsExperience integrating data from APIs ...

SnowflakeDeep hands-on experience with Spark/PySpark and other big data processing technologiesExpertise in open-source table formats and catalog services such as Apache IcebergExperience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI)Familiarity with AI evaluation and observability practices: evals ...

Understanding of Azure data services and API-based integration. Next.js 14, FastAPI Python, Pipeline orchestration, Data transformation, Data quality, Casbin RBAC, Kuzu knowledge graph, Apache Iceberg, MCP AI server integration and ODCS v2 data contracts layer. Experience in establishing the infrastructure, pipelines, processes, and governance required for production environment ...

consultancy environments Senior Data Engineering Consultant | DESIRED SKILLS Experience working across multi‐cloud platforms including Azure, AWS, or hybrid data environments Exposure to Databricks, Apache Spark, Azure Synapse, or similar distributed processing technologies Familiarity with systems integration and API development including REST‐based services Experience supporting pre‐sales activities ...