51 to 75 of 175 Remote Apache Spark Jobs

data pipeline orchestration tools* Containerisation (Docker, EKS, AKS, Kubernetes)* Proficiency in CI/CD principles and tools* Familiarity with open-source data tools (e.g., Spark, Kafka, PostgreSQL)* Infrastructure as Code (Terraform, Ansible)* Air-gapped development* Automation* Client/stakeholder management* Public sector domain understanding* Technical architecture**Experience:*** You will ...

concepts and how they can be applied to improve quality processes and drive continuous improvement. Exposure to data engineering and analytics tools (e.g., Databricks, Spark, Airflow or dbt). Awareness of cloud platforms (AWS, Azure or GCP). Experience with Git, CI/CD or Agile practices (from university ...

including lakehouses, data warehousing, ETL/ELT pipelines, orchestration, and dimensional data modelling. Hands‐on experience with modern data technologies such as dbt, Databricks, Spark, SQL, and Python. Understanding of semantic layers, data governance, and the foundations required for trustworthy AI‐powered analytics. Strong technical foundation, with experience designing ...

Strong hands-on experience with Azure Cosmos DB, including SQL API and Mongo API. Excellent knowledge of Cosmos DB Change Data Feed (CDF) and Spark integration. Strong PySpark and Apache Spark distributed computing experience. Experience with Cosmos DB SDKs, including .NET, Java, Python or Node.js. Expertise ...

Strong hands-on experience with Azure Cosmos DB, including SQL API and Mongo API. Excellent knowledge of Cosmos DB Change Data Feed (CDF) and Spark integration. Strong PySpark and Apache Spark distributed computing experience. Experience with Cosmos DB SDKs, including .NET, Java, Python or Node.js. Expertise ...

intellectually rewarding, involving large datasets, performance-critical pipelines, and precision engineeringWe value strong individual contributors — this is a hands-on role where your Scala Spark expertise will directly shape the platform's futureWe are a team that is enthusiastic about what we are building — there is a genuine drive … have the opportunity to contribute to architectural decisions, not just execute themRole Overview/What will you do:Build and optimise large-scale Scala Spark data pipelines on Databricks, focusing on the reconciliation domain — writing clean, modular, and production-ready codeDrive modernisation efforts by refactoring legacy Hadoop-based pipelines ...

BigQuery, Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing data warehouse and data lake architectures, including data modeling. Working experience with Terraform and CI/CD tooling such ...

gateways for AI applications Infrastructure as Code experience with Terraform, Ansible, or other IaC tools Experience building and managing data pipelines using tools like Apache Spark AI safety and governance experience with model evaluation, LLM judges, bias detection, and responsible AI practices Familiarity with statistical fundamentals and running ...

supporting or delivering migrations from Azure Synapse to Microsoft Fabric Advanced SQL: Strong SQL and T-SQL development, optimisation and troubleshooting skills. Python and Spark: Experience building data processing and transformation solutions using Python and Spark. Data engineering fundamentals: Strong understanding of ETL/ELT, data warehousing, dimensional modelling ...

Lambda, Fargate Exposure to modern API development (e.g. REST, gRPC, GraphQL) Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience developing distributed, micro services-based applications Experience with data modelling and curation for large-scale datasets Experience with near real time ...

culture where people feel safe to challenge ideas, learn, and grow. Nice to have Hands-on familiarity with the modern data stack — for example, Spark, Flink, Kafka, Debezium, Airflow, dbt, or similar — and with cloud platforms (AWS/GCP or equivalent) and modern DevOps practices. Experience leading teams that ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS. The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled. As platform landscape evolved towards cloud-native, AI enabled ...

schemas for both structured and unstructured data with a snowflake environment Enhance cloud data capabilities, working with AWS S3, Azure Data Lake, and Apache Spark Collaborate with cross-functional teams to ensure data availability for advanced analytics and reporting Automate workflows, implement CI/CD pipelines, and maintain ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

Llama IndexFine tuning of ModelsKDB knowledgeIdeally, some familiarity with data workflow management tools such as Airflow as well as big data technologies such as Apache Spark/Ignite or other caching and analytics technologies.A working knowledge of FX markets and financial instruments would be beneficial.undefinedWhat we’ll provide ...

agent frameworks. Inference and edge: TensorRT, ONNX Runtime, vLLM, Nvidia Triton Inference Server, KServe, Nvidia Jetson (e.g. Jetson Thor). Data and platforms: Databricks, Apache Spark, MLflow, SQL, AWS, GCP, Docker. Working at Rowden We are committed to building a flexible, inclusive, and enabling company. ...

Experience integrating multiple business systems, applications and third-party data sources using cloud-based technologies. Desirable experience with Azure Data Factory, Microsoft Purview, Python, Spark, DataOps, DevOps, AI platforms or large-scale cloud modernisation projects. ...

grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience is a strong plus Databases & Data Warehousing Comfortable working with: RDBMS data warehousing solutions Strong understanding of data modelling aligned with business ...

Proficiency in programming languages such as Python or R Solid experience with data visualisation tools - preferably Power BI Familiarity with big data technologies (e.g. Spark, Hadoop) and cloud platforms (e.g. Azure, AWS) Understanding of data engineering principles and ETL processes Analytical thinking and problem-solving mindset Strong attention ...

Proficiency in programming languages such as Python or R Solid experience with data visualisation tools - preferably Power BI Familiarity with big data technologies (e.g. Spark, Hadoop) and cloud platforms (e.g. Azure, AWS) Understanding of data engineering principles and ETL processes Analytical thinking and problem-solving mindset Strong attention ...

reliable data processing in a structured migration environment. Hands‐on experience building and operating data pipelines on Azure, AWS or Google Cloud. Experience with Spark, PySpark, Databricks or comparable distributed processing technologies. Experience building modular, tested and documented transformations using dbt or an equivalent approach. Strong SQL and Python ...

code. LLMOps & AI Model Management: Familiarity with tools like MLflow, LangChain, Hugging Face, Kubeflow, or similar platforms. Data Processing: Working knowledge of Databricks/Spark or comparable large-scale data processing tools. SQL: Solid capabilities in data querying and preparation. Nice to have: Time-series forecasting or optimisation, ideally ...

Snowflake, Databricks). Experience with data integration (ETL/ELT, APIs). Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) a plus. Domain Knowledge: Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations. In-depth knowledge of regulatory ...