1 to 25 of 56 Remote Apache Spark Jobs

help other members of the team. · Experience provisioning and managing AWS services using infrastructure as code such as AWS CloudFormation or Terraform. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Knowledge of Serverless technologies, framework and best practices. · Experience provisioning … managing Apache Airflow. · Exposure to using AI tools to enhance productivity and quality. Why Join RealityMine? At RealityMine, we believe our people are at the heart of everything we do. That’s why we go the extra mile to support every team member to unlock their full potential. Whether ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training simulation ...

aiming for a career in data engineering, it can feel like youre staring at a never-ending list of tools and technologies SQL, Python, Spark, Kafka, Airflow, dbt, Snowflake, Redshift, Terraform, Kubernetes, and the list goes on. Scroll job boards and LinkedIn, and its easy to conclude that unless ...

data estate. Deep expertise in Microsoft Azure cloud services - ADF, ADLS, Synapse, Purview. Databricks - Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large-scale distributed computing and performance tuning. Microsoft Fabric - OneLake and DirectLake integration. Azure Synapse Analytics for enterprise-scale data warehousing. Infrastructure ...

maintaining ETL/ELT data pipelines Proficiency in Python or Java, and strong SQL skills Experience with data pipeline frameworks (e.g. Airflow, dbt, Spark) Hands-on experience with AWS ecosystem Strong understanding of CI/CD and software engineering best practices Experience working in Data Engineering or DataOps roles ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

Production Hands-On Experience with Deep Learning Frameworks (TensorFlow, PyTorch, XGBoost) Deep Knowledge of Data Warehousing, ETL/ELT Architectures & Heavy Analytic Environments (Apache Beam, Hadoop, Spark, Hive) Responsibilities: Leading Customer-Facing AI Deployments across Europe, from the Initial Consulting Phase right through to the Live Implementation … Production ML Systems Architecting & Optimising Scalable Data Pipelines, ETL/ELT Solutions & Distributed ML Environments (using Hadoop, Spark & Apache Beam). Deploying Recommendation Engines & Deep Learning Models (TensorFlow, PyTorch) to solve specific Client Problems. Running Technical Design Sessions, Tailoring ML Ops Architectures using Vertex AI & Gen AI. Writing ...

data estate. Deep expertise in Microsoft Azure cloud services - ADF, ADLS, Synapse, Purview. Databricks - Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large-scale distributed computing and performance tuning. Microsoft Fabric - OneLake and DirectLake integration. Azure Synapse Analytics for enterprise-scale data warehousing. Infrastructure ...

/CD platform (Github Actions & Admin), Password/Secret management (hashicorp vault). Strong Data related programming skills SQL/Python/Spark/Scala. Database technologies in relation to Data Warehousing/Data Lake/Lake housing patterns and relevant experience when handling structured and non-structured data ...

analytical data modeling in BigQuery using dbt . Build reliable data pipelines utilizing SQL, Python, and distributed processing frameworks (Apache Spark, Scio, Beam, or Flink). Develop clear, story-driven dashboards using tools like Looker or Tableau . Champion data quality , implement CI/CD for dbt models ...

experience building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

validation (e.g., schema tests, data quality checks) Manage data documentation and lineage within DBT Design and develop scalable data pipelines using Databricks and Apache Spark Build and optimize data workflows for batch and real-time processing Transform and process large datasets using PySpark, SQL, and Delta Lake Implement … terraform enterprise , Snowflake terraform provider knowledge. Snowflake : Snow sight , Python SDK for snowflake , Stage , storage integration , Data marketplace , direct shares , listings , catalog linked database , Apache Iceberg , external tables , Catalog integrations , Snowpipe , external volume , Cortex ai ,privatelink. Airflow : Dag creation , operators , connections , deployment on Kubernetes , Astronomer. We are a Disability ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

functions Highly proficient coding skills in SQL, Python & PySpark Proven knowledge in at least one of the following big data technologies – HIVE, Hadoop, MongoDB, Spark Experience within the Life Science, Medical Research or Pharmaceutical sector highly advantageous Analytically minded with the ability to solve problems creatively and efficiently Vast ...

Solid understanding of data governance, data lineage, security practices, and regulatory compliance frameworks. Experience working with large-scale data storage and processing technologies (SQL, Spark, NoSQL, ETL/ELT tools). Excellent communication and stakeholder management skills, with the proven ability to translate complex data concepts into actionable, business ...

modern data engineering best practices You have experience with Databricks, ideally also Azure You have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and Pandas You're excited to join a start-up in a role that you can shape and influence ...

experience with data science libraries and frameworks. Expertise in statistical modelling, machine learning, and predictive analytics. Familiarity with big data technologies (eg, Hadoop, Spark) and cloud platforms (eg, AWS, Azure). Experience working with large, complex datasets and a track record of delivering impactful data science projects ...

frameworks. Strong expertise in statistical modelling, predictive analytics, and machine learning. In-depth understanding of data architecture and big data technologies (eg, Hadoop, Spark, AWS). Exceptional problem-solving skills and ability to think strategically. Outstanding communication abilities to convey technical concepts to non-technical stakeholders. Experience in managing ...