1 to 25 of 79 Remote Apache Spark Jobs

notified via phone or email within ten (10) business days of submittal. Essential Duties & Responsibilities Build and optimize ETL data pipelines using Databricks and Apache Spark Develop scalable data models leveraging Spark Data Frames and Spark SQL for efficient querying, aggregation, and integration against numerous diverse … Education Requirements: Bachelor's (US accredited Required) Level of Experience Requirements : Python: 4 years (Preferred) SQL: 4 years (Preferred) SharePoint Development: 3 years (Preferred) Spark: 3 years (Preferred) AWS: 3 years (Preferred) S3: 3 years (Preferred) AI: 2 years (Preferred) Databricks: 2 years (Required) License/Certification U.S. federal ...

Captivation Software is looking for a mid level software engineer who shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

Pandas, numpy, scipy, scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place … scale. Candidate must have experience: Building end-to-end data pipelines leveraging Python Using orchestration tools to deploy data pipelines, including configuring and updating Spark Jobs Containerizing and deploying applications in cloud environments like AWS. Working with MySQL and PostgreSQL including performance tuning, schema design, and query optimization ...

help other members of the team. · Experience provisioning and managing AWS services using infrastructure as code such as AWS CloudFormation or Terraform. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Knowledge of Serverless technologies, framework and best practices. · Experience provisioning … managing Apache Airflow. · Exposure to using AI tools to enhance productivity and quality. Why Join RealityMine? At RealityMine, we believe our people are at the heart of everything we do. That’s why we go the extra mile to support every team member to unlock their full potential. Whether ...

services, we strive to create groundbreaking solutions that provide real value to our clients. Key Responsibilities Design, develop, and maintain scalable data pipelines using Apache Spark (PySpark and/or Scala) Build and optimize data workflows on Databricks , including Delta Lake, notebooks, and scheduled jobs Ingest, transform … curate large-scale structured and semi-structured datasets Perform performance tuning and cost optimization of Spark workloads and Databricks clusters Implement data quality checks, monitoring, and error handling Collaborate with analytics and business stakeholders to deliver well-modeled, analytics-ready data Support batch processing and, where applicable, streaming data ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training simulation ...

actionable. Key Responsibilities: Data Lake Architecture - Design and build our core performance data lake on columnar storage foundations. Select, integrate, and optimize table formats (Apache Iceberg, Parquet, Avro) to balance query performance, storage cost, and schema evolution. Implement hot and cold storage tiering strategies that keep recent data instantly … based storage engines, or custom data pipelines. Experience running MLPerf submissions or similar large-scale audited benchmarks. Contributions to OSS projects such as Apache Iceberg, Apache Spark, Trino, llm-d, vLLM, or PyTorch. Exposure to benchmarking large GPU fleets or multi-region clusters. Experience with CUDA kernels ...

with relational and/or NoSQL databases. Hands-on experience building and optimizing ETL/ELT pipelines. Familiarity with big data technologies such as Apache Spark or Hadoop. Experience with BI tools and building dashboards (e.g., Tableau, Power BI, Looker, etc.). Strong problem-solving skills with ...

e.g., Kafka), and cloud services (e.g., AWS, GCP, Azure). Programming Skills: Any programming skills in languages like Python (NumPy, pandas, PySpark), Java (Core Spark with Java, functional interface, collections), or Scala with experience in automation and scripting would be great. Infrastructure Tools: Experience with containerization and orchestration tools ...

aiming for a career in data engineering, it can feel like youre staring at a never-ending list of tools and technologies SQL, Python, Spark, Kafka, Airflow, dbt, Snowflake, Redshift, Terraform, Kubernetes, and the list goes on. Scroll job boards and LinkedIn, and its easy to conclude that unless ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google ...

engineering/data platform development 3+ years in a lead or architect role Strong programming expertise in Python Hands-on experience with: PySpark/Spark SQL & data modeling Workflow orchestration (Airflow, Prefect) Experience in wealth management/asset management/financial services Strong understanding of: Investment data models Market ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

data estate. Deep expertise in Microsoft Azure cloud services - ADF, ADLS, Synapse, Purview. Databricks - Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large-scale distributed computing and performance tuning. Microsoft Fabric - OneLake and DirectLake integration. Azure Synapse Analytics for enterprise-scale data warehousing. Infrastructure ...

C#/C++/C Database: SQL Server, PostgreSQL, Snowflake, Redshift, Aurora, Presto, BigQuery, Oracle DevOps: git, docker, subversion, Kubernetes, Jenkins Additional Technologies: Spark, Databricks, Kafka, Kinesis, Hadoop, Lambda, EMR Popular Certifications: AWS Cloud Practitioner, Microsoft Azure Data Fundamentals, Google Associate Cloud Engineer Additional Information We want everyone ...

C#/C++/C Database: SQL Server, PostgreSQL, Snowflake, Redshift, Aurora, Presto, BigQuery, Oracle DevOps: git, docker, subversion, Kubernetes, Jenkins Additional Technologies: Spark, Databricks, Kafka, Kinesis, Hadoop, Lambda, EMR Popular Certifications: AWS Cloud Practitioner, Microsoft Azure Data Fundamentals, Google Associate Cloud Engineer Additional Information We want everyone ...

C#/C++/C Database: SQL Server, PostgreSQL, Snowflake, Redshift, Aurora, Presto, BigQuery, Oracle DevOps: git, docker, subversion, Kubernetes, Jenkins Additional Technologies: Spark, Databricks, Kafka, Kinesis, Hadoop, Lambda, EMR Popular Certifications: AWS Cloud Practitioner, Microsoft Azure Data Fundamentals, Google Associate Cloud Engineer Additional Information We want everyone ...

analytical data modeling in BigQuery using dbt . Build reliable data pipelines utilizing SQL, Python, and distributed processing frameworks (Apache Spark, Scio, Beam, or Flink). Develop clear, story-driven dashboards using tools like Looker or Tableau . Champion data quality , implement CI/CD for dbt models ...

sponsored). What Sets You Apart: Palantir Foundry Data Engineer Certification preferred. Foundry Solution Architect certification preferred. Foundry Application Developer certification preferred. Familiarity with Spark, Databricks, or similar distributed processing frameworks. Advanced degree or certifications in cloud computing, AI/ML, data engineering, or solution architecture. Experience with event ...

development effort across the full spectrum of government Modeling and Simulation environments. Experience with geospatial engineering Experience with Big Data processing tools, such as Spark/Hadoop, Google's BigQuery, AWS's Athena, and EMR Experience with public clouds such as AWS, Google, Azure Strong data visualization skills Benefits ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

ongoing improvements in reliability, scalability, and data quality What You'll Need: 5+ years of experience in data engineering with strong Python, SQL, and Spark skills Hands-on experience building and automating ETL/ELT pipelines and working with data warehouses Solid understanding of data modeling, schema design ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...