1 to 25 of 139 Remote Apache Spark Jobs

in a hybrid role-60% administration and 40% development/support-to help us scale our data and DataOps infrastructure. You'll work with cutting-edge technologies like Databricks, Apache Spark, Delta Lake, and AWS CloudOps, Cloud Security, while supporting mission-critical data pipelines and integrations. If you're a hands-on engineer with strong Python skills, deep … knack for solving complex data challenges, we want to hear from you. Key Responsibilities Design, develop, and maintain scalable ETL pipelines and integration frameworks. Administer and optimize Databricks and Apache Spark environments for data engineering workloads. Build and manage data workflows using AWS services such as Lambda, Glue, Redshift, SageMaker, and S3. Support and troubleshoot DataOps pipelines, ensuring … years of experience in integration framework development with a strong emphasis on Databricks, AWS, and ETL. Required Technical Skills Strong programming skills in Python and PySpark. Expertise in Databricks, Apache Spark, and Delta Lake. Proficiency in AWS CloudOps, Cloud Security, including configuration, deployment, and monitoring. Strong SQL skills and hands-on experience with Amazon Redshift. Experience with ETL More ❯

Be Doing You'll be a key contributor to the development of a next-generation data platform, with responsibilities including: Designing and implementing scalable data pipelines using Python and Apache Spark Building and orchestrating workflows using AWS services such as Glue , Lambda , S3 , and EMR Serverless Applying best practices in software engineering: CI/CD , version control , automated … testing , and modular design Supporting the development of a lakehouse architecture using Apache Iceberg Collaborating with product and business teams to deliver data-driven solutions Embedding observability and quality checks into data workflows Participating in code reviews, pair programming, and architectural discussions Gaining domain knowledge in financial data and sharing insights with the team What They're Looking For … for experience with type hints, linters, and testing frameworks like pytest) Solid understanding of data engineering fundamentals: ETL/ELT, schema evolution, batch processing Experience or strong interest in Apache Spark for distributed data processing Familiarity with AWS data tools (e.g., S3, Glue, Lambda, EMR) Strong communication skills and a collaborative mindset Comfortable working in Agile environments and More ❯

experience in a leadership or technical lead role, with official line management responsibility. Strong experience with modern data stack technologies, including Python, Snowflake, AWS (S3, EC2, Terraform), Airflow, dbt, Apache Spark, Apache Iceberg, and Postgres. Skilled in balancing technical excellence with business priorities in a fast-paced environment. Strong communication and stakeholder management skills, able to translate More ❯

contract assignment. Key Requirements: Proven background in AI and data development Strong proficiency in Python , including data-focused libraries such as Pandas, NumPy, and PySpark Hands-on experience with Apache Spark (PySpark preferred) Solid understanding of data management and processing pipelines Experience in algorithm development and graph data structures is advantageous Active SC Clearance is mandatory Role Overview … You will play a key role in developing and delivering advanced AI solutions for a Government client . Responsibilities include: Designing, building, and maintaining data processing pipelines using Apache Spark Implementing ETL/ELT workflows for large-scale data sets Developing and optimising Python-based data ingestion tools Collaborating on the design and deployment of machine learning models … performance across distributed systems Contributing to data architecture and storage strategy design Working with cloud data platforms (AWS, Azure, or GCP) to deploy scalable solutions Monitoring, troubleshooting, and tuning Spark jobs for performance and cost efficiency Engaging regularly with customers and internal stakeholders This is an excellent opportunity to join a high-profile organisation on a long-term contract More ❯

rather huge and includes Python (Pandas, numpy, scipy, scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place in Chantilly, VA, McLean, VA and …/standards. Develop and deliver documentation for each project including ETL mappings, code use guide, code location and access instructions. Design and optimize Data Pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi and Kubernetes containers Ensure the pedigree and provenance of the data is maintained such that the access to data is … years' experience with: Data lifecycle engineering Development and maintenance of extract, transform and load (ETL) tools and services Cloud and on-prem data storage and processing solutions Python, SQL, Spark and other data engineering programming COTS and open source data engineering tools such as ElasticSearch and NiFi Processing data within the Agile Lifecycle Desired Skills (Optional) Experience using AI More ❯

rather huge and includes Python (Pandas, numpy, scipy, scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place in Chantilly, VA, McLean, VA and …/standards. Develop and deliver documentation for each project including ETL mappings, code use guide, code location and access instructions. Design and optimize Data Pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi and Kubernetes containers Ensure the pedigree and provenance of the data is maintained such that the access to data is … years' experience with: Data lifecycle engineering Development and maintenance of extract, transform and load (ETL) tools and services Cloud and on-prem data storage and processing solutions Python, SQL, Spark and other data engineering programming COTS and open source data engineering tools such as ElasticSearch and NiFi Processing data within the Agile Lifecycle Desired Skills (Optional) Experience using AI More ❯

engineering, data security practices, data platforms, and analytics 3+ years Databricks Platform Expertise - SME Level Proficiency including: Databricks, Delta Lake, and Delta Sharing Deep experience with distributed computing using Apache Spark Knowledge of Spark runtime internals and optimization Ability to design and deploy performant end-to-end data architectures 4+ years of ETL Pipeline Development building robust … U.S. for three (3) full years out of the last five (5) years Technologies you'll use: Databricks on Azure for data engineering and ML pipeline support. SQL, Python, Spark, Tableau. Git, Jira, CI/CD tools (e.g., Jenkins, CodeBuild). Jira, Confluence, SharePoint. Mural, Miro, or other collaboration/whiteboarding tools. What we'd like you to have … or similar. Machine Learning and Analytical Skills including: MLOps - Working knowledge of ML deployment and operations Data Science Methodologies - Statistical analysis, modeling, and interpretation Big Data Technologies - Experience beyond Spark with distributed systems Experience with deployment pipelines, including Git-based version control and CI/CD pipelines and DevOps practices using Terraform for IaC. Emergency management domain knowledge a More ❯

software engineering (backend, API, or full-stack) Proficient in Python, Java, or C# Experienced with REST APIs (FastAPI, AWS Lambda) and OpenAPI specifications Skilled in data pipeline orchestration (dbt, Apache Airflow, Apache Spark, Iceberg) Knowledgeable in federal compliance frameworks (NIST 800-53, HIPAA, FISMA High) Preferred/Bonus: Prior work with DHA, VA, or federal healthcare IT More ❯

experience or Masters degree and 3 years of professional experience Programming knowledge in some of the following languages: Python, SQL, Java. Strong experience with workflow orchestration tools such as Apache Airflow or Prefect. Database experience with some of the following products: Elasticsearch, PostgreSQL, MySQL, Oracle, MongoDB Experience working with Docker, Kubernetes Proficient with git and pull request workflows. Experience More ❯

data solutions Design and implement data models and pipelines for relational, dimensional, data lakehouse, data warehouse, and data mart environments Utilize Azure services including Azure Data Factory, Synapse Pipelines, Apache Spark Notebooks, Python, and SQL to build and optimize pipelines Redevelop existing SSIS ETL processes using Azure Data Factory and Synapse Pipelines Prepare and manage data for analytics More ❯

governance best practices. Demonstrated experience showing strong critical thinking and problem solving skills paired with a desire to take initiative Experience working with big data processing frameworks such as Apache Spark, and streaming platforms like Kafka or AWS Kinesis. AWS certification (Data Analytics, Developer, or Solutions Architect) is a plus. Experience with event-driven architectures and real-time More ❯

Experience with tools like AWS (S3, Glue, Redshift, SageMaker) or other cloud platforms. Familiarity with Docker, Terraform, GitHub Actions, and Vault for managing secrets. Experience in coding SQL, Python, Spark, or Scala to work with data. Experience with databases used in Data Warehousing, Data Lakes, and Lakehouse setups. You know how to work with both structured and unstructured data. More ❯

Experience with tools like AWS (S3, Glue, Redshift, SageMaker) or other cloud platforms. Familiarity with Docker, Terraform, GitHub Actions, and Vault for managing secrets. Experience in coding SQL, Python, Spark, or Scala to work with data. Experience with databases used in Data Warehousing, Data Lakes, and Lakehouse setups. You know how to work with both structured and unstructured data. More ❯

Experience with tools like AWS (S3, Glue, Redshift, SageMaker) or other cloud platforms. Familiarity with Docker, Terraform, GitHub Actions, and Vault for managing secrets. Experience in coding SQL, Python, Spark, or Scala to work with data. Experience with databases used in Data Warehousing, Data Lakes, and Lakehouse setups. You know how to work with both structured and unstructured data. More ❯

plus work experience BS & 5+ years of work experience MS & 4+ years of work experience Proficient with machine learning and statistics Proficient with Python, deep learning frameworks, Computer Vision, Spark Have produced production level algorithms Proficient in researching, developing, synthesizing new algorithms and techniques Excellent communication skills Desirable Qualifications Master's or PhD level degree 5+ years of work More ❯

on experience with cloud data platforms such as Snowflake, Redshift, Athena, or BigQuery, including optimization techniques and custom parsers/transpilers. Practical knowledge of distributed and analytical engines (e.g., Apache Spark, Trino, PostgreSQL, DuckDB) with skills in query engines, performance tuning, and integration in local and production environments. Experience building developer tooling such as CLI tools, SDKs, and More ❯

platform. Candidate Profile: Proven experience as a Data Engineer, with strong expertise in designing and managing large-scale data systems. Hands-on proficiency with modern data technologies such as Spark, Kafka, Airflow, or dbt. Strong SQL skills and experience with cloud platforms (Azure preferred). Solid programming background in Python, Scala, or Java. Knowledge of data warehousing solutions (e.g. More ❯

engineering, mathematics, and physics. 5+ years of professional work experience. Programming knowledge in some of the following languages: Python, SQL, Java. Strong experience with workflow orchestration tools such as Apache Airflow or Prefect. Database experience with some of the following products: Elasticsearch, PostgreSQL, MySQL, Oracle, MongoDB Experience working with Docker, Kubernetes Proficient with git and pull request workflows. Experience More ❯

/Starburst Enterprise/Galaxy administration and CLI operations Container Orchestration : Proven track record with Kubernetes/OpenShift in production environments Big Data Ecosystem : Strong background in Hadoop, Hive, Spark, and cloud platforms (AWS/Azure/GCP) Systems Architecture : Understanding of distributed systems, high availability, and fault-tolerant design Security Protocols : Experience with LDAP, Active Directory, OAuth2, and More ❯

/ELT pipelines , data modeling , and data warehousing . Experience with cloud platforms (AWS, Azure, GCP) and tools like Snowflake, Databricks, or BigQuery . Familiarity with streaming technologies (Kafka, Spark Streaming, Flink) is a plus. Tools & Frameworks: Airflow, dbt, Prefect, CI/CD pipelines, Terraform. Mindset: Curious, data-obsessed, and driven to create meaningful business impact. Soft Skills: Excellent More ❯

related ML libraries Experience with Docker, Kubernetes, and cloud AI platforms such as AWS Bedrock, Sagemaker, Azure ML, or GCP Vertex AI Working knowledge of data tools such as Spark, Pandas, SQL/NoSQL databases Security: TS clearance required Nice to Have Experience with LangChain, hybrid retrieval orchestration frameworks, or custom AI agent architectures Experience implementing custom authentication/ More ❯

Science, Engineering, related field, or equivalent experience - 3+ years of experience with data warehouse architecture, ETL/ELT tools, data engineering, and large-scale data manipulation using technologies like Spark, EMR, Hive, Kafka, and Redshift - Experience with relational databases, SQL, and performance tuning, as well as software engineering best practices for the development lifecycle, including coding standards, reviews, source More ❯

AGILE software development methodologies and use of standard software development tool suites Desired Technical Skills Security+ certification is highly desired. Experience with big data technologies like: Hadoop, Accumulo, Ceph, Spark, NiFi, Kafka, PostgreSQL, ElasticSearch, Hive, Drill, Impala, Trino, Presto, etc. Experience with containers, EKS, Diode, CI/CD, and Terraform are a plus. Work could possibly require some on More ❯

Excellent problem-solving skills and ability to work independently in a fast-paced environment. Desirable: Experience with NLP, computer vision, or time-series forecasting. Familiarity with distributed computing frameworks (Spark, Ray). Experience with MLOps and model governance practices. Previous contract experience in a similar ML engineering role. Contract Details Duration: 6–12 months (extension possible) Location: London (Hybrid More ❯

Excellent problem-solving skills and ability to work independently in a fast-paced environment. Desirable: Experience with NLP, computer vision, or time-series forecasting. Familiarity with distributed computing frameworks (Spark, Ray). Experience with MLOps and model governance practices. Previous contract experience in a similar ML engineering role. Contract Details Duration: 6–12 months (extension possible) Location: London (Hybrid More ❯