production issues. Optimize applications for performance and responsiveness. Stay Up to Date with Technology: Keep yourself and the team updated on the latest Python technologies, frameworks, and tools like Apache Spark, Databricks, Apache Pulsar, ApacheAirflow, Temporal, and Apache Flink, sharing knowledge and suggesting improvements. Documentation: Contribute to clear and concise documentation for software, processes … Experience with cloud platforms like AWS, GCP, or Azure. DevOps Tools: Familiarity with containerization (Docker) and infrastructure automation tools like Terraform or Ansible. Real-time Data Streaming: Experience with Apache Pulsar or similar systems for real-time messaging and stream processing is a plus. Data Engineering: Experience with Apache Spark, Databricks, or similar big data platforms for processing … large datasets, building data pipelines, and machine learning workflows. Workflow Orchestration: Familiarity with tools like ApacheAirflow or Temporal for managing workflows and scheduling jobs in distributed systems. Stream Processing: Experience with Apache Flink or other stream processing frameworks is a plus. Desired Skills Asynchronous Programming: Familiarity with asynchronous programming tools like Celery or asyncio. Frontend Knowledge More ❯
you'll have the opportunity to grow the data function and step into a leadership role. The Role: Designing and building robust ETL pipelines using tools like dbt or ApacheAirflow Integrating data from APIs, databases, and SaaS platforms into BigQuery Structuring clean, queryable data models to support analytics and reporting Collaborating with analysts to deliver insightful dashboards … via Looker Establishing data governance and quality processes Requirements: GCP (BigQuery), but open to other cloud backgrounds ETL: dbt, ApacheAirflow, or similar BI: Looker (preferred), or other BI tools Languages: SQL, Python, Java Experienced data engineer, with strong ETL and cloud data warehouse experience Proficiency in SQL and data modelling best practices Experience with BI tools and More ❯
you'll have the opportunity to grow the data function and step into a leadership role. The Role: Designing and building robust ETL pipelines using tools like dbt or ApacheAirflow Integrating data from APIs, databases, and SaaS platforms into BigQuery Structuring clean, queryable data models to support analytics and reporting Collaborating with analysts to deliver insightful dashboards … via Looker Establishing data governance and quality processes Requirements: GCP (BigQuery), but open to other cloud backgrounds ETL: dbt, ApacheAirflow, or similar BI: Looker (preferred), or other BI tools Languages: SQL, Python, Java Experienced data engineer, with strong ETL and cloud data warehouse experience Proficiency in SQL and data modelling best practices Experience with BI tools and More ❯
you'll have the opportunity to grow the data function and step into a leadership role. The Role: Designing and building robust ETL pipelines using tools like dbt or ApacheAirflow Integrating data from APIs, databases, and SaaS platforms into BigQuery Structuring clean, queryable data models to support analytics and reporting Collaborating with analysts to deliver insightful dashboards … via Looker Establishing data governance and quality processes Requirements: GCP (BigQuery), but open to other cloud backgrounds ETL: dbt, ApacheAirflow, or similar BI: Looker (preferred), or other BI tools Languages: SQL, Python, Java Experienced data engineer, with strong ETL and cloud data warehouse experience Proficiency in SQL and data modelling best practices Experience with BI tools and More ❯
a well maintainable system. You will also be responsible for building and maintaining robust and scalable workflow systems. We utilize the latest technologies to build our systems using Python, ApacheAirflow, Docker, SQL, GraphQL etc. We'll Trust You To: Design, build, and deploy reliable and scalable services and applications - Build and maintain high quality software. Strong communication … dives into system design and implementation A keen interest in keeping abreast of technological advances and proven success at incorporating new technology into existing systems Knowledge and experience with Apacheairflow Salary Range = 160000 - 240000 USD Annually + Benefits + Bonus The referenced salary range is based on the Company's good faith belief at the time of More ❯
external stakeholders. This is an onsite position in Crystal City, VA. Job Duties Include (but not limited to): Design, develop, and implement scalable data pipelines and ETL processes using ApacheAirflow, with a focus on data for AI applications. Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures. Build and maintain high … with mission objectives. Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions. Optimize data storage, retrieval, and processing efficiency. Qualifications Required Skills & Experience: Experience with ApacheAirflow for workflow orchestration. Strong programming skills in Python. Experience with ElasticSearch/OpenSearch for data indexing and search functionalities. Understanding of vector databases, embedding models, and vector … real-time data processing. Understanding of LLM prompt engineering and associated ETL applications. Knowledge of SuperSet for data visualization and analytics. Familiarity with Kubernetes for container orchestration. Exposure to Apache Spark for large-scale data processing. Education & Certifications: Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). Advanced degrees are a More ❯
data architectures, such as lakehouse. Experience with CI/CD pipelines, version control systems like Git, and containerization (e.g., Docker). Experience with ETL tools and technologies such as ApacheAirflow, Informatica, or Talend. Strong understanding of data governance and best practices in data management. Experience with cloud platforms and services such as AWS, Azure, or GCP for … deploying and managing data solutions. Strong problem-solving and analytical skills with the ability to diagnose and resolve complex data-related issues. SQL (for database management and querying) Apache Spark (for distributed data processing) Apache Spark Streaming, Kafka or similar (for real-time data streaming) Experience using data tools in at least one cloud service - AWS, Azure or More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
aligns with Engineering Test & Technology (ET&T) software objectives and integrates seamlessly with the BGS Government solutions. Technology Solution Identification: Assess technology solutions for data integration (ETL capabilities e.g. ApacheAirflow), data storage (PostgreSQL, Neo4j, Amazon S3), big data processing (Apache Spark), data analytics & machine learning (TensorFlow, PyTorch, etc.), data visualization (Apache Superset), containerization and orchestration More ❯
systems, with a focus on data quality and reliability. Design and manage data storage solutions, including databases, warehouses, and lakes. Leverage cloud-native services and distributed processing tools (e.g., Apache Flink, AWS Batch) to support large-scale data workloads. Operations & Tooling Monitor, troubleshoot, and optimize data pipelines to ensure performance and cost efficiency. Implement data governance, access controls, and … ELT pipelines and data architectures. Hands-on expertise with cloud platforms (e.g., AWS) and cloud-native data services. Comfortable with big data tools and distributed processing frameworks such as Apache Flink or AWS Batch. Strong understanding of data governance, security, and best practices for data quality. Effective communicator with the ability to work across technical and non-technical teams. … Additional Strengths Experience with orchestration tools like Apache Airflow. Knowledge of real-time data processing and event-driven architectures. Familiarity with observability tools and anomaly detection for production systems. Exposure to data visualization platforms such as Tableau or Looker. Relevant cloud or data engineering certifications. What we offer: A collaborative and transparent company culture founded on Integrity, Innovation and More ❯
for the first time, Dev10 equips you with the tools to succeed. You'll train in technologies including Relational and Non-Relational Databases, APIs, Python, Pandas, Excel, Dash, Kafka, Airflow, Apache Spark, and Machine Learning. Now hiring for our Data Engineer cohort starting Monday, December 1, 2025. To apply, please submit a formal application through our website: Dev10 More ❯
for the first time, Dev10 equips you with the tools to succeed. You'll train in technologies including Relational and Non-Relational Databases, APIs, Python, Pandas, Excel, Dash, Kafka, Airflow, Apache Spark, and Machine Learning. Now hiring for our Data Engineer cohort starting Monday, December 1, 2025. To apply, please submit a formal application through our website: Dev10 More ❯
for the first time, Dev10 equips you with the tools to succeed. You'll train in technologies including Relational and Non-Relational Databases, APIs, Python, Pandas, Excel, Dash, Kafka, Airflow, Apache Spark, and Machine Learning. Now hiring for our Data Engineer cohort starting Monday, December 1, 2025. To apply, please submit a formal application through our website: Dev10 More ❯
data cataloging and metadata management using tools like AWS Glue Data Catalog. Demonstrated self-sufficiency in exploring new tools, troubleshooting issues, and continuously improving processes. Hands-on experience with ApacheAirflow for orchestrating complex data workflows and ensuring reliable execution. Understanding of cloud security and governance practices including IAM, KMS, and data access policies. Experience with monitoring and More ❯
data cataloging and metadata management using tools like AWS Glue Data Catalog. Demonstrated self-sufficiency in exploring new tools, troubleshooting issues, and continuously improving processes. Hands-on experience with ApacheAirflow for orchestrating complex data workflows and ensuring reliable execution. Understanding of cloud security and governance practices including IAM, KMS, and data access policies. Experience with monitoring and More ❯
data cataloging and metadata management using tools like AWS Glue Data Catalog. Demonstrated self-sufficiency in exploring new tools, troubleshooting issues, and continuously improving processes. Hands-on experience with ApacheAirflow for orchestrating complex data workflows and ensuring reliable execution. Understanding of cloud security and governance practices including IAM, KMS, and data access policies. Experience with monitoring and More ❯
data cataloging and metadata management using tools like AWS Glue Data Catalog. Demonstrated self-sufficiency in exploring new tools, troubleshooting issues, and continuously improving processes. Hands-on experience with ApacheAirflow for orchestrating complex data workflows and ensuring reliable execution. Understanding of cloud security and governance practices including IAM, KMS, and data access policies. Experience with monitoring and More ❯
