diverse sources, transform it into usable formats, and load it into data warehouses, data lakes or lakehouses. Big Data Technologies: Utilize big data technologies such as Spark, Kafka, and Flink for distributed data processing and analytics. Cloud Platforms: Deploy and manage data solutions on cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP), leveraging cloud-native services More ❯
for data transformation and performance optimization. Hands-on experience with data pipeline tools (Airflow, dbt, Kafka, or equivalent). Strong knowledge of big data processing frameworks (Apache Spark, Databricks, Flink, etc.). Cloud & Infrastructure: Experience with cloud computing platforms (AWS, Azure, Google Cloud). Familiarity with modern data architectures (data lakes, lakehouses, warehouses). Exposure to containerization and orchestration More ❯
designing and supporting multi-tenant SaaS data platforms with strategies for data partitioning, tenant isolation, and cost management Exposure to real-time data processing technologies such as Kafka, Kinesis, Flink, or Spark Streaming, alongside batch processing capabilities Strong knowledge of SaaS compliance practices and security frameworks Core Competencies Excellent problem-solving abilities with the capacity to translate requirements into More ❯
DAST pipeline integration, STIG automation (Ansible), SBOM generation (CycloneDX/Syft/Grype), and supply-chain security (SLSA/SCVS). Real-time/streaming analytics (Kafka Streams/Flink/Spark), geospatial, or time-series processing for mission applications. GDIT IS YOUR PLACE At GDIT, the mission is our purpose, and our people are at the center of More ❯
Familiarity with MLOps, model serving, and AI workload optimization. Knowledge of A/B testing frameworks and feature flag systems. Experience with real-time data platforms (e.g., Apache Kafka, Flink). Background in microservices and distributed systems architecture. Open-source contributions. Understanding of security, compliance, and risk management in distributed systems. Exposure to chaos engineering and resilience testing. Minimum More ❯
external tools and data sources Understanding of secure, real-time data access methodologies. Streaming Data Processing: Hands-on experience with real-time streaming data processing technologies (e.g., Apache Kafka, Flink, Spark Streaming, Kinesis). Proficiency with cloud-native data platforms (e.g., AWS Kinesis/MSK/S3, Azure Event Hubs/Data Lake, GCP Pub/Sub/BigQuery More ❯
Experience with Apache Spark Hands-on experience with AWS data services (e.g., S3, Glue, Lambda, MSK) Capable in batch or streaming data processing using technologies such as Spark, Kafka, Flink, and DLT Proficient in CI/CD pipelines, automated testing, code quality enforcement, and environment management for production-grade data systems Expert in orchestration and transformation frameworks such as More ❯
Liverpool, Merseyside, North West, United Kingdom Hybrid / WFH Options
Forward Role
Excellent stakeholder management and documentation skills Team leadership experience with ability to mentor and develop engineering talent Nice to haves: Knowledge of data streaming platforms such as Kafka or Flink Exposure to graph databases or vector database technologies Professional certifications in Azure or AWS cloud platforms If you're ready to take the lead on transformative data engineering projects More ❯
Configure and manage data analytic frameworks and pipelines using databases and tools such as (but not limited to) NoSQL, SQL, NiFi, Kafka, HDInsight, MongoDB, Cassandra, Neo4j, GraphDB, OrientDB, Spark, Flink, Hadoop, Kafka, Hive, and others. • Apply distributed systems concepts and principles such as consistency and availability, liveness and safety, durability, reliability, fault-tolerance, consensus algorithms. • Administrate cloud computing and More ❯
leading data and ML platform infrastructure, balancing maintenance with exciting greenfield projects. develop and maintain our real-time model serving infrastructure, utilising technologies such as Kafka, Python, Docker, ApacheFlink, Airflow, and Databricks. Actively assist in model development and debugging using tools like PyTorch, Scikit-learn, MLFlow, and Pandas, working with models from gradient boosting classifiers to custom GPT More ❯
to Date with Technology: Keep yourself and the team updated on the latest Python technologies, frameworks, and tools like Apache Spark, Databricks, Apache Pulsar, Apache Airflow, Temporal, and ApacheFlink, sharing knowledge and suggesting improvements. Documentation: Contribute to clear and concise documentation for software, processes, and systems to ensure team alignment and knowledge sharing. Your Qualifications Experience: Professional experience … pipelines, and machine learning workflows. Workflow Orchestration: Familiarity with tools like Apache Airflow or Temporal for managing workflows and scheduling jobs in distributed systems. Stream Processing: Experience with ApacheFlink or other stream processing frameworks is a plus. Desired Skills Asynchronous Programming: Familiarity with asynchronous programming tools like Celery or asyncio. Frontend Knowledge: Exposure to frontend frameworks like React More ❯
delivering under tight deadlines without compromising quality. Your Qualifications 12+ years of software engineering experience, ideally in platform, infrastructure, or data-centric product development. Expertise in Apache Kafka, ApacheFlink, and/or Apache Pulsar. Deep understanding of event-driven architectures, data lakes, and streaming pipelines. Strong experience integrating AI/ML models into production systems, including prompt engineering More ❯
delivering under tight deadlines without compromising quality. Your Qualifications 12+ years of software engineering experience, ideally in platform, infrastructure, or data-centric product development. Expertise in Apache Kafka, ApacheFlink, and/or Apache Pulsar. Deep understanding of event-driven architectures, data lakes, and streaming pipelines. Strong experience integrating AI/ML models into production systems, including prompt engineering More ❯
with diverse stakeholders to identify priorities for the Fixed income data acquisition platform and the respective ownership of the roadmap We'd love to see: - Experience with Kafka, ApacheFlink, or Trino - Experience using SQL or Python to explore data - Experience in tick level databases and time series analytics - Systems thinkers - an interest or a natural preference to think More ❯
with a view to becoming an expert BS degree in Computer Science or meaningful relevant work experience Preferred Qualifications Experience with large scale data platform infrastructure such as Spark, Flink, HDFS, AWS/S3, Parquet, Kubernetes is a plus More ❯
Frameworks: Spring Boot, Flask Familiarity with multiple of the following: Analytical development: Machine Learning, Stream, Batch Virtualization Environments: Kubernetes, Containers (Docker, Containerd), Cloud (AWS/HCI) Streaming Frameworks: Kafka, Flink MongoDB, ArangoDB, Redis, PostgreSQL Desired skills: ElasticSearch Additional information: Work performed in contractor facility in Annapolis Junction Both low and high side access Potential telework Individual Capabilities Required: Seven More ❯
Columbia, Maryland, United States Hybrid / WFH Options
HII Mission Technologies
effectively navigating and contributing within the Risk Management Framework (RMF) process Proficiency in system design and meticulous documentation Experience in streaming and/or batch analytics (e.g. Kafka, Spark, Flink, Storm, MapReduce, Hadoop) Experience in distributed databases, NoSQL databases, full text-search engines (e.g. Elasticsearch, MongoDB, Solr) Experience in designing enterprise APIs. Experience in RESTful web services Experience in More ❯
of the platform Your Qualifications 12+ years of software engineering experience in enterprise-scale, data-centric, or platform environments Deep expertise in distributed data technologies such as Apache Kafka, Flink, and/or Pulsar Strong background in event-driven architectures, streaming pipelines, and data lakes Hands-on experience with AI/ML production systems, including prompt-based LLM integrations More ❯
lifecycle management and model versioning Hands-on experience with Databricks Model Serving for production ML deployments Proficiency with GenAI frameworks/tools and technologies such as Apache Airflow, Spark, Flink, Kafka/Kinesis, Snowflake, and Databricks. Demonstrable experience in parameter-efficient fine-tuning, model quantization, and quantization-aware fine-tuning of LLM models Hands-on knowledge of Chain-of More ❯
design and problem-solving skills We'd love to see : Experience building platforms that serve a large number of tenants Familiarity with large-scale data processing frameworks (e.g., Spark, Flink, Kafka) Passion for data quality, performance analysis, anomaly detection Familiarity with testing infrastructure, CI/CD workflows, or developer productivity tooling Experience with owning end-to-end delivery of More ❯
Liverpool, Merseyside, England, United Kingdom Hybrid / WFH Options
red recruitment
to manage secure, high-performance database environments. Excellent communication and cross-functional collaboration skills. A passion for continuous learning and innovation. Desirable: AzureSynapse/Sharedo/Databricks Python/Flink/Kafka technology If you are interested in this Senior Database Developer position and have the relevant skills and experience required, please apply now! Red Recruitment (Agency More ❯
plus Experience with Terraform and Kubernetes is a plus! A genuine excitement for significantly scaling large data systems Technologies we use (experience not required): AWS serverless architectures Kubernetes Spark Flink Databricks Parquet. Iceberg, Delta lake, Paimon Terraform Github including Github Actions Java PostgreSQL About Chainalysis Blockchain technology is powering a growing wave of innovation. Businesses and governments around the More ❯
MongoDB and Elasticsearch Experience with modern web UI frameworks such as Angular, Vue, React or Ember API development experience Familiarity with streaming/event-based architecture (Apache Spark, ApacheFlink) Familiarity with NiFi The Benefits Package Wyetech believes in generously supporting employees as they prepare for retirement. The company automatically contributes 20% of each employee's gross compensation to More ❯
Apache Hadoop/Cloudera) (all genders) Aufgaben Administrate, monitor and optimize our Big Data environment based on Apache Hadoop from Cloudera (AWS-Cloud) Manage and maintain services like Kafka, Flink, NiFi, DynamoDB and Iceberg Tables IaaC deployment via Terraform Plan and execute updates/upgrades Advise our Data Engineers and Data Scientists on the selection of Hadoop services for More ❯
Apache Hadoop/Cloudera) (all genders) Aufgaben Administrate, monitor and optimize our Big Data environment based on Apache Hadoop from Cloudera (AWS-Cloud) Manage and maintain services like Kafka, Flink, NiFi, DynamoDB and Iceberg Tables IaaC deployment via Terraform Plan and execute updates/upgrades Advise our Data Engineers and Data Scientists on the selection of Hadoop services for More ❯