10 of 10 Remote Spark Streaming Jobs

features while working with a talented and fun team. Responsibilities include: Development and maintenance of Real-Time Data Processing applications using frameworks like Spark Streaming, Spark Structured Streaming, Kafka Streams, and Kafka Connect. Manipulation of streaming data, including ingestion, transformation … a team. Documenting processes and sharing knowledge with the team. Preferred skills: Strong knowledge of Scala. Familiarity with distributed computing frameworks such as Spark, KStreams, Kafka. Experience with Kafka and streaming frameworks. Understanding of monolithic vs. microservice architectures. Familiarity with Apache ecosystem including Hadoop modules (HDFS … YARN, HBase, Hive, Spark) and Apache NiFi. Experience with containerization and orchestration tools like Docker and Kubernetes. Knowledge of time-series or analytics databases such as Elasticsearch. Experience with AWS services like S3, EC2, EMR, Redshift. Familiarity with data monitoring and visualization tools such as Prometheus and Grafana. More ❯

scalability, including understanding and explaining features like KRAFT. Integrating Kafka with other data processing tools and platforms such as Kafka Streams, Kafka Connect, Spark Streaming, Schema Registry, Flink, and Beam. Collaborating with cross-functional teams to understand data requirements and design solutions that meet business needs. … source Kafka Focus on disaster recovery aspects Knowledge of Kafka resiliency and new features like KRAFT Experience with real-time technologies such as Spark Required Skills & Experience Extensive experience with Apache Kafka and real-time architecture including event-driven frameworks. Strong knowledge of Kafka Streams, Kafka Connect, Spark Streaming, Schema Registry, Flink, and Beam. Experience with cloud platforms such as GCP Pub/Sub. Excellent problem-solving skills. Knowledge & Experience/Qualifications Knowledge of Kafka data pipelines and messaging solutions to support critical business operations and enable real-time data processing. Monitoring Kafka performance More ❯

record of building and managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real More ❯

record of building and managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real More ❯

governance, data quality, and security standards. Fluent in French or Dutch ; good knowledge of English is a plus. Nice to Have Experience with streaming data technologies (e.g., Apache Kafka, Spark Streaming). Exposure to CI/CD pipelines and DevOps practices. Familiarity with BI More ❯

By leveraging Azure-native services, such as Azure Data Factory (ADF) for orchestration, Delta Lake for ACID-compliant data storage, and Databricks Structured Streaming for real-time data processing, the Data Engineer will help unlock insights, enhance pricing accuracy, and drive innovation. The role also includes optimising Databricks … PRA regulatory compliance. Innovation & Continuous Improvement – Identify and implement emerging data technologies within the Azure ecosystem, such as Delta Live Tables (DLT), Structured Streaming, and AI-driven analytics to enhance business capabilities. Skills, Knowledge And Expertise Hands-on experience in building ELT pipelines and working with large-scale … datasets using Azure Data Factory (ADF) and Databricks. Strong proficiency in SQL (T-SQL, Spark SQL) for data extraction, transformation, and optimisation. Proficiency in Azure Databricks (PySpark, Delta Lake, Spark SQL) for big data processing. Knowledge of data warehousing concepts and relational database design, particularly with More ❯

Airflow for scheduling DBT for data transformation Montecarlo for monitoring The platform here encompasses the end-to-end, first for real time/streaming use cases but also for our analytical/warehouse needs. We're building for scale. As such, much of what we design and implement … processes, we're aiming for the platform to be as self-sustaining as possible. Stay up-to-date with the latest data and streaming engineering technologies and trends. Use that knowledge and subject matter expertise to mentor the more junior members of the team, and work with other … stream technologies, ideally Kafka, but Kinesis, Pulsar or similar would also be applicable. Experience designing and implementing stream processing applications (kStreams, kSQL, Flink, Spark Streaming). Experience with Data Warehousing tools like Snowflake/Bigquery/Databricks, and building pipelines on these. Experience working with modern More ❯

forefront of enterprise-scale AI adoption , working with AWS cloud-native event-driven architectures that will shape the future of AI-powered data streaming and automation. The Role As an AWS Platform Engineer , you will play a key role in designing, deploying, and optimising event-driven architectures in … an AWS cloud environment. You will work alongside AI specialists, DevOps engineers, and data teams to build highly scalable, real-time data streaming solutions that drive AI-driven automation and decision-making. This is a client-facing role , requiring: Exceptional communication skills to engage with both technical and … remaining three days. Essential Skills and Experience Strong expertise in AWS cloud services , with practical experience or theoretical knowledge of AWS MSK (Managed Streaming for Apache Kafka) Deep understanding of Kafka (Apache Kafka, Confluent Kafka) and event-driven microservices architecture Proficiency in Infrastructure as Code tools (Terraform, AWS More ❯

and cross-functional teams, performing analysis of their data requirements and documenting it. Big data tools and stream-processing systems such as: Hadoop, Spark, Kafka, Storm, Spark-Streaming. Relational SQL and NoSQL databases, including Postgres and Cassandra. Experience designing and implementing knowledge graphs for data integration More ❯

and cross-functional teams, performing analysis of their data requirements and documenting it. Big data tools and stream-processing systems such as: Hadoop, Spark, Kafka, Storm, Spark-Streaming. Relational SQL and NoSQL databases, including Postgres and Cassandra. Experience designing and implementing knowledge graphs for data integration More ❯