1 to 25 of 48 Spark Streaming Jobs

data from diverse sources, transform it into usable formats, and load it into data warehouses, data lakes or lakehouses. Big Data Technologies: Utilize big data technologies such as Spark, Kafka, and Flink for distributed data processing and analytics. Cloud Platforms: Deploy and manage data solutions on cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP), leveraging … for data manipulation and scripting. Strong understanding of data modelling concepts and techniques, including relational and dimensional modelling. Experience in big data technologies and frameworks such as Databricks, Spark, Kafka, and Flink. Experience in using modern data architectures, such as lakehouse. Experience with CI/CD pipelines, version control systems like Git, and containerization (e.g., Docker). Experience … deploying and managing data solutions. Strong problem-solving and analytical skills with the ability to diagnose and resolve complex data-related issues. SQL (for database management and querying) Apache Spark (for distributed data processing) Apache Spark Streaming, Kafka or similar (for real-time data streaming) Experience using data tools in at least More ❯

data from diverse sources, transform it into usable formats, and load it into data warehouses, data lakes or lakehouses. Big Data Technologies: Utilize big data technologies such as Spark, Kafka, and Flink for distributed data processing and analytics. Cloud Platforms: Deploy and manage data solutions on cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP), leveraging … for data manipulation and scripting. Strong understanding of data modelling concepts and techniques, including relational and dimensional modelling. Experience in big data technologies and frameworks such as Databricks, Spark, Kafka, and Flink. Experience in using modern data architectures, such as lakehouse. Experience with CI/CD pipelines and version control systems like Git. Knowledge of ETL tools and … deploying and managing data solutions. Strong problem-solving and analytical skills with the ability to diagnose and resolve complex data-related issues. SQL (for database management and querying) Apache Spark (for distributed data processing) Apache Spark Streaming, Kafka or similar (for real-time data streaming) Experience using data tools in at least More ❯

Accountabilities · Develop robust, scalable data pipelines to serve the easyJet analyst and data science community. · Highly competent hands-on experience with relevant Data Engineering technologies, such as Databricks, Spark, Spark API, Python, SQL Server, Scala. · Work with data scientists, machine learning engineers and DevOps engineers to develop, develop and deploymachine learning models and algorithms aimed at … partitioning. · Hands-on IaC development experience with Terraform or CloudFormation. · Understanding of ML development workflow and knowledge of when and how to use dedicated hardware. · Significant experience with Apache Spark or any other distributed data programming frameworks (e.g. Flink, Hadoop, Beam) · Familiarity with Databricks as a data and AI platform or the Lakehouse Architecture. · Experience with data quality … solutions (e.g. access management, data privacy, handling of sensitive data (e.g. GDPR) · Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. · Understanding of the challenges faced in the design and development of a streaming data pipeline and the different options for More ❯

Accountabilities Develop robust, scalable data pipelines to serve the easyJet analyst and data science community. Highly competent hands-on experience with relevant Data Engineering technologies, such as Databricks, Spark, Spark API, Python, SQL Server, Scala. Work with data scientists, machine learning engineers and DevOps engineers to develop, develop and deploy machine learning models and algorithms aimed … partitioning. Hands-on IaC development experience with Terraform or CloudFormation. Understanding of ML development workflow and knowledge of when and how to use dedicated hardware. Significant experience with Apache Spark or any other distributed data programming frameworks (e.g. Flink, Hadoop, Beam) Familiarity with Databricks as a data and AI platform or the Lakehouse Architecture. Experience with data quality … access management, data privacy, handling of sensitive data (e.g. GDPR) Desirable Skills Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. Understanding of the challenges faced in the design and development of a streaming data pipeline and the different options for More ❯

Accountabilities Develop robust, scalable data pipelines to serve the easyJet analyst and data science community. Highly competent hands-on experience with relevant Data Engineering technologies, such as Databricks, Spark, Spark API, Python, SQL Server, Scala. Work with data scientists, machine learning engineers and DevOps engineers to develop, develop and deploy machine learning models and algorithms aimed … partitioning. Hands-on IaC development experience with Terraform or CloudFormation. Understanding of ML development workflow and knowledge of when and how to use dedicated hardware. Significant experience with Apache Spark or any other distributed data programming frameworks (e.g. Flink, Hadoop, Beam) Familiarity with Databricks as a data and AI platform or the Lakehouse Architecture. Experience with data quality … solutions (e.g. access management, data privacy, handling of sensitive data (e.g. GDPR) Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. Understanding of the challenges faced in the design and development of a streaming data pipeline and the different options for More ❯

data pipeline design, ETL/ELT processes, data warehousing, and data modeli ng. Design, build, and maintain scalable and reliable data infrastruct ure (e.g., data lakes, data warehouses, streaming platforms) on cloud platforms (e.g., AWS, Azure, GCP). Develop and optimi ze data ingestion, transformation, and loading proces ses from various sources.Ensu re data quality, integrity, and governa … Informatica, Talend, dbt). Solid understanding of data warehousing conc epts (e.g., Kimball, Inmon) and technologies (e.g., Snowflake, Redshift, BigQuery) Experience w ith big data technolo gies (e.g., Spark, Hadoop, Kafka). Familiarity w ith cloud platf orms (AWS, Azure, or GCP) and their data services. Excell ent problem-solving and analytical sk ills. Str ong communication and … ills, with the ability to explain complex technical concepts to non-technical audien ces. Preferred Qualificat io ns: Master's d egree in a relevant field Experience with streaming data proce ssing (e.g., Kafka Streams, Flink, Spark Streaming).Knowledg e of data modeling techniques and tools. Experience with containerization technol ogies (e.g., Docker, Kubernetes More ❯

Technical Skills Required · Hands-on software development experience with Python and experience with modern software development and release engineering practices (e.g. TDD, CI/CD). · Experience with Apache Spark or any other distributed data programming frameworks. · Comfortable writing efficient SQL and debugging on cloud warehouses like Databricks SQL or Snowflake. · Experience with cloud infrastructure like AWS or … Hands-on development experience in an airline, e-commerce or retail industry · Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. · Experience implementing end-to-end monitoring, quality checks, lineage tracking and automated alerts to ensure reliable and trustworthy data across the platform. More ❯

Technical Skills Required Hands-on software development experience with Python and experience with modern software development and release engineering practices (e.g. TDD, CI/CD). Experience with Apache Spark or any other distributed data programming frameworks. Comfortable writing efficient SQL and debugging on cloud warehouses like Databricks SQL or Snowflake. Experience with cloud infrastructure like AWS or … Hands-on development experience in an airline, e-commerce or retail industry Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. Experience implementing end-to-end monitoring, quality checks, lineage tracking and automated alerts to ensure reliable and trustworthy data across the platform. More ❯

with a variety of stakeholders and cross-functional teams, performing analysis of their data requirements and documenting it. Big data tools and stream-processing systems such as: Hadoop, Spark, Kafka, Storm, Spark-Streaming. Relational SQL and NoSQL databases, including Postgres and Cassandra. Experience designing and implementing knowledge graphs for data integration and analysis. Data pipeline and More ❯

AWS. Build out real-world architectures that business engineering teams buy into and build their applications around. Required Qualifications, Capabilities, and Skills: Experience across the data lifecycle with Spark-based frameworks for end-to-end ETL, ELT & reporting solutions using key components like Spark SQL & Spark Streaming. Strong knowledge of multi-threading and high … mindset geared towards a fantastic end-to-end engineering experience supported by excellent tooling and automation. Preferred Qualifications, Capabilities, and Skills: Good understanding of the Big Data stack (Spark/Iceberg). Ability to learn new technologies and patterns on the job and apply them effectively. Good understanding of established patterns, such as stability patterns/anti-patterns More ❯

disciplines. Bonus Qualifications: Master's degree in computer science, engineering, or related field. Expertise with TigerGraph and optimizing queries (using GSQL) for large-scale graph traversals. Knowledge of streaming frameworks (Apache Flink, Spark Streaming, etc.). Expertise with Spring framework (Spring MVC, Spring Integration, Spring Batch). More ❯

engineering with a proven track record of building and managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real-time data ingestion and storage. More ❯

meet functional/non-functional project requirements Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, streaming and 'big data' technologies Implement data pipelines to ingest data to the platform, standardize and transform the data Support the development of analytics tools that utilize the data pipeline … industry leader. Work with data and analytics experts to strive for greater functionality in our data systems. Design and architect solutions with Big Data technologies (e.g Hadoop, Hive, Spark, Kafka) Design and implement systems that run at scale leveraging containerized deployments Design, build, and scale data pipelines across a variety of source systems and streams (internal, third-party … Science, Computer Engineering, Informatics, Information Systems, or another quantitative field Minimum 5 years of experience in a Data Engineer role Required Skills: Experience with big data tools: Hadoop, Spark, etc. Experience with relational SQL and NoSQL databases, including Postgres Experience with AWS cloud or remote services: EC2, EMR, RDS, Redshift Experience with stream-processing systems: Kafka, Storm, Spark More ❯

Experience with cron jobs , job orchestration, and error monitoring tools. Good to have Experience with Azure Bicep or other Infrastructure-as-Code tools. Exposure to real-time/streaming data (Kafka, Spark Streaming, etc.). Understanding of data mesh , data contracts , or domain-driven data architecture . Hands on experience with MLflow and Llama More ❯

Key Responsibilities Design and implement real-time data pipelines using tools like Apache Kafka, Apache Flink, or Spark Streaming. Develop and maintain event schemas using Avro, Protobuf, or JSON Schema. Collaborate with backend teams to integrate event-driven microservices. Ensure data quality, lineage, and observability across streaming systems. Optimize performance and scalability of streaming applications. Implement CI/CD pipelines for data infrastructure. Monitor and troubleshoot production data flows and streaming jobs. Required Skills & Qualifications 3+ years of experience in data engineering or backend development. Strong programming skills in Python, Java, or Scala. Hands-on experience with Kafka, Kinesis, or similar messaging systems. Familiarity with stream processing frameworks like Flink … Kafka Streams, or Spark Structured Streaming. Solid understanding of event-driven design patterns (e.g., event sourcing, CQRS). Experience with cloud platforms (AWS, GCP, or Azure) and infrastructure-as-code tools. Knowledge of data modeling, schema evolution, and serialization formats More ❯

Key Responsibilities Design and implement real-time data pipelines using tools like Apache Kafka, Apache Flink, or Spark Streaming. Develop and maintain event schemas using Avro, Protobuf, or JSON Schema. Collaborate with backend teams to integrate event-driven microservices. Ensure data quality, lineage, and observability across streaming systems. Optimize performance and scalability of streaming applications. Implement CI/CD pipelines for data infrastructure. Monitor and troubleshoot production data flows and streaming jobs. Required Skills & Qualifications 3+ years of experience in data engineering or backend development. Strong programming skills in Python, Java, or Scala. Hands-on experience with Kafka, Kinesis, or similar messaging systems. Familiarity with stream processing frameworks like Flink … Kafka Streams, or Spark Structured Streaming. Solid understanding of event-driven design patterns (e.g., event sourcing, CQRS). Experience with cloud platforms (AWS, GCP, or Azure) and infrastructure-as-code tools. Knowledge of data modeling, schema evolution, and serialization formats More ❯

Key Responsibilities Design and implement real-time data pipelines using tools like Apache Kafka, Apache Flink, or Spark Streaming. Develop and maintain event schemas using Avro, Protobuf, or JSON Schema. Collaborate with backend teams to integrate event-driven microservices. Ensure data quality, lineage, and observability across streaming systems. Optimize performance and scalability of streaming applications. Implement CI/CD pipelines for data infrastructure. Monitor and troubleshoot production data flows and streaming jobs. Required Skills & Qualifications 3+ years of experience in data engineering or backend development. Strong programming skills in Python, Java, or Scala. Hands-on experience with Kafka, Kinesis, or similar messaging systems. Familiarity with stream processing frameworks like Flink … Kafka Streams, or Spark Structured Streaming. Solid understanding of event-driven design patterns (e.g., event sourcing, CQRS). Experience with cloud platforms (AWS, GCP, or Azure) and infrastructure-as-code tools. Knowledge of data modeling, schema evolution, and serialization formats More ❯

engineering with a proven track record of building and managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real-time data ingestion and storage. More ❯

AI-driven analytics and automation. By leveraging Azure-native services, such as Azure Data Factory (ADF) for orchestration, Delta Lake for ACID-compliant data storage, and Databricks Structured Streaming for real-time data processing, the Data Engineer will help unlock insights, enhance pricing accuracy, and drive innovation. The role also includes optimising Databricks query performance, implementing robust security … data and ensure FCA/PRA regulatory compliance. Innovation & Continuous Improvement – Identify and implement emerging data technologies within the Azure ecosystem, such as Delta Live Tables (DLT), Structured Streaming, and AI-driven analytics to enhance business capabilities. Skills, Knowledge and Expertise Hands-on experience in building ELT pipelines and working with large-scale datasets using Azure Data Factory … ADF) and Databricks. Strong proficiency in SQL (T-SQL, Spark SQL) for data extraction, transformation, and optimisation. Proficiency in Azure Databricks (PySpark, Delta Lake, Spark SQL) for big data processing. Knowledge of data warehousing concepts and relational database design, particularly with Azure Synapse Analytics. Experience working with Delta Lake for schema evolution, ACID transactions, and time More ❯

Teamwork makes the stream work. Roku Is Changing How The World Watches TV Roku is the #1 TV streaming platform in the US and Mexico, and we've set our sights on powering every television in the world. Roku pioneered streaming to the TV. Our mission is to be the TV streaming platform that … where you’ll work across product, data, and engineering teams to build scalable and efficient solutions for content delivery and marketing performance. About The Role Roku pioneered TV streaming and continues to innovate and lead the industry. The Roku Channel has us well-positioned to help shape the future of streaming. Our continued success depends on building customer … on product and engineering requirements. An understanding of key metrics across various business units and how they interrelate. Experience developing large-scale data pipelines. Experience in dealing with streaming and batch compute frameworks like Spring Kafka, Kafka Streams, Flink, Spark Streaming, Spark. Experience with large-scale computing platforms such as Hadoop, Hive, Spark More ❯

in Computer Science, Statistics, Informatics, Information Systems, or another quantitative field. They should also have experience using the following software/tools: Experience with big data tools: Hadoop, Spark, Kafka, etc. Experience with relational SQL and NoSQL databases, including Postgres and Cassandra. Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc. Experience with AWS cloud … services: EC2, EMR, RDS, Redshift. Experience with stream-processing systems: Storm, Spark-Streaming, etc. Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc. Salary: 30000 per annum + benefits Apply For This Job If you would like to apply for this position, please fill in the information below and submit it More ❯

etc.). Understanding of data governance, data quality, and security standards. Fluent in French or Dutch ; good knowledge of English is a plus. Nice to Have Experience with streaming data technologies (e.g., Apache Kafka, Spark Streaming). Exposure to CI/CD pipelines and DevOps practices. Familiarity with BI tools (e.g., Power BI, Tableau More ❯

below. AI techniques (supervised and unsupervised machine learning, deep learning, graph data analytics, statistical analysis, time series, geospatial analysis, NLP, sentiment analysis, pattern detection, etc.) Python, R, or Spark for data insights Data Bricks/Data QISQL for data access and processing (PostgreSQL preferred, but general SQL knowledge is important) Latest Data Science platforms (e.g., Databricks, Dataiku, AzureML … SageMaker) and frameworks (e.g., TensorFlow, MXNet, scikit-learn) Software engineering practices (coding standards, unit testing, version control, code review) Hadoop distributions (Cloudera, Hortonworks), NoSQL databases (Neo4j, Elastic), streaming technologies (Spark Streaming) Data manipulation and wrangling techniques Development and deployment technologies (virtualisation, CI tools like Jenkins, configuration management with Ansible, containerisation with Docker, Kubernetes) Data More ❯

consolidated Data Platform with innovative features while working with a talented and fun team. Responsibilities include: Development and maintenance of Real-Time Data Processing applications using frameworks like Spark Streaming, Spark Structured Streaming, Kafka Streams, and Kafka Connect. Manipulation of streaming data, including ingestion, transformation, and aggregation. Researching and developing … tasks, and working effectively within a team. Documenting processes and sharing knowledge with the team. Preferred skills: Strong knowledge of Scala. Familiarity with distributed computing frameworks such as Spark, KStreams, Kafka. Experience with Kafka and streaming frameworks. Understanding of monolithic vs. microservice architectures. Familiarity with Apache ecosystem including Hadoop modules (HDFS, YARN, HBase, Hive, Spark More ❯

ensure high availability, resiliency, and scalability, including understanding and explaining features like KRAFT. Integrating Kafka with other data processing tools and platforms such as Kafka Streams, Kafka Connect, Spark Streaming, Schema Registry, Flink, and Beam. Collaborating with cross-functional teams to understand data requirements and design solutions that meet business needs. Implementing security measures to protect … on-premise/cloudera open source Kafka Focus on disaster recovery aspects Knowledge of Kafka resiliency and new features like KRAFT Experience with real-time technologies such as Spark Required Skills & Experience Extensive experience with Apache Kafka and real-time architecture including event-driven frameworks. Strong knowledge of Kafka Streams, Kafka Connect, Spark Streaming More ❯