11 of 11 Apache Spark Jobs in Northern Ireland

application components, microservices, and data integration layers to support high-throughput regulatory reporting. Big Data Processing: Develop and optimize distributed data processing jobs using Apache Spark, Hadoop, and Hive to aggregate and transform large-scale financial datasets. Real-Time Streaming: Build and maintain real-time data ingestion … streaming pipelines using Apache Kafka. Database Management: Design and optimize data models in MongoDB and relational databases, ensuring high performance for complex queries and data storage. Agile Execution: Drive sprint planning, task estimation, and daily progress within an Agile/Scrum framework to ensure timely delivery of project milestones. ...

background in software engineering, product management or data analytics. Strong proficiency in Python or Java, SQL, and data pipeline frameworks (e.g., Airflow, dbt, Spark), with solid experience with the AWS ecosystem. Excellent problem-solving skills and ability to work in a fast-paced environment. Bachelor’s degree in Computer ...

Snowflake, Databricks). Experience with data integration (ETL/ELT, APIs). Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) is a plus. Domain Knowledge Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations. In-depth knowledge ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS. The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled. As platform landscape evolved towards cloud-native, AI enabled ...

design artefacts and delivery roadmaps Key Requirements 7+ yearsâ experience in data engineering or data platform development Strong experience with modern cloud data platforms, Spark and distributed data processing Experience with data cataloguing and governance tooling, open table formats (e.g. Delta Lake, Iceberg) and streaming architectures Advanced Python ...

unstructured workloads at scale - Integrate multiple sources into modern cloud platforms - Collaborate across engineering and delivery on mission-critical programmes **The environment** Python, SQL, Spark, Java or Scala · AWS, Azure or GCP · modern distributed and cloud-native architectures **What we need** - Active DV clearance - essential - Production-scale pipeline experience ...

solutions, and lead their implementation. Good written and verbal communication skills. Any Beneficial/Nice to have experience: Experience with Big Data technology (HDFS, Spark, Impala) Understanding of Cloud‐native development principles. Experience with Container Orchestration tools (Docker, Kubernetes, OpenShift, etc.). Proficiency in using Jira for tracking tasks ...

tenant isolation, row- and column-level security, and PII handling. Define the streaming and batch ingestion architecture, building on our investments in NATS JetStream, Apache Flink, and Debezium CDC patterns, and direct the engineering team on execution. Specify the catalog, lineage, and data-quality foundation and own the operational … streaming vs batch). Hands-on experience with: Dremio, Snowflake, or Databricks AWS data services (S3, Glue, Athena, Lake Formation, or equivalent) NATS/Apache Kafka or similar, and CDC patterns (Debezium or equivalent) Stream processing (Apache Flink, Spark Structured Streaming, or equivalent) SQL and query optimisation ...

designing and building robust, scalable, distributed data systems and pipelines, using open source and public cloud technologies. Strong experience with data orchestration tools: e.g. Apache Airflow, Dagster. Experience with big data storage and processing technologies: e.g. DBT, Spark, SQL, Athena/Trino, Redshift, Snowflake, RDBMSs (PostgreSQL/MySQL … . Knowledge of event‐driven architectures and streaming technologies: e.g. Apache Kafka, Kafka Streams, Apache Flink. Experience with public cloud environments: e.g. AWS, GCP, Azure, Terraform. Strong knowledge of software engineering practices: e.g. testing, CI/CD (Jenkins, Github Actions), agile development, git/version control, containers etc. ...

regulations, and implement AI-driven automation while ensuring scalable, reliable data processing across big data stacks. The role emphasizes real-time streaming with Kafka, Spark-based batch processing, and close collaboration with Compliance to translate complex rules into robust software solutions. #J-18808-Ljbffr ...

driving delivery across complex engagements. Key Responsibilities Lead design and implementation of enterprise-scale data pipelines and lakehouse architectures Provide technical leadership across Databricks, Spark and cloud data platforms Drive delivery of data engineering workstreams within Agile programmes Engage with client stakeholders to translate business objectives into technical solutions … Produce architecture documentation, design artefacts and delivery roadmaps Key Requirements 7+ years experience in data engineering or data platform development Strong experience with Databricks, Spark and distributed data processing Experience with Unity Catalog, Delta Lake and streaming architectures Advanced Python and Microsoft SQL Server development Proven experience leading engineering ...