201 to 225 of 234 Apache Spark Jobs in the UK

model lifecycle: data prep, experimentation, deployment, and monitoring.* Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS.* Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing.* Ensure data quality, observability, lineage, and documentation across data and ML pipelines. … Direct experience with OpenAI APIs, LLM workflows, and prompt engineering.* Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering.* Experience with Databricks, Spark, and Delta Lake.* Strong SQL skills with experience working on large datasets.* Experience with AWS, including S3 and Lambda.* Familiarity with Redshift, Snowflake ...

Position Summary The Mid-Level AI Architect is responsible for designing and implementing AI and cloud architectures, with a focus on integrating generative AI technologies. This role requires a solid understanding of AI and cloud ...

improvements in tooling and processes. Ideal candidates will have 5+ years in backend engineering for data systems, strong experience with MongoDB, Postgres, Kinesis, Flink, Spark, ElasticSearch, Redis, and query engines; plus #J-18808-Ljbffr ...

. Knowledge of data warehousing concepts, including data warehouse technical architectures, infrastructure components, ETL/ELT and reporting/analytic tools and environments (e.g., Apache Beam, Hadoop, Spark, Pig, Hive, MapReduce). Understanding of the auxiliary practical concerns in production machine learning systems. About the job The Google ...

experience owning and operating production data infrastructure at significant scale. Proven capability delivering large-scale streaming data solutions using technologies such as Kafka, Flink, Spark Streaming, Dataflow, or similar. Demonstrated success leading end-to-end migrations of live production systems. Strong knowledge of cloud platforms and distributed data architectures. … with data scientists, analysts, and business stakeholders. Strong understanding of data platform reliability, performance, scalability, and cost optimisation. Experience with technologies such as ClickHouse, Apache Iceberg, BigQuery, dbt, Elasticsearch, Druid, GCP, or AWS is highly desirable. If you're an experienced Senior Data Engineer looking to lead complex data ...

REQ ID: FEQ227R246 Location: London, UK Recruiter: Dina Hussain At Databricks, our core values are at the heart of everything we do; a culture of proactiveness and a customer‐centric mindset guide us to build ...

Data Architect Inside IR35 up to £600+ Nottingham Hybrid Summary We're undertaking a fast paced data transformation into Databricks at E.ON Next using best practice data governance and architectural principles, and we are growing ...

engine that provides cross-asset coverage to calculate various Credit Risk measures. FRA platform fraimWRX is distributed as AKS images. FraimWRX relies on Redis, Apache Kafka, Spark, HBASE and HDFS. Risk technology are using Kubernetes to deploy fraimWRX with a Python based Coordinator and Job Runner REST components. ...

coupled with a strong interest in continuous learning. Flexibility to travel up to 20% of the time. Preferred Technical and Professional Experience Experience with Apache Kafka and Apache Flink. Experience helping customers build distributed systems or streaming solutions that use Kafka alongside technologies such as Spark, Flink ...

must have demonstrable experience designing, deploying, and operating big data platforms at scale. You will have hands-on experience with OSS data ecosystems (e.g., Spark, Trino, Airflow, Apache Superset, and similar technologies). You will have knowledge of cloud platforms and traditional on-premise hosting; hands-on experience ...

software and site reliability engineering. Preferred: Experience with distributed ML frameworks such as Horovod or TensorFlow Distributed. Familiarity with data engineering pipelines such as Apache Airflow or Apache Spark. Knowledge of chaos engineering tools and compliance frameworks such as GDPR, SOC 2, or ISO 27001. Relocation benefits ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine‐tuning, and deployment tools (e.g., HuggingFace, Langchain). Hands‐on experience working with Distributed Spark based systems. Experience with data engineering concepts or a good understanding of data engineering concepts. Pre‐sales or post‐sales experience working with external … clients across a variety of industry markets. Minimum of 5+ years of customer‐facing experience would be preferred. [Preferred] Experience working with Apache SparkTM to process large‐scale distributed datasets. Communication: Proven ability to communicate and teach complex technical concepts to both technical and non‐technical audiences. Core Traits ...

processing, including vector databases, fine-tuning LLMs, and deploying LLMs with tools such as HuggingFace, Langchain, and OpenAI Hands-on experience working with Distributed Spark based systems. Experience communicating and teaching technical concepts to non-technical and technical audiences alike Passion for collaboration, life-long learning, and driving … values through ML [Preferred] 2+ years customer-facing experience in a pre-sales or post-sales role [Preferred] Experience working with Apache SparkTM to process large-scale distributed datasets Can meet expectations for technical training and role-specific outcomes within 3 months of hire Can travel ...

strategy and promote best practices amongst engineers and data scientists. How to make an Impact Architect and Implement : Own the technical roadmap for our Spark‐based data processes to ensure our Airflow pipelines are performant, cost‐effective, and scalable. You will enhance our existing services through hands‐on development … deployment. Expert‐level Python and its data ecosystem (Numpy, Pandas), including designing frameworks for data tasks. Deep understanding of distributed data processing engines like Apache Spark. Ability to reason about the full data lifecycle from ingestion through to real‐time serving - your system designs will need to meet downstream ...

adoption of modern data mesh principles across complex enterprise environments. What You'll Be Doing Architect and deliver modern data solutions using Databricks, Spark, PySpark and SQL Design scalable batch and real-time data pipelines that ingest data from legacy and modern platforms Lead the development of cloud-based … decision making Provide technical leadership and architectural direction on large-scale transformation programmes Essential Experience Strong hands-on expertise with Databricks Extensive experience using Spark, PySpark and SQL Proven track record designing and implementing data lake architectures Experience building both batch and streaming data pipelines Strong knowledge of Kafka ...

At Kpler, we are dedicated to helping our clients navigate complex markets with ease. By simplifying global trade information and providing valuable insights, we empower organisations to make informed decisions in commodities, energy, and maritime ...

At Databricks, our core values are at the heart of everything we do; creating a culture of proactiveness and a customer‐centric mindset guides us to create a unified platform that makes data science and ...

At the company, our core values are at the heart of everything we do. A culture of proactiveness and a customer‐centric mindset guides us in creating a unified platform that makes data science and ...

JOB ID: FEQ427R333 Location: London, United Kingdom Industry: Retail/CPG (UKI Enterprise) Recruiter: Dina Hussain At the company, our core values are at the heart of everything we do. A culture of proactiveness and ...

modeling and the creation of type-safe and easily testable batch jobs. The team also orchestrates a compute environment that simplifies the execution of Spark jobs, facilitating the analysis of terabytes of data with ease and cost-effectiveness. The runtime environment is based on Hadoop clusters on EC2 spot … Identifying and implementing computational improvements Making improvements and/or fixing bugs by submitting patches to open-source projects in our stack, such as Spark, Iceberg, Hadoop, Airflow, and many others Acting as an expert on Spark and Scala Mentoring engineers (including through pairing, design discussions, and knowledge ...

replication tool, and Waggle Dance , a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg , a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures.We guarantee you’ll learn … focus on scalability, reliability, and observability.Strong knowledge of the JVM and server-side Kotlin/Java programming.Solid understanding of the Hadoop ecosystem, including Spark, Hadoop, and Hive.Strong experience with cloud platforms and data infrastructure, especially AWS, including EMR, S3, and Glue.Experience working in Agile environments and contributing through code ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn … scalability, reliability, and observability. Strong knowledge of the JVM and server-side Kotlin/Java programming. Solid understanding of the Hadoop ecosystem, including Spark, Hadoop, and Hive. Strong experience with cloud platforms and data infrastructure, especially AWS, including EMR, S3, and Glue. Experience working in Agile environments and contributing ...

experimentation frameworks that support business decision-making. Data Engineering & Model Productionisation: Develop robust analytical datasets and machine learning pipelines using Databricks, Python, SQL and Spark, ensuring solutions are scalable, reliable and production-ready. Stakeholder Partnership & Commercial Impact: Partner with business stakeholders to define hypotheses, success measures and value metrics … analytics solutions within commercial environments. Python, SQL & Data Platform Expertise: Advanced Python and SQL skills, with experience building reproducible machine learning workflows using Databricks, Spark or similar large‐scale data processing technologies. Machine Learning & Statistical Expertise: Deep understanding of supervised and unsupervised machine learning, feature engineering, model evaluation, experimentation ...

Hands‐on experience in system design, application development, testing, and operational stability Proficient in coding in one or more languages such as Java or Spark Experience developing, debugging, and maintaining code in a large corporate environment with modern programming and database querying languages Knowledge of the Software Development Life … within a technical discipline (e.g., cloud, artificial intelligence, machine learning, mobile) Ability to solve data‐oriented problems using technologies such as SQL, Relational DB, Spark, NoSQL, optimizing for performance Preferred Qualifications, Capabilities, and Skills Experience with Spark performance tuning of complex calculations on large datasets Experience delivering production ...

within a technical discipline (e.g., cloud, AI, machine learning, mobile). Ability to solve data-oriented problems using technologies such as SQL, relational databases, Spark, and NoSQL. Preferred Qualifications, Capabilities, and Skills Additional formal training or certification in software engineering concepts. Exposure to AI tools and technologies for software … development. Experience with Spark performance tuning for complex calculations on large datasets. Experience delivering production changes to complex software using Scala and Spark. If you’re ready to lead, innovate, and make an impact, we invite you to join our team and shape the future of finance technology with ...