1 to 25 of 151 Apache Spark Jobs in London

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

DescriptionRole – Senior ConsultantTechnology – Scala, Apache Spark, SQL, PythonNice to Have - AWS, UNIX shell scriptingLocation – London, UKContext and Job Description:•Infosys provides Integration/middleware services for global clients covering a multitude of Integration technologies and products.•The services are rendered typically following a global delivery model using … fundamental Data Structures and their usage •At least 5+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. •Having expertise in Spark Core, Spark SQL and Spark Streaming.•Experience with Python, Hadoop, HDFS, Hive ...

risk quants on massive-scale analytics for critical regulations like FRTB? Do you want to tame billions of rows of complex financial data with Apache Spark and deliver insights to senior leaders in a fraction of a second? Citi is looking for an elite, hands-on technologist … data pipelines to ingest and process billions of trade-level PV calculations from various stress engines. Develop and optimize large-scale aggregation jobs using Apache Spark, ensuring high performance and efficiency. Design and deliver a suite of "intelligent data APIs" that provide flexible, on-demand access to both ...

modern AI systems. Strong programming experience in Python. Data & Infrastructure Experience with some of the following technologies is highly desirable: Frameworks/Infrastructure Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Libraries Pandas Polars Engineering ...

Solution Engineering, or related disciplines within enterprise environments4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse PlatformStrong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practicesExperience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … intelligence solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologiesBuild, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta LakeDevelop governed datasets and data products that support reporting, analytics, and artificial intelligence initiativesPartner with business stakeholders ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

mindset and focus on delivering business impact. What will set you apart? Experience with distributed computing and data processing frameworks such as Ray, Dask, Apache Spark, Apache Beam, or similar technologies. Experience with event-driven architectures and messaging platforms such as Kafka, RabbitMQ, or similar technologies. Experience ...

hands-on Senior Databricks Solutions Architect to support a complex, high-performance data platform within a global organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements … Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously ...

DevOps knowledge, including Linux/Unix, OpenShift/Kubernetes, and Ansible. Experience building and supporting reactive, event-sourced systems. Hands-on experience with Apache Spark and Apache Flink. Strong understanding of Apache Envoy, and REST API design. Experience working with Protobuf, Avro, and Parquet data formats. ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data products ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding ...

evolving platform stack — from data ingestion and event processing to REST services and internal toolingArchitect and scale streaming and batch processing systems (Flink, Spark, Kafka) that power real-time and analytical workloads. Take responsibility for all aspects of software engineering: design, implementation, QA, and maintenance, with a CI/ ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

Data Warehousing concepts. Enterprise ETL tools such as Informatica, Talend, Datastage or Alteryx. Project experience using the any of the following technologies: Hadoop, Spark, Scala, Oracle, Pega, Salesforce. Cross and multi-platform experience. Financial services sector experience. Line management experience. Team building and leading. Mentoring/staff development experience.#TalanUK ...

delivering solutions that have created measurable business value.Strong experience using Python or Scala.Hands-on knowledge of Big Data and streaming technologies such as Spark and Kafka.Experience designing data models, ETL and ELT pipelines, data warehouses and data lakes.Practical experience with Google Cloud tools such as BigQuery, Dataflow, Dataproc ...

solutions that have created measurable business value. Strong experience using Python or Scala. Hands-on knowledge of Big Data and streaming technologies such as Spark and Kafka. Experience designing data models, ETL and ELT pipelines, data warehouses and data lakes. Practical experience with Google Cloud tools such as BigQuery ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache IcebergProficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)Hands-on experience with CI/… governance, security practices and enterprise integration controlsBackground in financial trading data or related domainsOfficial Databricks certifications such as Certified Data Engineer Professional or Apache Spark DeveloperWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness cover Private medical insurance and dental ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills ...

value. Excellent analytical, problem-solving, and communication skills. Proven experience leading data modernisation initiatives across multiple sectors. Technical Expertise Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog. Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation). Hands ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

experience with one or more cloud data ecosystems: Azure (Data Factory, Synapse, Fabric, ADLS, Databricks)AWS (Glue, Redshift, EMR)GCP (BigQuery, Dataflow)Databricks (Spark/PySpark)Snowflake (SQL, Snowpark, performance tuning)Desirable ExperienceStrong SQL and/or Python capability, and familiarity with tools such as dbt, Git, CI/ ...

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … experience in Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...