19 of 19 Apache Spark Jobs in the City of London

skills. Ability to think creatively and propose innovative solutions. To qualify for this role, you must have: Proven experience and knowledge with PySpark and Apache Spark including the fundamentals of how it works. Experience with cloud platforms (e.g., AWS, Azure, GCP) and data lake architectures. Strong experience ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

Skills MUST HAVE: Strong hands‐on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

role 7Dxperts is looking for a Senior Data Engineer to join our UK team. Key Responsibilities Build scalable pipelines using Databricks, Spark and PySpark. Develop ETL/ELT and data transformation workflows using Python and SQL. Create and maintain Databricks notebooks, jobs, workflows and clusters. Work with Delta Lake ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

classification techniques, and algorithms Fluency in a programming language (Python, C,C++, Java, SQL) Familiarity with Big Data frameworks and visualization tools (Cassandra, Hadoop, Spark, Tableau) Desirable Skills & Experience Familiarity with MLOps, AI governance, and responsible AI practices. Experience with modern AI frameworks and tooling such as LangChain, Hugging ...

optimisation and administration Strong programming experience with Python, Go and/or SQL Experience using dbt for data modelling and transformation Experience working with Apache Airflow or similar orchestration tooling Proven experience designing and building production systems on GCP and/or AWS Strong understanding of event-driven architectures … distributed systems Experience deploying and supporting batch and real-time machine learning workloads Experience working with distributed processing technologies such as Spark or Apache Beam Strong understanding of data quality principles and engineering best practices Experience optimising database performance, troubleshooting issues and maintaining production data systems Ability ...

data science and machine learning libraries such as scikit-learn, pandas, PyTorch and/or TensorFlow . Practical experience with Databricks , including MLflow and Spark-based data processing. Hands-on knowledge of Generative AI and LLMs , including prompt engineering, RAG architectures and vector search. Strong understanding of MLOps , including ...

machine learning and data science frameworks. Hands-on experience with large language models, generative AI, prompt engineering, and RAG architectures. Experience with Databricks, MLflow, Spark, and cloud-based AI platforms (Azure preferred). Strong understanding of MLOps, software engineering best practices, and production deployment pipelines. Experience leading technical teams ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

Hands-on experience with ETL/ELT pipelines, data engineering, and large-scale data processing Experience with containerisation (Docker) and scalable data systems (e.g. Spark, Kafka) Nice-to-Have Experience with AWS SageMaker or similar managed ML platforms Background in safety-critical or regulated industries (aerospace, aviation, or similar ...

such as Government, Public Sector, Security, Justice, Defence, Healthcare or other similarly challenging environments . Experience with large-scale data technologies such as SQL, Spark or Hadoop would also be beneficial. About You You will be a technically credible leader who is still comfortable getting into the detail. ...

Responsibilities Azure Platform Support & Monitoring Support and maintain Azure-based data solutions, including Azure Data Factory pipelines, datasets, linked services, triggers; Azure Databricks (Spark jobs, notebooks, clusters); Azure Machine Learning models and endpoints; Power BI dashboards and dataset refreshes. Monitor and troubleshoot failures in pipelines, jobs, and ML workflows … detailed SOPs, runbooks, knowledge‐base articles, and client documentation. Required Skills and Qualifications Azure Services – Azure Data Factory (pipelines, triggers, parameterization, monitoring); Azure Databricks (Spark, notebooks, job orchestration); Azure Machine Learning (pipelines, model deployment, monitoring); Power BI Service (dataset refreshes, access control, report diagnostics). DevOps & Automation – CI/ ...

statistics, machine learning, and mathematical modelling. Experience designing evaluation frameworks for AI systems, including model performance and monitoring. Hands-on experience with Databricks, MLflow, Spark, or similar cloud-based data platforms. Knowledge of responsible AI, model governance, and explainability. Strong communication skills with the ability to explain complex technical ...

their techniques * Experience with data science, big data analytics technology stack, analytic development for endpoint and network security, and streaming technologies (e.g., Kafka, Spark Streaming, and Kinesis). Amazon is an equal opportunities employer. We believe passionately that employing a diverse workforce is central to our success. We make ...

leading Data Management consultancy in London seeks a skilled professional experienced in PySpark and Apache Spark for a hands-on role. You will work closely with clients and play a crucial part in developing innovative solutions using cloud technologies like AWS, Azure, and GCP. Excellent communication and ability ...

Desirable Experience with cloud-based risk platforms and pricing engines Azure Kubernetes Service (AKS) Azure Data Lake Microservices and event-driven architectures Kafka, Redis, Spark, HBase, and HDFS Market Risk platforms covering VaR, Stress Testing, and FRTB Risk exposure reconciliation and reporting platforms Enterprise scheduling tools such as Control ...

Azure PaaS components, including Azure DevOps, Vault, and other security and configuration services (AKS experience is beneficial) Integrate and optimize data processing frameworks with Spark, Databricks, or Cloudera platform (Hue, Hive, HDFS) Support ETL pipelines built for master data management, customer profiles, and consent management workflows Ensure environments … release cycles Exposure to Azure Cloud and PaaS services, including Azure DevOps and Vault Familiarity with data engineering and big data ecosystems, including Spark and Cloudera stacks (hue, hive, HDFS) Excellent problem‐solving skills and ability to communicate technical solutions effectively Previous experience working on data‐driven projects ...

workflows, applications, transformations, and reusable components. Support the migration of data assets, ETL pipelines, and workloads from the Cloudera Hadoop platform (HDFS, Hive, Impala, Spark) to AWS Cloud. Design, develop, and optimise data ingestion, transformation, and loading processes into Amazon S3 Data Lake and Amazon Redshift. Develop and maintain … based data integration, transformation, validation, and reconciliation processes. Integrate and support data services including AWS Glue Data Catalog, Athena, Redshift Spectrum, EMR Spark Perform data migration, testing, reconciliation, and unit testing activities to ensure data integrity and successful platform migration. Create and maintain Unix/Linux shell scripts supporting ...