19 of 19 Permanent Apache Spark Jobs in Central London

mindset and focus on delivering business impact. What will set you apart? Experience with distributed computing and data processing frameworks such as Ray, Dask, Apache Spark, Apache Beam, or similar technologies. Experience with event-driven architectures and messaging platforms such as Kafka, RabbitMQ, or similar technologies. Experience ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

modern cloud engineering practices. Wherever you land, you’ll be working with some of the biggest datasets in the UK - using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Cloud-native AI engineering and large-scale data platforms ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … distributed data processing and performance tuning Hands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake and engines such as Spark, Flink, or Trino and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

Engineering Strong experience working with AWS in data engineering environments. Hands‐on experience with AWS Glue for data integration and processing. Strong experience using Apache Airflow for workflow orchestration and pipeline management. Advanced Python development skills. Experience with Apache Spark or similar distributed data processing technologies would ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything- spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

Private cloud platforms (HPE Private AI, Nvidia AI Enterprise, Dell AI Factories) Understanding of data engineering practices (ETL, data lakes, big data tools like Spark). Knowledge of designing scalable, distributed systems. Familiarity with model lifecycle management and MLOps practices. Ability to align AI initiatives with business outcomes ...

Responsibilities Azure Platform Support & Monitoring Support and maintain Azure-based data solutions, including Azure Data Factory pipelines, datasets, linked services, triggers; Azure Databricks (Spark jobs, notebooks, clusters); Azure Machine Learning models and endpoints; Power BI dashboards and dataset refreshes. Monitor and troubleshoot failures in pipelines, jobs, and ML workflows … detailed SOPs, runbooks, knowledge‐base articles, and client documentation. Required Skills and Qualifications Azure Services – Azure Data Factory (pipelines, triggers, parameterization, monitoring); Azure Databricks (Spark, notebooks, job orchestration); Azure Machine Learning (pipelines, model deployment, monitoring); Power BI Service (dataset refreshes, access control, report diagnostics). DevOps & Automation – CI/ ...

understanding of statistics, machine learning methodologies, and model evaluation. Experience delivering AI solutions from proof-of-concept through to production. Exposure to Databricks, MLflow, Spark, or similar cloud-based data platforms. Ability to influence senior stakeholders and lead technical strategy Desirable Experience Experience building multi-agent AI systems ...

essential: Background in financial services or regulated environments. Exposure to financial crime domains such as fraud, AML or sanctions. Experience with Spark or PySpark. Familiarity with AWS cloud services and data platforms. You will be assessed on key capabilities relevant to the role, including risk and controls, change ...

components . Build and maintain platform capabilities supporting batch and real-time data workloads . Develop and operate distributed data services using Kafka, Kinesis, Spark, Flink or equivalent technologies . Build and maintain cloud-native data platform capabilities using AWS and/or Azure . Support data ingestion, processing … Hands-on experience with real-time streaming and event-driven architectures , including Kafka, Kinesis or equivalent. Experience with distributed data processing technologies such as Spark, Flink, Kafka Streams or equivalent. Strong experience with AWS and/or Azure cloud environments. Experience with cloud services such as S3, Glue, Lambda ...

Desirable Experience with cloud-based risk platforms and pricing engines Azure Kubernetes Service (AKS) Azure Data Lake Microservices and event-driven architectures Kafka, Redis, Spark, HBase, and HDFS Market Risk platforms covering VaR, Stress Testing, and FRTB Risk exposure reconciliation and reporting platforms Enterprise scheduling tools such as Control ...

days a week and delivering high-quality, consultative outcomes. Ideal candidates will have strong communication skills and a track record in Big Data technologies (Spark, Hadoop, EMR, MapR) and NoSQL/MPP databases, with a passion for #J-18808-Ljbffr ...

Proven experience in consultative technical sales or professional services with a proven record of success Commercial solution architecture experience in Big Data technologies e.g. Spark, Hadoop, EMR, MapR etc. Strong verbal and written communications skills are a must, as well as the ability to lead effectively across internal ...

Scala), with a track record of building and maintaining reliable, production-grade software at scale. Hands-on experience with data transformation tools such as Spark and DBT, with a focus on building well-tested, maintainable data products is a strong advantage. Confidence in working with Kubernetes-native platforms … deploy and operate data workloads across cloud environments is preferred. Knowledge of Golang would be an advantage. Familiarity with Apache Iceberg table format and schema management via catalogs such as AWS Glue, with an appreciation for data quality and consistency is a strong advantage. Solid grounding in core ...

schema changes Troubleshoot data-quality and pipeline issues Implement data-quality checks, testing and documentation Work with distributed data technologies such as Presto, Spark or Hive Use workflow orchestration tools such as Airflow Participate in code reviews and Git-based development workflows Collaborate with Data Engineers, Analysts and Engineering … 5+ years' experience building production data pipelines Advanced SQL , including performance optimisation on large tables Strong Python for data engineering Experience with Presto, Spark, Hive or similar Experience with Airflow or another workflow scheduler Strong ETL/data pipeline development experience Git and code-review experience Strong understanding ...

workflows, applications, transformations, and reusable components. Support the migration of data assets, ETL pipelines, and workloads from the Cloudera Hadoop platform (HDFS, Hive, Impala, Spark) to AWS Cloud. Design, develop, and optimise data ingestion, transformation, and loading processes into Amazon S3 Data Lake and Amazon Redshift. Develop and maintain … based data integration, transformation, validation, and reconciliation processes. Integrate and support data services including AWS Glue Data Catalog, Athena, Redshift Spectrum, EMR Spark Perform data migration, testing, reconciliation, and unit testing activities to ensure data integrity and successful platform migration. Create and maintain Unix/Linux shell scripts supporting ...