126 to 150 of 254 Apache Spark Jobs in London

Hands-on experience with ETL/ELT pipelines, data engineering, and large-scale data processing. Experience with containerization (Docker) and scalable data systems (e.g., Spark, Kafka). Strong leadership presence, excellent communication, strategic thinking, and empathy with a hands-on execution mindset. Desirable Requirements: Experience with AWS SageMaker ...

such as Pandas, NumPy and Scikit-learn A solid grounding in statistical methodologies, along with data extraction, manipulation and feature engineering techniques Experience with Spark, agile delivery methods and CI/CD practices Familiarity with DevOps and MLOps tools and practices, such as Docker, Terraform and MLFlow Confidence influencing ...

batch processing, and scalable enterprise workloads, integrating with clients' incumbent technology estates. Implement hybrid‐cloud and multi‐cloud strategies using modern data infrastructure (Kafka, Spark, data lakes), MLOps practices, and deployment platforms. Ensure compliance with data privacy regulations (including UK GDPR) and responsible AI standards, incorporating governance, observability ...

agentic systems. Experience designing external‐facing APIs, including versioning, backwards compatibility, and OpenAPI/REST standards. Familiarity with data infrastructure such as Airflow, Spark, Iceberg, or Kafka. Experience with energy markets. Modo Energy is an equal opportunity employer. Employment decisions are made on the basis of qualifications, merit ...

agentic systems. Experience designing external-facing APIs, including versioning, backwards compatibility, and OpenAPI/REST standards. Familiarity with data infrastructure such as Airflow, Spark, Iceberg, or Kafka. Experience with energy markets. Modo Energy is an equal opportunity employer. Employment decisions are made on the basis of qualifications, merit ...

other teams with AI. By 12 months You’re putting AI to work in product discovery, building rapid prototypes that test ideas quickly and spark new ones. Working software you’ve built is landing regularly and being used across OE and by our clients, with a short, safe path ...

investors Databricks Ventures, ServiceNow Ventures, and Workday Ventures participating alongside returning investors Altimeter Capital, Avenir Growth Capital, D1 Capital Partners, K5 Global, NewView Capital, Spark Capital, Sutter Hill Ventures, and XN. This milestone follows Sigma reaching $200M in annual recurring revenue in April 2026, with more than 100% year ...

develop homogenous data repositories to deliver data led decision making. Ingest data from various sources, including on premise SQL databases, REST API’s, Apache Kafka streams, etc. Define and apply cleansing rules to the data to ensure meets quality expectations. Define and model data for its appropriate use including … etc.) Other languages, such as C#, PowerShell IaC, e.g. Terraform, ARM, Bicep, etc. Test driven development (TDD) Streaming technologies, such as Azure Stream Analytics, Spark Structured Streaming, etc. Power BI Engineering experience Certified SCRUM Developer (CSD) Data Governance tool experience e.g. Purview Machine learning and Artificial intelligence ...

Responsibilities Azure Platform Support & Monitoring Support and maintain Azure-based data solutions, including Azure Data Factory pipelines, datasets, linked services, triggers; Azure Databricks (Spark jobs, notebooks, clusters); Azure Machine Learning models and endpoints; Power BI dashboards and dataset refreshes. Monitor and troubleshoot failures in pipelines, jobs, and ML workflows … detailed SOPs, runbooks, knowledge‐base articles, and client documentation. Required Skills and Qualifications Azure Services – Azure Data Factory (pipelines, triggers, parameterization, monitoring); Azure Databricks (Spark, notebooks, job orchestration); Azure Machine Learning (pipelines, model deployment, monitoring); Power BI Service (dataset refreshes, access control, report diagnostics). DevOps & Automation – CI/ ...

Learning, or AI workloads. Experience with MLOps tooling such as MLflow, Kubeflow, Vertex AI, or similar technologies. Knowledge of workflow orchestration platforms such as Apache Airflow. Experience with data processing technologies such as Pandas, Apache Beam, Spark, or similar frameworks. Strong SQL skills and experience with analytical ...

meetups Continuously learn and improve engineering practices What we’re looking for batch processing distributed data processing Experience with technologies such as: Kafka Flink Spark Databricks experience is a plus Experience implementing: Data Lakes Big Data Analytics platforms Cloud-based solutions experience AWS preferred Software Engineering Expertise with: RDBMS … Knowledge of: MLOps Feature Engineering Exposure to data analysis and ML tools such as: scikit-learn pandas NumPy XGBoost LightGBM Familiarity with orchestration tools: Apache Oozie Apache Airflow Why you’ll love it here We want to make Prima a happy and empowering place to work. ...

play an important role in our growth. We want you to: Lead the design and development of scalable data pipelines using Python and Apache Spark. Architect and deliver modern data platforms on Microsoft Azure and Microsoft Fabric. Own end-to-end data engineering delivery, ensuring quality, performance, and reliability. … identify improvements, risks, and opportunities, driving continuous optimisation. This job might be for you if you have: Extensive hands-on experience with Python and Apache Spark in data engineering environments. Strong experience building solutions on Azure (e.g., Data Factory, Synapse, Data Lake). Practical experience with Microsoft Fabric ...

unify data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the creators of Lakehouse, Apache Spark, Delta Lake and MLflow. Benefits Databricks strives to provide comprehensive benefits and perks. For specific details in your region, refer ...

develop pipelines for analytics, ML and business intelligence in a security-conscious environment. The role requires 3+ years in data engineering, strong experience with Spark, Python, Kafka, SQL and Airflow, and a willingness to work hybrid in London or Cheltenham. You will contribute to client engagements and growth ...

lab. You will build and own production data pipelines using SQL and Python, with orchestration tools like Airflow and streaming frameworks such as Spark/Kafka. The role involves collaboration across teams and potential documentation tasks to structure work for frontier AI model training. The project runs 40 hours ...

Databricks. Familiarity with cloud‐based application development (AWS & Azure). Familiarity with one or more ML frameworks and technologies: scikit‐learn, xgboost, TensorFlow, PyTorch, Spark, SageMaker, Vertex AI, Kubeflow, Seldon, Triton. Strong communication skills, able to express ideas clearly and collaborate across teams. #J-18808-Ljbffr ...

understanding of data structures, algorithms, and software design principles Experience working with large-scale data processing, distributed computing, or cloud-based platforms (e.g., Hadoop, Spark, AWS) is highly desirable Familiarity with ESG investing principles and market trends is a plus Excellent problem-solving, analytical, and debugging skills Strong communication ...

heavily on Numpy/Pandas/Scikit-learn) Ability to interpret and write complex SQL queries### Experiencewith big data technologies, such as Hive and Spark Track record of leading the deployment and maintenance of models Hands-on### experiencewith AI-powered tools We strive to create an accessible and inclusive### ...

model training, evaluation and monitoring Experience with hyperparameter tuning at scale and experimentation frameworks Strong technical experience with SQL and Python; significant experience with Spark for large-scale data processing Excellent interpersonal skills with demonstrable collaborative leadership capabilities (coaching, mentoring, guiding, delivery ownership and stakeholder engagement) and strong standards ...

were reached. Desirable Skills and Experience Exposure to Microsoft Fabric or Databricks through coursework, projects, internships or early‐career work; familiarity with scikit‐learn, Spark, dbt, Power BI or comparable tools is useful. NLP, information extraction, embeddings, document analysis, network or graph analysis, or time‐series project experience. ...

delivered dataset, model or capability, including deciding what was good enough. Experience with a modern data stack comparable to ours: a cloud data lake, Spark or Databricks, and AWS. Clear communication with technical and non technical stakeholders, and the ability to cascade direction to a team so that each ...

cost optimisation principles, with the ability to apply this knowledge to multi-cloud and cloud-agnostic solution designs at enterprise scale Knowledge of major Apache spark data platform services and their associated architectures Leadership Essential Evidence of providing clear direction through communicating the strategic vision, adopting a corporate ...

enabled features in a fast-growing startup environment in central London. You'll own data architecture from ingestion to analytics, work with Python, Kafka, Spark, and cloud infrastructure (GCP), and collaborate with BackEnd and AI teams to deliver a robust data platform that scales with user demand. #J ...

bronze/silver/gold layers, and partner with analytics, ML, and trading teams to deliver trusted data assets. You will work with dbt, Spark, Airflow, Kafka, and streaming technologies to ensure data quality, governance, and efficient storage. #J-18808-Ljbffr ...

lead the design, development, and optimization of our data infrastructure in a hybrid role based in Hounslow. You will build scalable pipelines with Spark, Hadoop, Hive, and AWS, maintain data warehouses on Oracle/SQL Server, and drive data quality and security. You will leverage Informatica and scripting ...