51 to 75 of 186 Apache Spark Jobs in England

learn etc.) Have experience productionising machine learning models Are an expert in at least one of one of : predictive modelling, classification, regression, optimisation or recommendation systems Have experience with Spark Have knowledge of DevOps technologies such as Docker and Terraform and ML Ops practices and platforms like ML Flow Have experience with agile delivery methodologies and CI/CD More ❯

environment (Python, Go, Julia etc.) •Experience with Amazon Web Services (S3, EKS, ECR, EMR, etc.) •Experience with containers and orchestration (e.g. Docker, Kubernetes) •Experience with Big Data processing technologies (Spark, Hadoop, Flink etc) •Experience with interactive notebooks (e.g. JupyterHub, Databricks) •Experience with Git Ops style automation •Experience with ix (e.g, Linux, BSD, etc.) tooling and scripting •Participated in projects More ❯

source systems into our reporting solutions. Pipeline Development: Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data factory and engineering tools like Apache Spark to ensure seamless data flow. Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery processes to maintain data integrity. … in Azure data tooling such as Synapse Analytics, Microsoft Fabric, Azure Data Lake Storage/One Lake, and Azure Data Factory. Understanding of data extraction from vendor REST APIs. Spark/Pyspark or Python skills a bonus or a willingness to develop these skills. Experience with monitoring and failure recovery in data pipelines. Excellent problem-solving skills and attention More ❯

best practices for data security and compliance. Collaborate with stakeholders and external partners. Skills & Experience: Strong experience with AWS data technologies (e.g., S3, Redshift, Lambda). Proficient in Python, Apache Spark, and SQL. Experience in data warehouse design and data migration projects. Cloud data platform development and deployment. Expertise across data warehouse and ETL/ELT development in More ❯

of large-scale distributed data processing. Experience with developing extract-transform-load (ETL). Experience with distributed messaging systems like Kafka and RabbitMQ. Experience with distributed computing frameworks like Apache Spark and Flink. Bonus Points Experience working with AWS or Google Cloud Platform (GCP). Experience in building a data warehouse and data lake. Knowledge of advertising platforms. More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

with Microsoft Azure and Azure SQL Database Proficiency with Docker and containerisation tools Experience working with APIs for data extraction Desirable Skills Familiarity with big data technologies such as Spark and Kafka Experience with machine learning frameworks like TensorFlow or PyTorch Knowledge of data visualisation tools such as Power BI or Tableau Strong understanding of data modelling and database More ❯

require the following experience: Quant Degree such as Maths, Physics, Computer Science, Engineering etc Software Development experience in Python or Scala An understanding of Big Data technologies such as Spark, messaging services like Kafka or RabbitMQ, and workflow management tools like Airflow SQL & NoSQL expertise, ideally including Postgres, Redis, MongoDB etc Experience with AWS, and with tools like Docker More ❯

with Azure Data Factory, Azure Functions, and Synapse Analytics. Proficient in Python and advanced SQL, including query tuning and optimisation. Hands-on experience with big data tools such as Spark, Hadoop, and Kafka. Familiarity with CI/CD pipelines, version control, and deployment automation. Experience using Infrastructure as Code tools like Terraform. Solid understanding of Azure-based networking and More ❯

and delivering end-to-end AI/ML projects. Nice to Have: Exposure to LLMs (Large Language Models), generative AI , or transformer architectures . Experience with data engineering tools (Spark, Airflow, Snowflake). Prior experience in fintech, healthtech, or similar domains is a plus. More ❯

to design and deliver enterprise-scale data warehouses in regulated or complex environments. Expertise in ETL/ELT, and reporting system architectures. Strong technical skills in SQL, Python, PySpark, Apache Spark. Hands-on background as a data engineer or platform engineer - you can design and build. Excellent communication and relationship-building skills across technical and non-technical audiences. Demonstrated More ❯

in cloud environments (e.g. Snowflake, AWS). 6+ years of hands-on technical leadership in building large-scale, distributed data pipelines and reporting tools using big data technologies (e.g. Spark, Kafka, Hadoop), ensuring quality, scalability, and governance. Strong expertise in balancing trade-offs within complex distributed systems, focusing on data quality, performance, reliability, availability, and security. Proficient in software More ❯

in US or UK Preferred Experience: Data orchestration tools (e.g. , Airflow, Prefect)Experience deploying, monitoring, and maintaining ML models in production environments (MLOps)Familiarity with big data technologies ( e.g. , Spark, Hadoop)Background in time-series analysis and forecastingExperience with data governance and security best practicesReal-time data streaming is a plus (Kafka, Beam, Flink)Experience with Kubernetes is a More ❯

monitoring tools such as New Relic or Grafana Understand the use of feature stores and related data technologies for operational machine learning products Are proficient with Python and have Spark knowledge. Have leadership experience either through previous management or mentorship. Have good communication skills. Nice to have Experience deploying LLMs and agent-based systems Our technology stack Python and More ❯

platforms (Strapi or equivalent) Working on Developer Portals or documentation relating to shared development services Experience of Docker and/or Kubernetes an advantage. Working knowledge of DataBricks/Spark (using Python and associated frameworks) Working knowledge of Azure Data Lake and Azure Blob storage What we offer This is a permanent role The team is based from the More ❯

platforms (Strapi or equivalent) Working on Developer Portals or documentation relating to shared development services Experience of Docker and/or Kubernetes an advantage. Working knowledge of DataBricks/Spark (using Python and associated frameworks) Working knowledge of Azure Data Lake and Azure Blob storage What we offer This is a permanent role The team is based from the More ❯

include: 3+ years experience in data engineering roles, delivering integrated data-driven applications Hands-on experience with Microsoft Fabric components (Pipelines, Lakehouse, Warehouses) Proficient in T-SQL and either Apache Spark or Python for data engineering Comfortable working across cloud platforms, with emphasis on Microsoft Azure Familiarity with REST APIs and integrating external data sources into applications More ❯

and streaming data pipelines Azure Purview or equivalent for data governance and lineage tracking Experience with data integration, MDM, governance, and data quality tools . Hands-on experience with Apache Spark, Python, SQL, and Scala for data processing. Strong understanding of Azure networking, security, and IAM , including Azure Private Link, VNETs, Managed Identities, and RBAC . Deep knowledge More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Benefits At Databricks, we strive to provide comprehensive benefits and perks that More ❯

to refine and monitor data collection systems using Scala and Java. Apply sound engineering principles such as test-driven development and modular design. Preferred Background Hands-on experience with Spark and Scala in commercial environments. Familiarity with Java and Python. Exposure to distributed data systems and cloud storage platforms. Experience designing data schemas and analytical databases. Use of AI More ❯

to refine and monitor data collection systems using Scala and Java. Apply sound engineering principles such as test-driven development and modular design. Preferred Background Hands-on experience with Spark and Scala in commercial environments. Familiarity with Java and Python. Exposure to distributed data systems and cloud storage platforms. Experience designing data schemas and analytical databases. Use of AI More ❯

between the data warehouse and other systems. Create deployable data pipelines that are tested and robust using a variety of technologies and techniques depending on the available technologies (Nifi, Spark) Build analytics tools that utilise the data pipeline to provide actionable insights into client requirements, operational efficiency, and other key business performance metrics. Complete onsite client visits and provide More ❯

between the data warehouse and other systems. Create deployable data pipelines that are tested and robust using a variety of technologies and techniques depending on the available technologies (Nifi, Spark) Build analytics tools that utilise the data pipeline to provide actionable insights into client requirements, operational efficiency, and other key business performance metrics. Complete onsite client visits and provide More ❯

experience of utilising strong communication and stakeholder management skills when engaging with customers Significant experience of coding (ideally Python) Experience with big data processing tools such as Hadoop or Spark Cloud experience; GCP specifically in this case, including services such as Cloud Run, Cloud Functions, BigQuery, GCS, Secret Manager, Vertex AI etc. Experience with Terraform Prior experience in a More ❯