26 to 50 of 124 Apache Spark Jobs in England

platforms for performance and reliability Collaborate with Agile teams to deliver secure, high-quality solutions Write clean, tested, maintainable code Required Skills: Experience with Apache Spark, Hadoop or similar big data technologies AWS Apache NiFi Python or Java Experience building data pipelines and ETL processes Desirable ...

Public Services. Experience Level Required in Years: 10 - 15 years Mandatory Required Skills Hands-on experience with Quantexa platform/Decision Intelligence Strong in Apache Spark/PySpark Proficiency in Python/Scala Experience in ETL, Data Engineering, Data Warehousing Knowledge of entity resolution, graph analytics, link analysis … ecosystem – Hive/HDFS) Detailed Job Description Design and develop solutions using Quantexa Decision Intelligence Platform Build data ingestion, transformation, and ETL pipelines using Spark/PySpark Work on entity resolution, network analytics, and graph-based data models Integrate multiple structured/unstructured data sources for analytics use cases ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

What we’re looking for: Strong Python development and software engineering experience Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments ...

workflows. Key Requirements Passion for building scalable, resilient cloud data platforms. Strong experience with Databricks or Snowflake on AWS. Proven Python skills, including Spark and Airflow expertise. Advanced SQL skills and end-to-end data modelling experience. Experience building batch and real-time data integrations. Hands-on CICD skills ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

with Azure Cosmos DB, including query optimization, throughput management, and performance tuning. Thorough understanding of Cosmos DB Change Data Feed (CDF) and integration with Spark-based solutions. Experience working with Cosmos DB SDKs in technologies such as .NET, Java, Python, or Node.js. Strong understanding of distributed computing concepts … Apache Spark architecture. Hands-on experience developing data pipelines using PySpark. Experience with Pytest and automated testing frameworks. Strong understanding of partitioning, indexing strategies, and consistency models within Cosmos DB. Experience building scalable applications and working with concurrency patterns. Experience with Git, version control systems, and CI/ ...

reviews and continuous improvement Skills & Experience * Commercial experience in Data Engineering * Strong Python or Java development skills * Experience with AWS cloud services * Knowledge of Apache Spark, Hadoop, NiFi or similar big data technologies * Experience building ETL/ELT pipelines and processing large datasets * Understanding of software engineering best ...

with the ability to deliver incremental data modelling solutions in collaboration with engineering teams. Technical Environment (Desirable) Microsoft Fabric Azure Data Factory Python PySpark Apache Spark Desirable Skills Azure DevOps Contract Details Sector: Banking/Financial Services Location: London – Office based 4 days per week Contract Type ...

Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial ...

multidisciplinary team to support and enhance over 70 active BI products. What You'll Bring Essential Skills & Experience: Core Technologies: Azure Databricks (or similar Spark-based platforms) and Azure Data Factory (ADF). Languages: SQL and Python. Data Platforms: Azure Data Lake and SQL Server RDBMS. ETL Tools ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive ...

Work with container technologies Core Data Engineering Strong programming proficiency in Java or Python. Hands-on experience with at least one of: Kafka, Flink, Spark Solid understanding of stream processing concepts Understanding of software engineering best practices: testing, design patterns, CI/CD, Git. Experience building ETL/ ...

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...

backend roles Strong Python; experience with distributed data systems Relational DB experience (PostgreSQL preferred) NoSQLExposure to vector databases (Qdrant, Milvus, pg vector) Bonus: Spark/Airflow/Kafka, cloud (GCP/AWS/Azure), AI/ML platform experience, early-stage startup bac kground Why Join Build the data ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

Work with container technologies Core Data Engineering Strong programming proficiency in Java or Python. Hands-on experience with at least one of: Kafka, Flink, Spark Solid understanding of stream processing concepts Understanding of software engineering best practices: testing, design patterns, CI/CD, Git. Experience building ETL/ ...

scale datasets Preferred Qualifications Knowledge or interestin investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka Goldman Sachs Highlights At Goldman Sachs, we commit our people, capital ...

datasets Preferred Qualifications Knowledge or interest in investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka ABOUT GOLDMAN SACHS At Goldman Sachs, we commit our people, capital ...

Experience designing and enforcing data contracts and quality SLAs in distributed data ecosystems. Proficiency in Python or Java, and experience building pipelines with Airflow, Spark, or Kafka. Strong understanding of data governance principles, privacy regulations (such as the GDPR, CCPA), and secure data handling practices. Familiarity with metadata management ...

+ ETL/ELT experience Knowledge of Python/T-SQL Exposure to DBT Exposure to cloud platforms (Azure preferred) Nice to have: Databricks, Spark, CI/CD, big data This is a great opportunity to join a high-impact data team in a global organisation and shape ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...