51 to 75 of 769 Apache Spark Jobs in London

code to ensure high availability and accessibility Requirements Minimum of 8 years of experience in data engineering At least 5 years of hands-on experience with Azure data services (Apache Spark, Azure Data Factory, Synapse Analytics, RDBMS such as SQL Server) Proven leadership and management experience in data engineering teams Proficiency in PySpark, Python (with Pandas), T-SQL … Ability to manage multiple projects and meet deadlines Azure certifications such as Microsoft Certified: Azure Data Engineer Associate or Azure Solutions Architect Nice to Have Experience with Scala for Apache Spark Knowledge of other cloud platforms like AWS or GCP Our Benefits Include Group pension plan, life assurance, income protection, and critical illness cover Private medical insurance and More ❯

tools, and statistical packages. Strong analytical, problem-solving, and critical thinking skills. 8.Experience with social media analytics and understanding of user behaviour. 9.Familiarity with big data technologies, such as Apache Hadoop, Apache Spark, or Apache Kafka. 10.Knowledge of AWS machine learning services, such as Amazon SageMaker and Amazon Comprehend. 11.Experience with data governance and security best More ❯

the data field. Technical Skills Required ·Hands-on software development experience with Python and experience with modern software development and release engineering practices (, CI/CD). ·Experience with Apache Spark or any other distributed data programming frameworks. ·Comfortable writing efficient SQL and debugging on cloud warehouses like Databricks SQL or Snowflake. ·Experience with cloud infrastructure like AWS … Skills ·Hands-on development experience in an airline, e-commerce or retail industry ·Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark Streaming, Kafka, DLT or Beam. ·Experience implementing end-to-end monitoring, quality checks, lineage tracking and automated alerts to ensure reliable and trustworthy data across the platform. ·Experience of More ❯

Experience leading a small team of data engineers. Extensive knowledge as a Data Engineer. Proven success in designing and building data products on Databricks, Snowflake, GCP Big Data, Hadoop, Spark, etc. Excellent problem-solving, analytical, and troubleshooting skills. Strong communication and team collaboration abilities. Programming skills in Python (PySpark preferred), Scala, or SQL. Experience designing and implementing enterprise-level … knowledge of ETL processes. Ability to write production-grade, automated testing code. Experience deploying via CI/CD platforms like Github Actions or Jenkins. Proficiency with distributed frameworks like Apache Spark. Experience with cloud platforms (AWS, Azure, GCP) and services (S3, Redshift, BigQuery). Knowledge of data modelling, database systems, and SQL optimisation. Other key criteria Knowledge of UK More ❯

Python, or C# with Spring Boot or .NET Core. Data Platforms: Warehouses: Snowflake, Google BigQuery, or Amazon Redshift. Analytics: Tableau, Power BI, or Looker for client reporting. Big Data: Apache Spark or Hadoop for large-scale processing. AI/ML: TensorFlow or Databricks for predictive analytics. Integration Technologies: API Management: Apigee, AWS API Gateway, or MuleSoft. Middleware: Red More ❯

on platforms such as AWS, Azure, GCP, and Snowflake. Understanding of cloud platform infrastructure and its impact on data architecture. A solid understanding of big data technologies such as Apache Spark, and knowledge of Hadoop ecosystems. Knowledge of programming languages such as Python, R, or Java is beneficial. Exposure to ETL/ELT processes, SQL, NoSQL databases is More ❯

listed below. AI techniques (supervised and unsupervised machine learning, deep learning, graph data analytics, statistical analysis, time series, geospatial analysis, NLP, sentiment analysis, pattern detection, etc.) Python, R, or Spark for data insights Data Bricks/Data QISQL for data access and processing (PostgreSQL preferred, but general SQL knowledge is important) Latest Data Science platforms (e.g., Databricks, Dataiku, AzureML … SageMaker) and frameworks (e.g., TensorFlow, MXNet, scikit-learn) Software engineering practices (coding standards, unit testing, version control, code review) Hadoop distributions (Cloudera, Hortonworks), NoSQL databases (Neo4j, Elastic), streaming technologies (Spark Streaming) Data manipulation and wrangling techniques Development and deployment technologies (virtualisation, CI tools like Jenkins, configuration management with Ansible, containerisation with Docker, Kubernetes) Data visualization skills (JavaScript preferred) Experience More ❯

SQL with atleast 5 years of experience • Working experience in Palantir Foundry platform is must • Experience designing and implementing data analytics solutions on enterprise data platforms and distributed computing (Spark/Hive/Hadoop preferred). • Proven track record of understanding and transforming customer requirements into a best-fit design and architecture. • Demonstrated experience in end-to-end data … management, data modelling, and data transformation for analytical use cases. • Proficient in SQL (Spark SQL preferred). • Experience with JavaScript/HTML/CSS a plus. Experience working in a Cloud environment such as Azure or AWS is a plus. • Experience with Scrum/Agile development methodologies. • At least 7 years of experience working with large scale software systems. More ❯

independently and as part of a team. Preferred Qualifications: Master's degree in Computer Science, Data Science, or a related field. Experience with big data technologies such as Hadoop, Spark, or Kafka. Experience with data visualization tools such as Power BI, Tableau, or Qlik. Certifications in Azure data and AI technologies. Benefits Salary: We offer a competitive, market-aligned More ❯

Redshift, EMR, Glue) Familiarity with programming languages such as Python or Java Understanding of data warehousing concepts and data modeling techniques Experience working with big data technologies (e.g., Hadoop, Spark) is an advantage Excellent problem-solving and analytical skills Strong communication and collaboration skills Benefits Enhanced leave - 38 days inclusive of 8 UK Public Holidays Private Health Care including More ❯

You have worked in client delivery across a range of projects, including data analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant More ❯

programming languages such as Python and R, and ML libraries (TensorFlow, PyTorch, scikit-learn). Hands-on experience with cloud platforms (Azure ML) and big data ecosystems (e.g., Hadoop, Spark). Strong understanding of CI/CD pipelines, DevOps practices, and infrastructure automation. Familiarity with database systems (SQL Server, Snowflake) and API integrations. Strong skills in ETL processes, data More ❯

machine learning frameworks (e.g., scikit-learn, TensorFlow, XGBoost, PyTorch). Strong foundation in statistics, probability, and hypothesis testing. Experience with cloud platforms (AWS, GCP, Azure) and big data tools (Spark, Hive, Databricks, etc.) is a plus. Excellent communication and storytelling skills with the ability to explain complex concepts to non-technical stakeholders. Proven track record of delivering impactful data More ❯

AI/ML platforms or other advanced analytics infrastructure. Familiarity with infrastructure-as-code (IaC) tools such as Terraform or CloudFormation. Experience with modern data engineering technologies (e.g., Kafka, Spark, Flink, etc.). Why join YouLend? Award-Winning Workplace: YouLend has been recognised as one of the "Best Places to Work 2024" by the Sunday Times for being a More ❯

desired. Proven experience and a passion for developing and operating data-oriented solutions using Python, Airflow/Composer, Kafka, Snowflake, BigQuery, and a mix of data platforms such as Spark, AWS Athena, Postgres and Redis. Excellent SQL development, query optimization and data pipeline development skills required. Strong experience using public cloud platforms including AWS and GCP is required; experience More ❯

modelling tools, data warehousing, ETL processes, and data integration techniques. · Experience with at least one cloud data platform (e.g. AWS, Azure, Google Cloud) and big data technologies (e.g., Hadoop, Spark). · Strong knowledge of data workflow solutions like Azure Data Factory, Apache NiFi, Apache Airflow etc · Good knowledge of stream and batch processing solutions like Apache Flink, Apache Kafka/· Good knowledge of log management, monitoring, and analytics solutions like Splunk, Elastic Stack, New Relic etc Given that this is just a short snapshot of the role we encourage you to apply even if you don't meet all the requirements listed above. We are looking for individuals who strive to make an impact More ❯

with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog). Familiarity with big data technologies like Apache Spark, Hadoop, or similar. ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores Leadership & Strategy: Lead Data Engineering team More ❯

a proven track record of building and managing real-time data pipelines across a track record of multiple initiatives. Expertise in developing data backbones using distributed streaming platforms (Kafka, Spark Streaming, Flink, etc.). Experience working with cloud platforms such as AWS, GCP, or Azure for real-time data ingestion and storage. Programming skills in Python, Java, Scala, or … a similar language. Proficiency in database technologies (SQL, NoSQL, time-series databases) and data modelling. Strong understanding of data pipeline orchestration tools (e.g., Apache Airflow, Kubernetes). You thrive when working as part of a team. Comfortable in a fast-paced environment. Have excellent written and verbal English skills. Last but not least, you'll have no ego! What More ❯

re just getting started. How You'll Make an Impact: As a Senior Data Engineer , you'll design and implement large-scale distributed data processing systems using technologies like Apache Hadoop, Spark, and Flink. You'll build robust data pipelines and infrastructure that transform complex data into actionable insights, ensuring scalability and fault-tolerance across our platform that … that shares knowledge openly, helps each other unblock challenges, and values collective wins over individual credit. Access to Quality Tooling & Infrastructure: Having the right modern tools (e.g., Airflow, dbt, Spark, Flink, cloud platforms) and the ability to improve them when needed sets the foundation for effective work. Supportive Leadership & Growth Mindset: Managers who advocate for your growth, give regular More ❯

data ecosystem (e.g., Pandas, NumPy) and deep expertise in SQL for building robust data extraction, transformation, and analysis pipelines. Hands-on experience with big data processing frameworks such as Apache Spark, Databricks, or Snowflake, with a focus on scalability and performance optimization Familiarity with graph databases (e.g., Neo4j, Memgraph) or search platforms (e.g., Elasticsearch, OpenSearch) to support complex More ❯

data ecosystem (e.g., Pandas, NumPy) and deep expertise in SQL for building robust data extraction, transformation, and analysis pipelines. Hands-on experience with big data processing frameworks such as Apache Spark, Databricks, or Snowflake, with a focus on scalability and performance optimization Familiarity with graph databases (e.g., Neo4j, Memgraph) or search platforms (e.g., Elasticsearch, OpenSearch) to support complex More ❯

on platforms such as AWS, Azure, GCP, and Snowflake. Understanding of cloud platform infrastructure and its impact on data architecture. A solid understanding of big data technologies such as Apache Spark, and knowledge of Hadoop ecosystems. Knowledge of programming languages such as Python, R, or Java is beneficial. Exposure to ETL/ELT processes, SQL, NoSQL databases is More ❯

proficiency, with experience in MS SQL Server or PostgreSQL Familiarity with platforms like Databricks and Snowflake for data engineering and analytics Experience working with Big Data technologies (e.g., Hadoop, Apache Spark) Familiarity with NoSQL databases (e.g., columnar or graph databases like Cassandra, Neo4j) Research experience with peer-reviewed publications Certifications in cloud-based machine learning services (AWS, Azure More ❯

for you. The impact you'll make: Design and develop reusable Python packages (pip/conda) to productionize data science solutions. Process big data at scale using Hadoop/Spark and optimize workflows with Airflow/orchestration tools. Build scalable applications and REST/RPC APIs (Flask/FastAPI/gRPC) for global products. Advocate for engineering best practices … including CI/CD, DevOps, and containerization (Docker/Kubernetes). Mentor junior engineers and lead initiatives to enhance research tooling and dashboards. Languages: Python, SQL Tools: Spark, Hadoop, Airflow, Docker, FastAPI/Flask Cloud: AWS, CI/CD pipelines (Jenkins, Git) What you'll bring to us: 7+ years in data engineering/science, with expertise in Python … Spark, and SQL. Proven experience with big data processing and scalable system design. Strong knowledge of CI/CD, Agile frameworks, and cloud platforms (AWS). Bonus: Familiarity with ML techniques (regression, clustering) and retail data. Location: Hybrid model at the office located in Hammersmith Frequency: The employee must be available for up to 20-25% office presence More ❯

and monitoring systems. Skills/Abilities/Knowledge Proficiency in data modeling and database management. Strong programming skills in Python and SQL. Knowledge of big data technologies like Hadoop, Spark, and NoSQL databases. Deep experience with ETL processes and data pipeline development. Strong understanding of data warehousing concepts and best practices. Experience with cloud platforms such as AWS and … Science or Engineering Languages Excellent command of English. French and Spanish language skills are a bonus. Tools and Applications Programming languages and tools: Python, SQL. Big data technologies: Hadoop, Spark, NoSQL databases. ETL and data pipeline tools: AWS Glue, Airflow. Cloud platforms: AWS, Azure. Data visualization tools and data modeling software. Version control systems and collaborative development platforms. Our More ❯