151 to 175 of 770 Apache Spark Jobs in London

agile environment to deliver data solutions that support key firm initiatives. Build scalable and efficient batch and streaming data workflows within the Azure ecosystem. Apply distributed processing techniques using Apache Spark to handle large datasets effectively. Help drive improvements in data quality, implementing validation, cleansing, and monitoring frameworks. Contribute to the firm’s efforts around data security, governance More ❯

agile environment to deliver data solutions that support key firm initiatives. Build scalable and efficient batch and streaming data workflows within the Azure ecosystem. Apply distributed processing techniques using Apache Spark to handle large datasets effectively. Help drive improvements in data quality, implementing validation, cleansing, and monitoring frameworks. Contribute to the firm’s efforts around data security, governance More ❯

agile environment to deliver data solutions that support key firm initiatives. Build scalable and efficient batch and streaming data workflows within the Azure ecosystem. Apply distributed processing techniques using Apache Spark to handle large datasets effectively. Help drive improvements in data quality, implementing validation, cleansing, and monitoring frameworks. Contribute to the firm’s efforts around data security, governance More ❯

in Neo4j such as fraud detection, knowledge graphs, and network analysis. Optimize graph database performance, ensure query scalability, and maintain system efficiency. Manage ingestion of large-scale datasets using Apache Beam, Spark, or Kafka into GCP environments. Implement metadata management, security, and data governance using Data Catalog and IAM. Collaborate with cross-functional teams and clients across diverse More ❯

and SQL for data pipelines Experience with modern cloud data warehouses (like AWS Redshift, GCP BigQuery, Azure Synapse or Snowflake) Strong communication skills and fluency in English Experience with Apache Spark (in both batch and streaming) Experience with a job orchestrator (Airflow, Google Cloud Composer, Flyte, Prefect, Dagster) Hands-on experience with AWS Experience with dbt *Typeform drives More ❯

skills (ideally Python), focusing on testable, maintainable code. Expertise in cloud services (ideally AWS and Databricks), emphasizing secure, scalable architectures. Experience with large-scale streaming data systems (e.g., Kafka, Spark Streaming), especially on Databricks. Proficiency with low-latency time-series databases (e.g., Apache Druid). Proven leadership in building and deploying high-availability, distributed data systems. Understanding of More ❯

Azure Machine Learning or GCP Cloud ML Engine, Azure Data Lake, Azure Databricks or GCP Cloud Dataproc. Familiarity with big data technologies and distributed computing frameworks, such as Hadoop, Spark, or Apache Flink. Experience scaling an “API-Ecosystem”, designing, and implementing “API-First” integration patterns. Experience working with authentication and authorization protocols/patterns. Other Information: Occasional travel More ❯

could deploy infrastructure into different environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data Meshes) and the different use More ❯

at Zodiac Maritime while working with cutting-edge cloud technologies. Key responsibilities and primary deliverables Design, develop, and optimize end-to-end data pipelines (batch & streaming) using Azure Databricks, Spark, and Delta Lake. Implement Medallion Architecture to structure raw, enriched, and curated data layers efficiently. Build scalable ETL/ELT processes with Azure Data Factory and PySpark. Work with … reliability across pipelines. Collaborate with analysts to validate and refine datasets for reporting. Apply DevOps & CI/CD best practices (Git, Azure DevOps) for automated testing and deployment. Optimize Spark jobs, Delta Lake tables, and SQL queries for performance and cost efficiency. Troubleshoot and resolve data pipeline issues proactively. Partner with Data Architects, Analysts, and Business Teams to deliver More ❯

big data challenges, this role is for you. You will work with billions of events daily, managing petabyte-scale data on Redshift and S3, and develop data pipelines using Spark/Scala EMR, SQL ETL, Airflow, and Java services. We seek a talented, enthusiastic, and detail-oriented Data Engineer to design, analyze, model, and operate big data pipelines. You … modeling, warehousing, and ETL pipelines Proficiency in SQL Experience with scripting languages like Python or KornShell Unix experience Troubleshooting data and infrastructure issues Preferred Qualifications Experience with Hadoop, Hive, Spark, EMR Experience with ETL tools like Informatica, ODI, SSIS, BODI, DataStage Knowledge of distributed storage and computing systems Experience with reporting and analytics platforms We promote an inclusive culture More ❯

relational and NoSQL databases. Experience with data modelling. General understanding of data architectures and event-driven architectures. Proficient in SQL. Familiarity with one scripting language, preferably Python. Experience with Apache Airflow & Apache Spark. Solid understanding of cloud data services: AWS services such as S3, Athena, EC2, RedShift, EMR (Elastic MapReduce), EKS, RDS (Relational Database Services) and Lambda. Nice More ❯

tools, technologies, and best practices Experience with Azure SQL Database, Cosmos DB, NoSQL, MongoDB Experience with Agile, DevOps methodologies Awareness and knowledge of ELT/ETL, DWH, APIs (RESTful), Spark APIs, FTP protocols, SSL, SFTP, PKI (Public Key Infrastructure) and Integration testing Knowledge of Python, SQL, SSIS, and Spark languages. Demonstrative ability to develop complex SQL queries and More ❯

Mathematics, Finance, etc. Proficiency in Python, SQL , and one or more: R, Java, Scala Experience with relational/NoSQL databases (e.g., PostgreSQL, MongoDB) Familiarity with big data tools (Hadoop, Spark, Kafka), cloud platforms (Azure, AWS, GCP), and workflow tools (Airflow, Luigi) Bonus: experience with BI tools , API integrations , and graph databases Why Join Us? Work with large-scale, high More ❯

Mathematics, Finance, etc. Proficiency in Python, SQL , and one or more: R, Java, Scala Experience with relational/NoSQL databases (e.g., PostgreSQL, MongoDB) Familiarity with big data tools (Hadoop, Spark, Kafka), cloud platforms (Azure, AWS, GCP), and workflow tools (Airflow, Luigi) Bonus: experience with BI tools , API integrations , and graph databases Why Join Us? Work with large-scale, high More ❯

Mathematics, Finance, etc. Proficiency in Python, SQL , and one or more: R, Java, Scala Experience with relational/NoSQL databases (e.g., PostgreSQL, MongoDB) Familiarity with big data tools (Hadoop, Spark, Kafka), cloud platforms (Azure, AWS, GCP), and workflow tools (Airflow, Luigi) Bonus: experience with BI tools , API integrations , and graph databases Why Join Us? Work with large-scale, high More ❯

databases . A solid grasp of data governance, data modeling, and business intelligence best practices. Knowledge of Agile, DevOps, Git, APIs, microservices, and data pipeline development . Familiarity with Spark, Kafka, or Snowflake is a plus. Desirable Certifications: Microsoft Certified: Fabric Analytics Engineer Associate Why Join Us? Competitive salary up to £70,000 per year Opportunities for growth, training More ❯

in their work. Possess advanced knowledge of programming languages (Scala, Python, Java, C#) and understanding of domain modelling and application programming. Have experience with data management platforms (SQL, NoSQL, Spark/Databricks). Be familiar with modern engineering tools (Git, CI/CD), cloud technologies (Azure, AWS), and Infrastructure as Code (Terraform, Pulumi). Experience with various frameworks (React More ❯

AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions - Experience building large-scale, high-throughput, 24x7 data systems - Experience with big data technologies such as: Hadoop, Hive, Spark, EMR - Experience providing technical leadership and mentoring other engineers for best practices on data engineering Our inclusive culture empowers Amazonians to deliver the best results for our customers. If More ❯

Science, Computer Science, or a related field. 5+ years of experience in data engineering and data quality. Strong proficiency in Python/Java, SQL, and data processing frameworks including Apache Spark. Knowledge of machine learning and its data requirements. Attention to detail and a strong commitment to data integrity. Excellent problem-solving skills and ability to work in a More ❯

or Scala) Experience with cloud platforms (AWS, GCP, or Azure) Experience with data warehousing and lake architectures ETL/ELT pipeline development SQL and NoSQL databases Distributed computing frameworks (Spark, Kinesis, etc.) Software development best practices including CI/CD, TDD, and version control Strong understanding of data modelling and system architecture Excellent problem-solving and analytical skills Experience More ❯

derivatives. Technical Skills Proficiency in Python and SQL for data manipulation and model development. Experience in working with large-scale datasets and utilizing big data technologies (e.g., Azure, Databricks, Spark) Familiarity with data visualization tools such as Power BI, or matplotlib Strong knowledge of predictive modeling, machine learning techniques, and statistical analysis. Soft Skills Analytical mindset with a passion More ❯

in using advanced features of cloud platforms (AWS, Azure, Google Cloud) such as machine learning services and automated data pipeline tools. Familiarity with big data frameworks like Hadoop or Spark is beneficial. Skills in advanced data visualization tools and software beyond basic reporting—such as Tableau, Power BI, or even more sophisticated interactive web visualization frameworks like D3.js. Benefits More ❯

Bachelor's or Master's degree in Computer Science, Engineering, or relevant experience hands-on with data engineering Strong hands-on knowledge of data platforms and tools, including Databricks, Spark, and SQL Experience designing and implementing data pipelines and ETL processes Good knowledge of ML ops principles and best practices to deploy, monitor and maintain machine learning models in More ❯

programming language (Python, Java, or Scala) Extensive experience with cloud platforms (AWS, GCP, or Azure) Experience with: Data warehousing and lake architectures SQL and NoSQL databases Distributed computing frameworks (Spark, Kinesis etc) Software development best practices including CI/CD, TDD and version control. Strong understanding of data modelling and system architecture Excellent problem-solving and analytical skills Whilst More ❯

infrastructure automation, and the ML lifecycle in production environments. • Solid experience with cloud-native architecture (AWS, GCP or Azure), containerisation (e.g. Docker, Kubernetes), and big data processing frameworks (e.g. Spark, BigQuery). • Expertise in Python and SQL, with working knowledge of ML frameworks such as TensorFlow, PyTorch, or Scikit-learn. • Demonstrated ability to lead through ambiguity and complexity, making More ❯