one of the biggest financial services organisations in the world. The successful candidate will be responsible for designing, building, and maintaining data pipelines using Apache Spark and Scala. This includes tasks like: Extracting data from various sources (databases, APIs, files) Transforming and cleaning the data Loading the data into … data warehouses or data lakes (e.g., BigQuery, Amazon Redshift) Automating the data pipeline execution using scheduling tools (e.g., Airflow) Work with Big Data technologies: You'll likely work with various Big Data technologies alongside Spark, including: Hadoop Distributed File System (HDFS) for storing large datasets Apache Kafka for … real-time data streaming Apache Hive for data warehousing on top of HDFS Cloud platforms like AWS, Azure, or GCP for deploying and managing your data pipelines Key skills: At least 8+ Years of IT Experience with designing, building, and maintaining data pipelines. At least 4+ Years of experience more »
Python experience Experienced with AWS and services like S3. Experienced with Kafka for data streaming. Familiarity with BI reporting tools Good working experience with Airflow, PySpark and Apache Beam Worked with Java for building data applications – Advantageous Worked within the commodities space – Advantageous Not quite right for you more »
projects and new innovations to support company growth and profitability. Our Tech Stack Python Scala Kotlin Spark Google PubSub Elasticsearch Bigquery, PostgresQL Kubernetes, Docker, Airflow Ke y Responsibilities Designing and implementing scalable data pipelines using tools such as Apache Spark, Google PubSub etc. Optimising data storage and retrieval … Data Infrastructure projects, as well as designing and building data intensive applications and services. Experience with data processing and distributed computing frameworks such as Apache Spark Expert knowledge in one or more of the following languages - Python, Scala, Java, Kotlin Deep knowledge of data modelling, data access, and data more »
/Reinforcement Learning/Statistics applied to Machine Learning Experience with A/B testing Experience with data processing pipeline frameworks (e.g., Luigi or Airflow) Experience working with teams of Data Engineers or Product Managers Experience with functional programming concepts and architecture Keywords: Machine Learning, Data Science, GenAI, Cloud … Python, Recommendation systems, NLP, Scikit-learn, Pandas, Luigi, Airflow, Docker, Git Education Bachelor's or Master's degree in Computer Science, Information Systems, or equivalent Competencies Communicating Effectively Critical Thinking Managing Meetings Prioritizing and Organizing Work Solving Complex Problems Optimizely is committed to a diverse and inclusive workplace. Optimizely more »
Position: JAVA Full Stack Lead Software Engineer Duration: 6 months Number of Positions: 3 Special Working Conditions: 3 days at office (Glasgow) Day Rate: £450 Careers at TCS: It means more TCS is a purpose-led transformation company, built on more »
schemas to support business requirements Develop and maintain data ingestion and processing systems using various tools and technologies, such as SQL, NoSQL, ETL, Luigi, Airflow, Argo, etc. Implement data storage solutions using different types of databases, such as relational, non-relational, or cloud-based. Working collaboratively with the client …/Azure SQL, PostgreSQL) You have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving more »
. Experience with SQL and query design on large, complex datasets. Experience with cloud and big-data tools and frameworks like Databricks/Spark, Airflow, Snowflake, etc. Expertise designing and developing with distributed data processing platforms like Databricks/Spark. Experience using ELT/ETL tools such as DBT … FiveTran, etc. Understanding of Agile Delivery best practice Good knowledge of the relevant technologies e.g. SQL, Oracle, PostgreSQL, Python, ETL pipelines, Airflow, Hadoop, Parquet. Strong problem-solving and analytical abilities. Ability to present solutions and limitations to non-IT business experts ABOUT YOU Integrity, respect, intellectually curious and an more »
Cardiff, Wales, United Kingdom Hybrid / WFH Options
Creditsafe
the above project of redesigning the Creditsafe platform into the cloud space. You will be expected to work with technologies such as Python, Linux, Airflow, AWS DynamoDB, S3, Glue, Athena, Redshift, lambda, API Gateway, Terraform, CI/CD. KEY DUTIES AND RESPONSIBILITIES You will actively contribute to the codebase … and participate in peer reviews. Design and build metadata driven, event based distributed data processing platform using technologies such as Python, Airflow, Redshift, DynamoDB, AWS Glue, S3. As an experienced Engineer, you will play a critical role in the design, development, and deployment of our business-critical system. You more »
and our travelers. Our work spans across a variety of datasets and ML models and across a diverse technology stack ranging from Spark, Sagemaker, Airflow, Databricks, Kubernetes, AWS and much more! What you will do: Work in a cross-functional team of Machine Learning engineers and Machine Learning Science … of machine learning pipelines and machine learning frameworks such as TensorFlow and Pytorch Familiar with cloud services (e.g., AWS) and workflow orchestration tools (e.g., Airflow) Experience working with Agile/Scrum methodologies. Familiar with the e-commerce or travel industry. About Expedia Group Expedia Group (NASDAQ: EXPE) powers travel more »
in computer science, information technology, technology engineering or equivalent Strong interest in the data infrastructure landscape and related topics Experience with technologies such as Apache, Docker, Parquet, Python, Spark, Airbyte and Airflow Detailed understanding of Linux OS’s. Experience with ETL processes, data lakes, data warehouses, relational databases more »
to some SAS development on legacy projects when required. Python or PySpark and SQL will be your bread and butter, with any experience of Airflow being a great bonus. The core skillset: Python/PySpark for building scalable & robust data pipelines Experience with Airflow or other orchestration tools more »
leading marketplace platform is looking for a Principal Data Engineer to take a lead on several aspects of their data platform built in Python, Airflow, Kafka, Sagemaker, AWS and GCP. 💰 Salary: up to £130k + 15% bonus 📍 Location: can be based anywhere in the UK ✅ Must have requirements: Mastery … of Python and SQL Significant experience with orchestration tools, ideally Airflow Strong Data Architecture experience Cloud expertise in either GCP or AWS (ideally in both) Knowledge of MPP systems such as Athena, BigQuery, EMR, Hive, Iceberg Exposure to data streaming technologies such as Kafka or Kinesis more »
financial services or energy trading industry Expertise in Python and its ecosystem of libraries and frameworks for data processing, data analysis and data visualisation Airflow, detailed understanding of architecture including schedulers, executers, operators Cloud Environments, understanding of principles, technologies and services for AWS/Azure Kubernetes EKS/AKS … including high availability Desired experience: Worked with Python 3.9+ Familiar with Python test automation Experience with SQL and Timeseries databases Familiar with Parquet, Arrow, Airflow, Databricks Experience with cloud AWS services, such as S3, EC2, RDS etc Quality engineering best practice and tooling including TDD, BDD This is an more »
Senior AWS Data Engineer - London (Hybrid) - Permanent - £65,000-£70,000 Media Agency, Snowflake, SQL, Python, dbt, Airflow, Marketing *You must be based in London, and have full permanent right to work in the UK to apply for this role* I'm currently working with a leading media agency … ASAP and I will be in touch. Senior AWS Data Engineer - London (Hybrid) - Permanent - £65,000-£70,000 Media Agency, Snowflake, SQL, Python, dbt, Airflow, Marketing more »
and you’ll build the solution. They work in a mix of SQL, Python programming but also operate with modern tech such as Snowflake, Airflow, DBT, Kubernetes/Docker and cubeJS whilst also using Tableau CRM and Jupyter notebooks for gaining data insights. If you think there’s a … to be a SQL Server expert with strong architecture skills and also strong skills with Python, and DBT. The more experience with things like Airflow, cubeJS and Snowflake, plus other data viz tools would be a huge plus. Total compensation for this role is up to £250,000, split more »
ensuring reliability, scalability, and efficiency. Design and implement ETL processes and data pipelines primarily within the AWS environment. Utilize Python coding skills, particularly in Airflow, to enhance and optimize our infrastructure. Collaborate with cross-functional teams to understand data requirements and implement solutions. Gain proficiency in AWS services such … Science, Engineering, or related field. 0-3 years of experience in data engineering or related roles. Strong proficiency in Python programming, with experience in Airflow preferred. Familiarity with AWS services such as Lambda functions, Glue, and S3 is a plus. Solid understanding of SQL for ETL processes, including experience more »
Data Analyst - 12 Month Fixed Term Contract - (Maternity Cover) Salary: £46,382 PA Origin – Great homes, positive people, and strong communities An inspiring past and a bright future A home is the essential foundation for a good quality of life more »
Nottingham, Nottinghamshire, East Midlands, United Kingdom
Microlise
data practices Possess strong knowledge of data tools, data management tools, and various data and information technologies. E.g. DAMA DMBOK, Microsoft SQL Server, Couchbase, Apache Druid, Spark, Kafka, Airflow, etc In-depth understanding of modern data principles, methodologies, and tools Excellent communication and collaboration skills, with the ability … native computing concepts and experience working with hybrid or private cloud platforms is a plus. Demonstrable technical experience working with a Microsoft, Redhat, and Apache data and software engineering environment. A team-oriented individual with a passion for engineered excellence and the ability to lead and motivate a team more »
fundamentals Experience in contributing to complex shared repositories What's nice to have: Experience in front-end development, Javascript and API development Familiarity with Airflow, DBT, Databricks Experience working with Enterprise Resource Planning (e.g. Oracle, SAP) and CRM systems RECRUITMENT PROCESS: 30-Minute CV Run-through Technical Task Run … fundamentals Experience in contributing to complex shared repositories What's nice to have: Experience in front-end development, Javascript and API development Familiarity with Airflow, DBT, Databricks Experience working with Enterprise Resource Planning (e.g. Oracle, SAP) and CRM systems RECRUITMENT PROCESS: 30-Minute CV Run-through Technical Task Run more »
Guildford, Surrey, South East, United Kingdom Hybrid / WFH Options
Allianz Insurance Plc
Location: Guildford/hybrid Salary: Dep on exp + car allowance + annual bonus + benefits Who we are Allianz is a global insurance company serving across 70 different countries, but from the very first day you join our team more »
Senior Data Engineer Mobysoft is one of the fastest growing SaaS providers in the UK and has been shortlisted in the "Top 50 fastest growing technology companies in the North" for four successive years. Mobysoft provides predictive analytical software that more »
Senior Data Engineer London/Bracknell - Flexible Up to £90,000 Austin Fraser has an exciting opportunity for a Senior Data Engineer to join one of the leading retail chains, running two of the busiests sites in the UK. This more »
Senior Data Engineer Bracknell - Flexible Up to £90,000 Austin Fraser has an exciting opportunity for a Senior Data Engineer to join one of the leading retail chains, running two of the busiests sites in the UK. This is a more »
Senior Data Engineer Bracknell - Flexible Up to £90,000 Austin Fraser has an exciting opportunity for a Senior Data Engineer to join one of the leading retail chains, running two of the busiests sites in the UK. This is a more »
