1 to 25 of 35 Remote Apache Spark Jobs

evolution of data architecture and help shape the organisations data engineering capabilities. Key Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark and Python. Develop and maintain robust ELT processes to ingest, transform and integrate data from multiple sources. Contribute to the design and evolution … platform performance. Essential Skills & Experience Strong hands-on experience with Databricks . Proven experience designing, implementing and optimising data pipelines. Strong knowledge of Apache Spark and Python . Understanding of Lakehouse architecture and modern data engineering principles. At least five years experience across ELT, data engineering and data ...

Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platform Develop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta Lake Contribute to platform architecture, engineering standards and best practices across the data estate Work with Product, Analytics and Data Science … continuous improvement across the engineering team Job Requirements: Significant experience building and supporting scalable, production-grade data platforms Strong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQL Good understanding of cloud-based data engineering, ideally within Microsoft Azure environments Experience with Infrastructure as Code ...

designs, and support decommission planning for legacy components Core Technical Skills Azure Synapse & Data Platform Mandatory hands-on expertise with: Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool) Azure Data Lake Storage Gen2 (ADLS Gen2) Delta Lake on Azure (Synapse Lakehouse patterns) Oracle Golden Gate Replication for real … patterns (YAML, PySpark, SQL-driven generation from mapping documents) Databricks & Lakehouse Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred) Strong Apache Spark skills (PySpark/Spark SQL) Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse Ability ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

Senior Data Engineer Role (Python/Spark/Databricks) - Hybrid - Contract - Banking Role - Senior Data Engineer Rate - £850 p/d (Inside IR35) Duration - 6 months with very likely extension Location - Hybrid/Liverpool Street (London) - 3 days per week in a Liverpool Street office About the Role Global … delivering critical data platforms and analytics capabilities across the business. This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

understanding of data modeling. Familiarity with data governance, lineage, and quality frameworks. Fluency with AI tools and ADLC Working knowledge of Databricks Experience with Apache Spark Preferred: Hands-on experience with AWS data services (e.g., S3, Glue, Lambda, MSK). Proficient in CI/CD pipelines, automated testing ...

collaboration skills. Desirable Experience Experience with AWS services such as S3, Glue, Lambda, Step Functions, Redshift or related data engineering technologies. Familiarity with Apache Spark, Airflow, distributed processing, data lakes and data warehouses. Experience with infrastructure as code, Docker or automated deployment pipelines. Background in government, defence, national ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

into business processes and decision-making. Technical Skills: Hands-on experience in one or more of the following areas: Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analytics Cloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCP AI Technologies – GenAI frameworks, Agentic AI, vector databases ...

communication and technical writing skills. Ability to explain technical concepts to diverse audiences. Preferred Skills: Experience with large-scale data platforms such as Databricks, Spark, Hadoop, or Hive. Experience developing, evaluating, or deploying machine learning and AI solutions, including work with large language models, and cloud environments (e.g. ...

Redshift, Glue, EMR) or GCP (BigQuery, Dataflow, Pub/Sub) Strong coding abilities in Python/Java Advanced SQL skills Hands-on experience with Spark and Kafka Excellent written and spoken skills in Dutch & English Bachelor's degree Proactive and adaptable mindset in dynamic environments Proactive and adaptable Strong ...

approaches Experience with Generative AI techniques such as retrieval-augmented generation, fine-tuning, or prompt engineering Exposure to distributed data processing frameworks, such as Spark Exposure to fraud detection, anti-money laundering, anomaly detection, or graph and network-based modelling Awareness of model risk management, governance, and regulatory controls ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

effectively with technical teams including Data Scientists and MLOps Engineers. Desirable Skills Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/ ...

statistical modeling Experience with programming languages (Python, R, SQL) Strong understanding of data visualization tools (Tableau, Power BI) Experience with big data technologies (Hadoop, Spark) Strong understanding of data mining techniques and data modeling Excellent problem-solving and communication skills Qualifications Master's degree in Data Science, Computer Science ...

large language model (LLM) systems, including retrieval augmented generation (RAG) and fine-tuning. Build scalable ETL/ELT pipelines using tools such as Spark or dbt, so data is high quality, time-stamped, and ready for both training and real-time inference. Design and deploy data collection systems ...

Senior Data Engineer – SQL/Python/AWS Adria Solutions is recruiting a Senior Data Engineer for a growing business developing a new digital services platform. With established backing, the business is building its data ...

sets, model development, statistics, experimentation, data visualization, optimization, software development Cross-functional collaboration experience in ML product development Strong Python, Java, Kafka, Hadoop, SQL, Spark or equivalent; version control experience Excellent English communication skills and ability to drive initiatives and lead teams cross-functional collaboration leadership and mentoring clear ...

understanding of statistics, machine learning methodologies, and model evaluation. Experience delivering AI solutions from proof-of-concept through to production. Exposure to Databricks, MLflow, Spark, or similar cloud-based data platforms. Ability to influence senior stakeholders and lead technical strategy Desirable Experience Experience building multi-agent AI systems ...

articulate steps towards becoming a more data-driven bank excellent communication mentoring ability collaborative mindset Machine Learning/Data Science expertise Python programming Apache Spark ...