1 to 25 of 42 Remote/Hybrid Permanent Apache Spark Jobs

Data Engineer Salary up to £60,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via chelsea.hackett@searchability.com ABOUT THE CLIENT Due to continued growth, we’re seeking a skilled Data … optimise scalable data pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Software Developer Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required. The Role … looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data-driven applications and pipelines in a collaborative Dev and DevOps environment. The role focuses on writing and improving application code while working closely with DevOps ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

critical data at national scale. You must already hold active SC clearance to be considered. Responsibilities Design, build and maintain Databricks data pipelines using Apache Spark (PySpark & Spark SQL) Implement Delta Lake/Lakehouse architectures (Bronze, Silver, Gold) Develop scalable ELT/ETL workflows for batch ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

experience in data engineering or large-scale data platforms Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark, with a preference for streaming technologies Solid understanding of real-time data processing concepts such as event time, state handling, and backpressure Experience building ...

experience building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

Cloud Composer and IAM You have strong Python, SQL and PySpark skills You have experience with real-time data streaming using Kafka or Spark You have a good knowledge of Data Lakes, Data Warehousing, Data Modelling You're familiar with DevOps principles, containerisation and CI/CD tools such ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

proficiency in SQL and database technologies Experience with programming languages such as Python, Java, or Scala Hands-on experience with data pipeline tools (e.g., Apache Airflow, Kafka) Familiarity with cloud platforms (AWS, Azure, Google Cloud) Understanding of data warehousing solutions (e.g., Snowflake, Redshift, BigQuery) Strong problem-solving and analytical … skills Preferred Skills Experience with big data technologies (e.g., Hadoop, Spark) Knowledge of data modeling techniques and schema design Familiarity with containerization tools (Docker, Kubernetes) Experience with CI/CD pipelines and DevOps practices Understanding of data security and compliance standards ...

functions Highly proficient coding skills in SQL, Python & PySpark Proven knowledge in at least one of the following big data technologies – HIVE, Hadoop, MongoDB, Spark Experience within the Life Science, Medical Research or Pharmaceutical sector highly advantageous Analytically minded with the ability to solve problems creatively and efficiently Vast ...

code. LLMOps & AI Model Management: Familiarity with tools like MLFlow, LangChain, Hugging Face, Kubeflow, or similar platforms. Data Processing: Working knowledge of Databricks/Spark or comparable large-scale data processing tools. SQL: Solid capabilities in data querying and preparation. Data Architectures: Understanding of modern data infrastructure (lakehouses, data ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

Solid understanding of data governance, data lineage, security practices, and regulatory compliance frameworks. Experience working with large-scale data storage and processing technologies (SQL, Spark, NoSQL, ETL/ELT tools). Excellent communication and stakeholder management skills, with the proven ability to translate complex data concepts into actionable, business ...

levels. Knowledge of industry‐standard data models and frameworks such as Lambda, Medallion, and Data Mesh. Experience with real‐time data processing (Kafka, Spark) and BI tools (Power BI, Tableau). Strong analytical and problem‐solving skills, translating business requirements into effective data solutions. Equal Opportunities & Reasonable Adjustments ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

translate business needs into technical roadmaps What were looking for Strong experience in data architecture & large-scale systems design Expertise across Python, SQL, Spark, Scala Deep knowledge of AI/ML frameworks and modern data platforms Experience with cloud-native architecture and DevOps (Kubernetes, Terraform, CI/CD) Proven ...

data engineering within distributed systems Proven ability to build and optimise data pipelines at scale Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop Strong understanding of data ingestion and batch processing methodologies Experience with text or unstructured data is advantageous but not essential ...