51 to 75 of 88 Remote/Hybrid Apache Spark Jobs

design scalable Lakehouse solutions supporting critical business data. What you'll be doing Leading Azure Databricks architecture Designing enterprise Lakehouse solutions Building scalable Spark and Delta Lake pipelines Designing streaming and batch data platforms Driving engineering best practice Mentoring engineers Working closely with senior business stakeholders Defining future-state … data architecture Technical Environment Azure Databricks Apache Spark/PySpark Delta Lake Azure Data Factory Azure Data Lake Gen2 Azure Event Hubs Azure Synapse Python SQL Terraform CI/CD Medallion Architecture We're looking for Expert Databricks experience Strong Azure Data Platform experience Enterprise Data Architecture Hands ...

Experience with Infrastructure-as-Code tools such as Terraform Knowledge of DevOps, CI/CD, platform automation, and cloud‐native operational practices Familiarity with Apache Spark, dbt, and modern data integration technologies Delivery Experience Experience leading architecture for enterprise-wide Databricks implementations and cloud data modernization initiatives Proven ...

Strong commercial experience in data engineering and modern data platforms. Advanced programming skills in Java or Python. Expertise working with technologies such as Kafka, Spark, Flink or similar distributed data frameworks. Experience delivering solutions within AWS or other major cloud platforms. Strong understanding of data architecture, scalability and performance ...

scalable data analytical solutions in a large-scale business environment Experience in deep analytic focused work leveraging Distributed File System ecosystem tools: Linux, Python, Spark, Hive; and modern data platforms and tools (e.g. Databricks, NoSQL stores) and AI/ML frameworks (e.g. NLP). LLM integration patterns, RAG pipelines ...

language like Go, Java, JavaScript, Python Distributed Software Architecture exposure in high volume production scenarios Working with Data Mesh, BigData technologies such as EMR, Spark, Databricks Designing, tracking and testing to SLOs and Chaos Engineering toError Budgets Implementing Business Continuity (BCP) and Disaster Recovery (DRP) plans including tracking ...

writing, talks, or threads about building with AI. MLOps and model serving experience (BentoML, MLflow, Vertex AI, SageMaker). Streaming and batch ingestion pipelines (Spark, Airflow, Beam, Glue). Healthcare or life sciences domain exposure. AWS Professional certification or other relevant industry certifications. What We Offer At Newpage ...

Strong experience with MLOps, production ML systems, model lifecycle management, and monitoring.* Experience working with large-scale data systems and distributed computing frameworks (e.g. Spark, Hadoop).* Knowledge of experimental design, scientific methodology, and analysis.* Strong partner management and confirmed ability to influence large organisations without direct authority.* Demonstrated ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

loop, calibration, guardrails, auditability) Experience with hyperparameter tuning at scale and experimentation frameworks Strong technical experience with SQL and Python; significant experience with Spark for large-scale data processing Excellent interpersonal skills with demonstrable collaborative leadership capabilities (coaching, mentoring, guiding, delivery ownership and stakeholder engagement) and strong standards ...

Advanced degree (Master’s or PhD) in a quantitative discipline (Mathematics, Computer Science, Data Science, Physics or Statistics). Proficiency in designing, implementing (python, spark, cloud environments) and deploying quantitative models in a large financial institution, preferably in Front Office. Experience in data analytics or screening model development. Subject ...

collaboration and delivery. What We’re Looking For Strong proficiency in Python, Scala, or Java, with hands-on experience of Big Data technologies including Spark, Hadoop, or similar distributed processing frameworks. Experience building real-time event streaming solutions using technologies such as Kafka, Spark Streaming, or Amazon Kinesis … solutions. Bonus Points For Expertise in data modelling, schema design, and working with structured and semi-structured datasets. Experience with distributed technologies including Hadoop, Spark, HDFS, Hive, Databricks, or AWS Lake Formation. Knowledge of automating ingestion and transformation frameworks within cloud-native data platforms. Experience delivering solutions within highly ...

collaboration and delivery. What We’re Looking For Strong proficiency in Python, Scala, or Java with hands-on experience in Big Data technologies including Spark, Hadoop, or similar distributed processing frameworks. Practical experience building real-time event streaming pipelines using technologies such as Kafka, Spark Streaming, or Amazon … environments. Bonus Points For Expertise in data modelling, schema design, and working with structured and semi-structured datasets. Experience with distributed technologies including Hadoop, Spark, HDFS, Hive, Databricks, or AWS Lake Formation. Knowledge of automating cloud-based ingestion and transformation frameworks. Experience delivering data engineering solutions within highly regulated ...

Role : Senior Data Manager/Lead Location : London (Hybrid) Comp : £115,000 P/A (FTC) Duration: 12 Months Skills: Python (Programming Language), SQL, Apache Spark, PySpark, Azure Databricks, Data Lakes, Data Warehouse Architecture, Leadership, Software Development, Management The role We're partnering with a large, data-driven ...

kubernetes) Experience with Key value stores Experience in git and basic Unix Commands You might also have (Desirable) Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. A dedicated ...

abuse detection, alongside a clear sense of when simpler rules-based approaches are the better choice Experience with cloud data tooling such as BigQuery, Spark, dbt, Airflow or similar Experience building tooling for operational or investigative teams, and partnering closely with the people who use it Interest ...

help deliver robust, secure and high-performing data products. What youll be doing: Design and build scalable data pipelines using Python, SQL, Spark and Databricks. Develop, deploy and optimise machine learning models, including NLP and generative AI solutions. Build data ingestion pipelines across batch and streaming environments. Design data … visualisations using Power BI or Tableau. Mentor junior team members and promote engineering best practices. What you need: Commercial experience with Python, SQL, Spark and modern data engineering frameworks. Experience deploying machine learning models into production. Strong knowledge of cloud platforms such as Azure, AWS or GCP. Experience with ...

like ER/Studio, ERwin, or Sparx EA (preferred). Experience reverse engineering models from existing databases. Proficiency in distributed data processing frameworks (e.g., Spark, Flink, Hadoop). Experience with API management and gateway tools and services. Experience of RESTful APIs for ingesting and exposing data. Proven track record … with data governance, quality, lineage, and security practices. Experience with real-time/streaming data technologies, data Ingestion/ETL (e.g., Apache Kafka, Apache NiFi, Kinesis, Pub/Sub). Experience in, or knowledge of, DevSecOps Tooling and Processes. Self-starter with the ability to appropriately prioritise ...

infrastructure (Solr, Kubernetes). Knowledge of key‐value stores, Git, and basic Unix commands. Desirable Experience with large data processing technologies such as Apache Spark. Hybrid Working Hybrid working is available. Team members are required to work onsite on Mondays and Tuesdays, and from home for the remaining days. ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...

with product stakeholders to align technical strategy with business goals and resolve blockers across the team. Apply efficient data transformations using distributed computing (e.g., Spark, Dask) and ensure robust test coverage for production systems. Mentor junior team members and utilise model explainability methods to drive feature performance improvements. About ...

system efficiency at scale Familiarity with a range of tools and technologies including: Development: Kotlin, Java Data storage/processing: DynamoDB, MySQL, Snowflake, Databricks, Spark Infrastructure as code and CI/CD: Terraform, GitLab CI/CD//You don't need to have experience in every area ...

Data Engineer Location: London (Hybrid) Salary: Up to £75,000 + Bonus + Benefits Technology: SQL, Python, Spark - Azure, AWS, GCP - ETL/ELT - Data Governance The Opportunity Are you a Data Engineer looking for an opportunity to work on large-scale data transformation programmes whilst helping shape enterprise … data lineage. Support Data Governance initiatives, ensuring data security, privacy and compliance standards are maintained. Work with large and complex datasets using SQL, Python, Spark, Hadoop and modern data technologies. Collaborate with technical and business stakeholders within Agile delivery environments. Support data architecture and data modelling activities across enterprise ...

Desirable SC clearance alongside NPPV3 Functional/typed Scala — Cats, Cats Effect, FS2, http4s, Circe — a strong plus Streaming/big-data frameworks (Kafka, Spark, Flink) for integrating with partner pipelines Government, policing or law-enforcement data experience Data lineage, provenance, audit or integrity/verification tooling Secure ...