1 to 25 of 28 Apache Spark Jobs in the East of England

design, implement, and optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data … Databricks implementation, including architecture, development, deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture ...

solutions. Pipeline Development : Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data factory and engineering tools like Apache Spark to ensure seamless data flow. Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery … using SQL to query and manipulate data. Proficiency in cloud data tooling such as Synapse Analytics, Microsoft Fabric, Azure Data Factory or AWS equivalents. Spark/Pyspark or Python skills. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills. MS Fabric interest, skills or experience ...

field will drive your journey. Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient in Python programming and software engineering best ...

field will drive your journey. Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient in Python programming and software engineering best ...

understanding of Terraform enterprise. Experience in delivering system design, application development, testing, and operational stability. Knowledge of Big Data distributed compute frameworks like Spark, Glue, MapReduce etc. Excellent troubleshooting, analytical, and communication skills. Preferred Qualifications, Capabilities, and Skills: Multi Region disaster recovery setup, monitoring and testing. Experience in Data ...

warehouses (BigQuery preferred, or Snowflake/Redshift). ~ Strong experience with orchestration tools (Airflow, Dagster), ELT pipelines (Fivetran, dbt), and distributed data processing frameworks (Apache Spark, Flink). ~ Hands-on experience using AI tools to accelerate engineering workflows - code generation, code review, pipeline debugging, or documentation ~ Strong coding ...

warehouses (BigQuery preferred, or Snowflake/Redshift). Strong experience with orchestration tools (Airflow, Dagster), ELT pipelines (Fivetran, dbt), and distributed data processing frameworks (Apache Spark, Flink). Hands-on experience using AI tools to accelerate engineering workflows - code generation, code review, pipeline debugging, or documentation Strong coding ...

Veeva Link s next-gen Data Platform Improve our current environment with features, refactoring, and innovation Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Experiment in your domain to improve precision, recall, or cost … savings Requirements Expert skills in Java or Python Experience with Apache Spark or PySpark Experience writing software for the cloud (AWS or GCP) Speaking and writing in English enables you to take part in day-to-day conversations in the team and contribute to deep technical discussions Nice ...

Analytics, Artificial Intelligence or a related field, or equivalent experience, would be effective, although practical experience is equally valuable."Nice to Have"Azure, Databricks, Apache Spark or comparable cloud and data platforms. Data pipelines, ETL and workflow-orchestration technologies. Machine-learning methods such as forecasting, prediction or classification. ...

help evolve platform standards and documentation What we re looking for 3 5 years of data engineering experience Strong Python & SQL skills Experience with Apache Spark/PySpark Ability to monitor, troubleshoot , and resolve pipeline failures Familiarity with lakehouse concepts Experience with Git, Databricks, s treaming/… Great Expectations Nice to have Exposure to YAML Experience building analytics data models in Spark What you ll get in return Autonomy and ownership we ll set the vision and share the context, then we trust you to run with things that s why we hire you! Work alongside ...

journey. What we’re looking for Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Databricks is a plus. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient … Feature engineering. Exposure to common data analysis and ML technologies such as on scikit-learn, pandas, NumPy, XGBoost, LightGBM. Exposure to tools like Apache Oozie, Apache Airflow. £70,000 - £105,000 a year Why you’ll love it here: We want to make Prima a happy and empowering ...

journey. What we’re looking for Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Databricks is a plus. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient … Feature engineering. Exposure to common data analysis and ML technologies such as on scikit-learn, pandas, NumPy, XGBoost, LightGBM. Exposure to tools like Apache Oozie, Apache Airflow. £70,000 - £105,000 a year Why you’ll love it here: We want to make Prima a happy and empowering ...

related field, or equivalent practical experience. Experience with AWS or GCP, Kubernetes, SQL warehouse and batch-processing technologies such as Trino, Presto or Spark, Kafka or other streaming systems, and Go or Python would be useful. Node.js experience is a plus. We value transferable evidence over an exact match ...

related field, or equivalent practical experience. Experience with AWS or GCP, Kubernetes, SQL warehouse and batch-processing technologies such as Trino, Presto or Spark, Kafka or other streaming systems, and Go or Python would be useful. Node.js experience is a plus. We value transferable evidence over an exact match ...

Experience supporting the Intelligence Community or Department of Defense environments Experience with Python and working with databases Experience with big data frameworks such as Spark, PySpark, or Elastic Search Experience with data visualization platforms such as Kibana, Foundry Workshop, or Palantir Knowledge of secure coding practices and development ...

core component of the role is applying, modifying and designing algorithms and mathematical models to solve business problems using big data architectures (Hadoop, Spark, Cloud). In this role, your primary modelling focus will be neural network‐based forecasting, with a strong emphasis on rigorous evaluation against robust baselines ...

secure and scalable solutions Preferred qualifications, capabilities, and skills Experience with modern data platforms such as Databricks or Snowflake Deep hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with ...

scaling patterns and reuse. Preferred qualifications, capabilities, and skills Experience with modern data platforms such as Databricks or Snowflake Deep hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with ...

leadership teams. Minimum Qualifications Masters in Statistics, Mathematics, Data Science, Machine Learning, Physics, Engineering, Computer Science or equivalent Proficiency in data querying using SQL, Spark, or equivalent technologies Proficiency in scripting languages for data processing and analysis (ex: Python, R, or Scala) Excellent communication and multi-functional collaboration skills ...

simple: to empower the next generation of short-term rental creators by providing flexible, user-friendly, and technology-driven tools that remove friction and spark innovation. Whether someone is managing their first property or scaling a portfolio, we re here to help turn ambition into sustainable success. ...

Python is required . Proficiency in at least one object-oriented language. Experience with big data technologies such as HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, or Presto. Experience with AWS, GCP, or Looker (advantageous but not essential). Solid background in data modelling, including the design, implementation ...

Supporting the design, scoping and implementation of data flows across the wider healthcare data ecosystem. Leveraging cloud and big data technologies, including Azure, Databricks, Spark and Data Lakes, to deliver high-performing data solutions. Adopting and promoting data engineering best practices, healthcare data standards and continuous improvement across … engineering best practices. Good command-line and Unix/Linux skills. Experience working with cloud platforms, ideally Microsoft Azure. Experience using technologies such as Spark, Databricks and data lake architectures. Knowledge of data storage and transformation formats such as Apache Parquet and Delta Lake. Experience working within Agile ...

Supporting the design, scoping and implementation of data flows across the wider healthcare data ecosystem. Leveraging cloud and big data technologies, including Azure, Databricks, Spark and Data Lakes, to deliver high-performing data solutions. Adopting and promoting data engineering best practices, healthcare data standards and continuous improvement across … engineering best practices. Good command-line and Unix/Linux skills. Experience working with cloud platforms, ideally Microsoft Azure. Experience using technologies such as Spark, Databricks and data lake architectures. Knowledge of data storage and transformation formats such as Apache Parquet and Delta Lake. Experience with containerisation ...

Supporting the design, scoping and implementation of data flows across the wider healthcare data ecosystem. - Leveraging cloud and big data technologies, including Azure, Databricks, Spark and Data Lakes, to deliver high-performing data solutions. - Adopting and promoting data engineering best practices, healthcare data standards and continuous improvement across … engineering best practices. - Good command-line and Unix/Linux skills. - Experience working with cloud platforms, ideally Microsoft Azure. - Experience using technologies such as Spark, Databricks and data lake architectures. - Knowledge of data storage and transformation formats such as Apache Parquet and Delta Lake. - Experience with containerisation ...

such as Pipeline Builder or AIP, or similar platforms for application develop ment and data integration Experience with distributed data and compute, such as Spark or Databricks, Hadoop or Hive, Kafka, or AWS EMR, and cloud environments used by the IC Experience with geospatial analysis tools, such as ArcGIS … such as Pipeline Builder or AIP, or similar platforms for application develop ment and data integration Experience with distributed data and compute, such as Spark or Databricks, Hadoop or Hive, Kafka, or AWS EMR, and cloud environments used by the IC Experience with geospatial analysis tools, such as ArcGIS ...