1 to 25 of 39 Apache Spark Jobs in the East of England

design, implement, and optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data … Databricks implementation, including architecture, development, deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture ...

knowledge of optimisation techniques for large-scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands-on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache … cost optimisation strategies for cloud data platforms Experience with data quality frameworks and implementation Experience with data visualisation tools like Power BI or Apache Superset Experience with other cloud data platforms like AWS, GCP or Oracle Experience with modern unified data platforms like Databricks or Microsoft Fabric Experience with ...

solutions. Pipeline Development : Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data factory and engineering tools like Apache Spark to ensure seamless data flow. Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery … using SQL to query and manipulate data. Proficiency in cloud data tooling such as Synapse Analytics, Microsoft Fabric, Azure Data Factory or AWS equivalents. Spark/Pyspark or Python skills. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills. MS Fabric interest, skills or experience ...

field will drive your journey. Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient in Python programming and software engineering best ...

field will drive your journey. Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient in Python programming and software engineering best ...

field will drive your journey. Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient in Python programming and software engineering best ...

field will drive your journey. Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient in Python programming and software engineering best ...

understanding of Terraform enterprise. Experience in delivering system design, application development, testing, and operational stability. Knowledge of Big Data distributed compute frameworks like Spark, Glue, MapReduce etc. Excellent troubleshooting, analytical, and communication skills. Preferred Qualifications, Capabilities, and Skills: Multi Region disaster recovery setup, monitoring and testing. Experience in Data ...

warehouses (BigQuery preferred, or Snowflake/Redshift). Strong experience with orchestration tools (Airflow, Dagster), ELT pipelines (Fivetran, dbt), and distributed data processing frameworks (Apache Spark, Flink). Hands-on experience using AI tools to accelerate engineering workflows - code generation, code review, pipeline debugging, or documentation Strong coding ...

warehouses (BigQuery preferred, or Snowflake/Redshift). Strong experience with orchestration tools (Airflow, Dagster), ELT pipelines (Fivetran, dbt), and distributed data processing frameworks (Apache Spark, Flink). Hands-on experience using AI tools to accelerate engineering workflows - code generation, code review, pipeline debugging, or documentation Strong coding ...

warehouses (BigQuery preferred, or Snowflake/Redshift). Strong experience with orchestration tools (Airflow, Dagster), ELT pipelines (Fivetran, dbt), and distributed data processing frameworks (Apache Spark, Flink). Hands-on experience using AI tools to accelerate engineering workflows - code generation, code review, pipeline debugging, or documentation Strong coding ...

Veeva Link’s next-gen Data Platform Improve our current environment with features, refactoring, and innovation Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Experiment in your domain to improve precision, recall, or cost … savings Requirements Expert skills in Java or Python Experience with Apache Spark or PySpark Experience writing software for the cloud (AWS or GCP) Speaking and writing in English enables you to take part in day-to-day conversations in the team and contribute to deep technical discussions Nice ...

Veeva Link s next-gen Data Platform Improve our current environment with features, refactoring, and innovation Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Experiment in your domain to improve precision, recall, or cost … savings Requirements Expert skills in Java or Python Experience with Apache Spark or PySpark Experience writing software for the cloud (AWS or GCP) Speaking and writing in English enables you to take part in day-to-day conversations in the team and contribute to deep technical discussions Nice ...

Analytics, Artificial Intelligence or a related field, or equivalent experience, would be effective, although practical experience is equally valuable."Nice to Have"Azure, Databricks, Apache Spark or comparable cloud and data platforms. Data pipelines, ETL and workflow-orchestration technologies. Machine-learning methods such as forecasting, prediction or classification. ...

part of the feature design process. Work on Veeva Link s next-gen Data Platform Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Expert skills in Java or Python Experience writing software ...

part of the feature design process. Work on Veeva Link s next-gen Data Platform Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Expert skills in Java or Python Experience writing software ...

part of the feature design process. Work on Veeva Link s next-gen Data Platform Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Expert skills in Java or Python Experience writing software ...

help evolve platform standards and documentation What we re looking for 3 5 years of data engineering experience Strong Python & SQL skills Experience with Apache Spark/PySpark Ability to monitor, troubleshoot , and resolve pipeline failures Familiarity with lakehouse concepts Experience with Git, Databricks, s treaming/… Great Expectations Nice to have Exposure to YAML Experience building analytics data models in Spark What you ll get in return Autonomy and ownership we ll set the vision and share the context, then we trust you to run with things that s why we hire you! Work alongside ...

journey. What we re looking for Expert in batch, distributed data processing and near real-time streaming data pipelines with technologies like Kafka, Flink, Spark etc. Experience in Databricks is a plus. Experience in Data Lake/Big Data Analytics platform implementation with cloud based solution; AWS preferred. Proficient … Feature engineering. Exposure to common data analysis and ML technologies such as on scikit-learn, pandas, NumPy, XGBoost, LightGBM. Exposure to tools like Apache Oozie, Apache Airflow. £70,000 - £105,000 a year Why you ll love it here: We want to make Prima a happy and empowering ...

part of the feature design process. Work on Veeva Link s next-gen Data Platform Work with JVM-based languages or Python on Spark-based data pipelines Operate ML models in close cooperation with our data science team Expert skills in Java or Python Experience writing software ...

related field, or equivalent practical experience. Experience with AWS or GCP, Kubernetes, SQL warehouse and batch-processing technologies such as Trino, Presto or Spark, Kafka or other streaming systems, and Go or Python would be useful. Node.js experience is a plus. We value transferable evidence over an exact match ...

related field, or equivalent practical experience. Experience with AWS or GCP, Kubernetes, SQL warehouse and batch-processing technologies such as Trino, Presto or Spark, Kafka or other streaming systems, and Go or Python would be useful. Node.js experience is a plus. We value transferable evidence over an exact match ...

with modern NLP stack and multi-modal representation learning is a plusWe'd love to see that you've worked with big data systems (Spark, S3, and Airflow) and can program (Java, Scala, or Python)Good understanding of system architecture. Have experience in big data technologies and streaming architecture ...

Experience supporting the Intelligence Community or Department of Defense environments Experience with Python and working with databases Experience with big data frameworks such as Spark, PySpark, or Elastic Search Experience with data visualization platforms such as Kibana, Foundry Workshop, or Palantir Knowledge of secure coding practices and development ...

core component of the role is applying, modifying and designing algorithms and mathematical models to solve business problems using big data architectures (Hadoop, Spark, Cloud). In this role, your primary modelling focus will be neural network‐based forecasting, with a strong emphasis on rigorous evaluation against robust baselines ...