21 of 21 Apache Spark Jobs in Scotland

techniques, including machine learning and their application in financial services.* Proficiency in data tools, programming languages, and database technologies, in particular in Azure Databricks, Spark, Spark SQL and Python/pyspark.* Understanding of data governance, security, and data quality management principles.* Experience working with large, complex datasets … producing audit evidence**Essential Skills:*** Data pipeline development and optimisation* Data modelling and architecture (in Azure Databricks)* Programming and scripting for data processing (Spark, Spark SQL, Python, pyspark)* Data science and machine learning* Data governance and quality assurance* Problem-solving and analytical thinking* Stakeholder engagement and technical communication ...

Experience building applications on public cloud platforms like AWS, GCP, and Azure, with a preference for AWS. Proficiency in real-time streaming applications using Apache Spark Streaming, Apache Kafka, Amazon Kinesis, and emerging cloud serverless managed services. #J-18808-Ljbffr ...

processes. Technical Skills Programming: Proficiency in Python, Java, Scala, or similar languages. Big Data Technologies: Hands-on experience with big data tools e.g. (Databricks, Apache Spark, Hadoop). Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. Expertise in relational databases (e.g. … postgres, sql server) Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment. Experience ...

have Programming: Proficiency in Python, Java, Scala, or similar languages for data processing. Big Data Technologies: Hands-on experience with tools such as Databricks, Apache Spark, and Hadoop. Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. Database Management: Expertise in relational … databases (e.g. Postgres, SQL Server). Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines ...

code. LLMOps & AI Model Management: Familiarity with tools like MLflow, LangChain, Hugging Face, Kubeflow, or similar platforms. Data Processing: Working knowledge of Databricks/Spark or comparable large-scale data processing tools. SQL: Solid capabilities in data querying and preparation. Nice to have: Time-series forecasting or optimisation, ideally ...

into business processes and decision-making.**Technical Skills:**Hands-on experience in one or more of the following areas:Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analyticsCloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCPAI Technologies – GenAI frameworks, Agentic AI, vector databases, semantic search ...

lifecycle management. Experience working within Agile development teams. Active UK SC security clearance. Highly Desirable AWS – experience building or supporting cloud-native data solutions. Apache Kafka – experience with event streaming, messaging or real-time data processing. Kubernetes – experience with container orchestration and distributed application deployment. Experience developing Java-based … data processing or event-driven applications. Additional Advantageous Experience Apache Spark, Flink, Hadoop or other Big Data technologies. Working with high-volume or complex enterprise datasets. Experience delivering solutions within government, defence or other secure environments. Why Consider This Opportunity? Long-term contract running until December 2027 . ...

external customers. Cloud data platform experience (AWS, Azure or GCP) and streaming/real-time data. Wider data product experience across platforms such as Apache Spark, Databricks and Snowflake. Exposure to AI/ML data pipelines and emerging agent interfaces such as MCP. Hands-on master data management ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance ...

Mathematics, or a related field. Applied experience in data science, machine learning, or related areas. Strong programming skills in Python with experience in PySpark, Spark SQL, and manipulating data. Experience LLM Orchestration and Agentic AI libraries Proficiency in using GenAI models (OpenAI or similar) to solve business problems. Experience ...

role: Python SQL Cloud platform architecture (AWS, Azure, GCP) Data Lakes/Lakehouses and analytical tools (Databricks, Azure Fabric/OneLake, AWS Lake Formation, Spark, Athena, etc.) CI/CD and other DevOps practices such as IaC Testing Additional experience with the following would be beneficial but not essential … vector databases Streaming technologies (Kafka, Kinesis, Flink, etc.) Containers and related services (Docker, Kubernetes, container Registries, etc) Data modelling approaches (Kimball, Imnon) Orchestration tools - Apache Airflow, Prefect or cloud-native tools Backend software development (Java, APIs, Scalability, Logging and Monitoring etc.) MLFlow and other MLOps/Machine Learning Engineering ...

scripts, with a particular focus on building and optimizing multi‐threaded, multi‐process, concurrent, and asynchronous solutions. Experience with distributed computing tools (such as Apache Spark for Python) is a plus. Good to have experience in designing and developing AI‐powered solutions using Large Language Models (LLMs), retrieval ...

scripts, with a particular focus on building and optimizing multi-threaded, multi-process, concurrent, and asynchronous solutions. Experience with distributed computing tools (such as Apache Spark for Python) is a plus. Good to have experience in designing and developing AI-powered solutions using Large Language Models (LLMs), retrieval ...

approaches Experience with Generative AI techniques such as retrieval-augmented generation, fine-tuning, or prompt engineering Exposure to distributed data processing frameworks, such as Spark Exposure to fraud detection, anti-money laundering, anomaly detection, or graph and network-based modelling Awareness of model risk management, governance, and regulatory controls ...

Candidate Requirements: Extensive experience building and deploying machine learning models in production, with strong hands-on Python capability. Experience with distributed computing frameworks (e.g., Spark or similar) and scalable data processing environments. Exposure to cloud-based platforms (e.g., AWS, Azure, GCP or Databricks) and modern ML/MLOps practices. ...

contribute to best practice engineering across cross-functional teams. Based in Scotland with hybrid work from Edinburgh or Glasgow, you’ll work with DeltaLake, Spark, Kafka, and Unity Catalog, applying DevOps ideas to CI/CD pipelines and GDPR-compliant data #J-18808-Ljbffr ...

essential: Background in financial services or regulated environments. Exposure to financial crime domains such as fraud, AML or sanctions. Experience with Spark or PySpark. Familiarity with AWS cloud services and data platforms. You will be assessed on key capabilities relevant to the role, including risk and controls, change ...

building and deploying machine learning models in production, alongside strong hands‐on Python skills. You should be comfortable with distributed computing tools such as Spark or PySpark, cloud platforms and modern MLOps practices. Experience managing and developing data scientists is essential, as is the ability to explain technical decisions ...

need to potentially work in either Edinburgh or Glasgow depending on client expectations. What You’ll Do Design and develop robust pipelines using DeltaLake, Spark Structured Streaming, and Unity Catalog Build real‐time event‐driven solutions with tools such as Kafka and Azure Event Hubs Apply DevOps principles … knowledge of DevOps tools and CI/CD processes Bonus Points For Development experience in Scala or Java Familiarity with Cloudera, Hadoop, HIVE, and Spark ecosystem Understanding of data privacy regulations, including GDPR, and experience working with sensitive data Ability to learn and adapt new technologies quickly to meet ...

Hands-on experience in system design, application development, testing, and operational stability Proficient in coding in one or more languages such as Java or Spark Experience developing, debugging, and maintaining code in a large corporate environment with modern programming and database querying languages Knowledge of the Software Development Life … within a technical discipline (e.g., cloud, artificial intelligence, machine learning, mobile) Ability to solve data-oriented problems using technologies such as SQL, Relational DB, Spark, NoSQL, optimizing for performance Preferred Qualifications, Capabilities, and Skills: Experience with Spark performance tuning of complex calculations on large datasets Experience delivering production ...

Hands-on experience in system design, application development, testing, and operational stability Proficiency in coding in one or more languages such as Java or Spark, with experience developing, debugging, and maintaining code Knowledge of modern programming and database querying languages, including SQL, relational databases, and NoSQL Solid understanding … processes within a technical discipline (e.g., cloud, AI, machine learning, mobile) Ability to solve data-oriented problems using technologies such as SQL, relational databases, Spark, and NoSQL Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, test creation, troubleshooting ...