151 to 172 of 172 Remote Apache Spark Jobs

compliance and Data Quality Assessment status Building data processing workflows on AWS (S3, Glue, EMR, Athena, RDS/Redshift) and, where applicable, Databricks, Spark, and Hive for big-data processing Creating data products that feed Power BI dashboards and Salesforce reports, ensuring consistent definitions, refresh schedules, snapshot management … discipline Strong understanding of AWS data services including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Athena, QuickSight, and Amazon SNS Experience with Spark and Hive for big-data processing Experience building job workflows on Databricks or comparable platforms Familiarity with workflow orchestration tools such as Airflow, Luigi ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

strong programming skills in Python, Rust, or Java Experience building and maintaining data pipelines using processing/streaming frameworks (e.g., Kafka, Flink, Beam, Spark) and orchestration tools (e.g., Airflow) Experience architecting data stores and schemas for AI workflows (e.g., RAG) Active Google Cloud certifications, or willingness to obtain within … bias for action US Citizen Preferred Qualifications Deep expertise in the Google Cloud Platform (GCP) ecosystem, specifically building streaming and batch pipelines using Dataflow (Apache Beam), Pub/Sub, BigQuery, and Cloud Composer (Airflow) Strong background in data modeling and architecture across relational (e.g., PostgreSQL), NoSQL (e.g., Firestore, MongoDB ...

work on a Big Data Project in the finance sector. Required Experience: Java Big Data experience On-premises cluster experience Databases Kubernetes Kafka Apache Spark Preffered Experience: Finance sector experience ...

deliver end-to-end data pipelines on cloud platforms (Google Cloud Platform preferred). Building scalable data workflows using distributed technologies such as Spark, Flink, Storm, or similar Develop robust processing and migration pipelines, including support for legacy DataStage decommissioning and modernisation. Work with a variety of database technologies … Java, Python or Scala. Experience with cloud data services (GCP preferred; Azure/AWS acceptable). Experience with distributed data processing frameworks such as Spark (Core/SQL/Streaming), Flink, or Storm. Understanding of designing scalable data models for varied access patterns. Working on large scale big data ...

Pandas, numpy, scipy, scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer ...

platform role Strong hands-on experience with AWS (EC2, S3, RDS, Lambda, Redshift, Glue, Athena) Experience with real-time and batch data processing - Apache Flink, Kafka, Spark, or similar Proficiency in one or more of: Python, Java, Scala Experience with data modelling , ETL processes, and distributed systems Comfortable ...

Strong hands-on experience wi th AWS (EC2, S3, RDS, Lambda, Redshift, Glue, Athena) Experience wi th real-time and batch data process ing — Apache Flink, Kafka, Spark, or similar Proficiency in one or more o f: Python, Java, Scala Experience wi th data modell ing, ETL processes ...

Engineer, you will play a crucial role in designing, developing, and maintaining our Advana data infrastructure and systems. Your expertise in ETL, Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks will be essential in ensuring efficient data processing and analysis. Responsibilities Design, develop, and implement … data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks. Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing. Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy ...

house architectures, modern BI platforms, ML/MLOps, Conversational Analytics, Generative AI, and Agentic AI frameworks Secondary Skills: Looker, Tableau, Power BI, SQL, Python, Spark, Kafka Experience: 15+ years overall experience (with 5+ years in AI/Data Architecture roles) Job Summary: We are seeking a highly skilled … Looker AI/ML: TensorFlow, PyTorch, scikit-learn, LLM frameworks MLOps: CI/CD, feature stores, model registries, monitoring tools Data: SQL, Python, Spark, Kafka BI: Looker, Tableau, Power BI (or equivalent) Regards, DEVENDER REDDY Balin Technologies Email ...

across the organization Technical Environment Cloud-native AI/ML and analytics environments (AWS, Azure) Distributed data platforms and enterprise analytics ecosystems Python, R, Spark, TensorFlow, PyTorch, Databricks, and related ML frameworks MLOps pipelines, model deployment platforms, and automation frameworks Real-time streaming and event-driven analytics systems DevSecOps … environments and cloud-native analytics platforms Experience with DataOps, CI/CD integration, and Agile/SAFe delivery models Strong experience with Python, R, Spark, TensorFlow, PyTorch, Databricks, and related AI/ML technologies Experience developing dashboards, technical reports, analysis plans, and reproducible analytics workflows Experience collaborating across engineering ...

across the organization Technical Environment Cloud-native AI/ML and analytics environments (AWS, Azure) Distributed data platforms and enterprise analytics ecosystems Python, R, Spark, TensorFlow, PyTorch, Databricks, and related ML frameworks MLOps pipelines, model deployment platforms, and automation frameworks Real-time streaming and event-driven analytics systems DevSecOps … environments and cloud-native analytics platforms Experience with DataOps, CI/CD integration, and Agile/SAFe delivery models Strong experience with Python, R, Spark, TensorFlow, PyTorch, Databricks, and related AI/ML technologies Experience developing dashboards, technical reports, analysis plans, and reproducible analytics workflows Experience collaborating across engineering ...

across the organization Technical Environment Cloud-native AI/ML and analytics environments (AWS, Azure) Distributed data platforms and enterprise analytics ecosystems Python, R, Spark, TensorFlow, PyTorch, Databricks, and related ML frameworks MLOps pipelines, model deployment platforms, and automation frameworks Real-time streaming and event-driven analytics systems DevSecOps … environments and cloud-native analytics platforms Experience with DataOps, CI/CD integration, and Agile/SAFe delivery models Strong experience with Python, R, Spark, TensorFlow, PyTorch, Databricks, and related AI/ML technologies Experience developing dashboards, technical reports, analysis plans, and reproducible analytics workflows Experience collaborating across engineering ...

model versioning Hands-on experience with Databricks Model Serving for production ML deployments Proficiency with GenAI frameworks/tools and technologies such as Apache Airflow, Spark, Flink, Kafka/Kinesis, Snowflake, and Databricks. Demonstrable experience in parameter-efficient fine-tuning, model quantization, and quantization-aware fine-tuning ...

solving complex challenges and improving data quality as the platform scales. ·The environment is modern and cloud based, with exposure to technologies such as Spark, Snowflake and BigQuery across AWS, GCP and Azure. ·You will have the opportunity to support and mentor other engineers, contribute to engineering standards … engineers What We Are Looking For ·Proven Python and SQL experience within production environments ·Experience delivering data transformation projects end to end ·Experience with Spark, Snowflake, BigQuery or similar technologies ·Solid experience working with AWS, GCP or Azure ·Comfortable working with complex, imperfect datasets ·Experience owning solutions from design ...

working We need an AI Engineer with a strong data engineering core to build AI‐powered analytics for front‐office trading teams using Databricks, Spark, and modern LLM workflows. Key Responsibilities: Design and deliver AI‐driven analytics for traders, including seasonality, correlations, regression, forecasting, and scenario modelling. Build scalable … reusable data pipelines in Databricks using PySpark, Spark SQL, Delta Lake, and Unity Catalog. Perform statistical and econometric analysis on large market and fundamentals time‐series datasets. Work directly with traders and analysts to turn ambiguous questions into production‐ready tools and clear insights. Implement LLM and agentic workflows ...

scalable pipelines Optimise data flows and processing efficiency Handle large and complex datasets reliably Core Technical Skills Streaming & Messaging Technologies Good practical understanding of Apache Kafka Experience working with: Event-driven data pipelines Streaming ingestion patterns Asynchronous processing architectures Understanding of: Topics and partitions Consumer groups Data reliability … reusable and version-controlled infrastructure definitions Databricks (Nice to Have) Exposure to or experience with Databricks Understanding of: Distributed data processing Notebook-driven development Spark-based workloads Collaboration & Stakeholder Engagement Work closely with: Data analysts/Data scientists/Engineers/Business stakeholders Translate business and analytical requirements into ...

generative AI inference and agentic applications. Your depth of expertise in technologies including Golang and Python programming languages, popular distributed compute frameworks including Spark/Dask/Ray/Flink, container (e.g., Kubernetes) and serverless (e.g., AWS Lambda) runtime environments, and ML+AI workload patterns will provide an amplifying technical … control and data plane implementations required to realize a highly available, multi-tenant, large scale and a secure machine learning platform Develop Ray and Spark distributed compute engine solutions to accelerate diverse workloads from LLM pre-training and reinforcement learning to large-scale data processing, while maximizing compute unit ...

generative AI inference and agentic applications. Your depth of expertise in technologies including Golang and Python programming languages, popular distributed compute frameworks including Spark/Dask/Ray/Flink, container (e.g., Kubernetes) and serverless (e.g., AWS Lambda) runtime environments, and ML+AI workload patterns will provide an amplifying technical … control and data plane implementations required to realize a highly available, multi-tenant, large scale and a secure machine learning platform Develop Ray and Spark distributed compute engine solutions to accelerate diverse workloads from LLM pre-training and reinforcement learning to large-scale data processing, while maximizing compute unit ...

generative AI inference and agentic applications. Your depth of expertise in technologies including Golang and Python programming languages, popular distributed compute frameworks including Spark/Dask/Ray/Flink, container (e.g., Kubernetes) and serverless (e.g., AWS Lambda) runtime environments, and ML+AI workload patterns will provide an amplifying technical … control and data plane implementations required to realize a highly available, multi-tenant, large scale and a secure machine learning platform Develop Ray and Spark distributed compute engine solutions to accelerate diverse workloads from LLM pre-training and reinforcement learning to large-scale data processing, while maximizing compute unit ...

Senior Software Engineer - Real-Time Data (Kafka/Spark) Chester | Hybrid working | 12-month contract | Inside IR35 | Global Investment Bank A leading global investment bank is looking for a hands-on Senior Engineer to build high-performance, real-time data platforms supporting regulatory compliance across global operations. This … processing , ideal for someone who enjoys solving complex data challenges at scale. What You'll Do Design and build Kafka-based streaming pipelines and Spark processing jobs Develop low-latency, high throughput systems handling large-scale datasets Create data matching and reconciliation solutions across multiple sources Work on event ...