1 to 25 of 104 Remote Apache Spark Jobs

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. Design … build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Design and oversee the delivery of secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

you. You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise …/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high levels of data quality, governance and pipeline reliability. Optimise performance across large-scale distributed data processing platforms. Collaborate with ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.What … lake & Fabric. Understand which Azure data focused components to use for which purpose.Microsoft certified architectDatabricksDesign and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Ability to Design, Build and deploy AI/ ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. What … which Azure data focused components to use for which purpose. Microsoft certified architect Databricks Design and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Ability to Design, Build and deploy ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. Key Responsibilities Design and architect Databricks-based data platforms and solutions. Build scalable data lakehouse and data warehouse architectures. Design … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

production-grade solutions using BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Composer . Frameworks & Storage: Strong understanding of batch/streaming tools (e.g., Apache Spark, Apache Beam ) and when to apply different storage paradigms (relational, columnar, document, data lakes). Modern Best Practices: Active practice with ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

strategy. What We're Looking For Essential 5+ years' experience in Data Engineering or a similar role. Strong experience with Python, SQL, Apache Spark, and Cloud platforms (AWS, Azure or GCP) Experience designing and operating production-scale data platforms. Desirable Experience working with AWS. Experience supporting machine learning ...

strategy. What We're Looking For Essential 5+ years' experience in Data Engineering or a similar role. Strong experience with Python, SQL, Apache Spark, and Cloud platforms (AWS, Azure or GCP) Experience designing and operating production-scale data platforms. Desirable Experience working with AWS. Experience supporting machine learning ...

version control. Data Engineering Stack Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL. Orchestration Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect. Data Quality and Lineage Familiarity with tools ...

Know Salary: Up to £80,000 per annum. Team: Work with cross‐functional teams and mentor junior engineers. Tools: Microsoft Fabric, Azure ML, MLflow, Spark, Python, SQL, CI/CD, and ML frameworks like TensorFlow, PyTorch, Scikit‐learn. Scope: End‐to‐end AI/ML solutions for multi‐product …/ML. Familiarity with Microsoft Fabric (Data Engineering, Data Science, Real‐Time Analytics) and Azure ML, including MLflow. Proficiency with tools such as Apache Spark, Azure Data Factory, GIT, Python and SQL. Skilled in developing, training and deploying machine learning models with frameworks like TensorFlow, PyTorch or Scikit ...

modern cloud-native data platforms Developing scalable ETL and ELT pipelines Working with AWS and Azure data services Engineering platforms using technologies such as Spark, Databricks and Kafka Building reliable, automated data pipelines supporting AI and Machine Learning workloads Collaborating with software, cloud and platform engineers in genuinely multidisciplinary … particularly interested in speaking with engineers with experience across areas such as: Data Engineering Cloud Data Platforms AWS and/or Azure Python SQL Apache Spark Databricks Kafka Data Modelling ETL/ELT Infrastructure as Code CI/CD for data platforms Whether your title is Data Engineer ...

native solutions that support analytics, automation, and digital transformation. Data Engineering, Full Stack & Platform Development Design and implement scalable data pipelines using tools like Apache Spark, Airflow, or dbt.Build and maintain data lakes, warehouses, and real-time streaming solutions. Develop APIs and microservices to expose data securely ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

strategic thinking. Essential Skills & Experience Strong Microsoft Fabric expertise (Lakehouse, Warehouse, Pipelines, Notebooks) OneLake architecture and management Advanced SQL/T-SQL PySpark and Spark optimisation Data warehousing, modelling and ETL/ELT development Power BI semantic modelling Experience delivering enterprise cloud data platforms Experience integrating multiple business systems ...

development . 3+ years of experience with cloud data platforms such as AWS, Azure, or Google Cloud Platform . 3+ years of experience with Spark, Databricks, Snowflake, Hadoop, or other big data technologies . Nice to Have: Experience with orchestration tools such as Apache Airflow, Azure Data Factory ...

Experience working for a software vendor, SaaS provider, or professional services organisation. Familiarity with Ataccama or similar enterprise data platforms. Exposure to Hadoop/Spark, NoSQL, XSLT, Hive, Kafka, Snowflake, or Databricks. Python or other programming languages. Familiarity with SDLC, code management and upgrades, and Unix/Linux. Perks ...

scale data platforms and end-to-end technical solutions within enterprise environments. \n Strong hands-on experience with open-source data technologies such as Apache Spark, Trino, Airflow, Apache Superset, or similar platforms. \n Demonstrable security architecture expertise, including threat modelling, IAM (RBAC/ABAC), encryption, secrets ...

this role leads the engineering of cloud-native data services for the firm’s strategic platform, driving the migration to Databricks and delivering scalable Spark/Scala solutions on AWS.The overall objective of this role is to lead applications systems analysis and programming activities. Proven leadership in building … matching engines, AI/GenAI services and large-scale data platforms. This role will drive transformation of enterprise data capabilities using Databricks, scala/spark and cloud native architectures, ensuring platform is stable, scalable, resilient, secure and AI enabled.* As platform landscape evolved towards cloud-native, AI enabled ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...