16 to 40 of 373 Apache Spark Jobs

Software Engineer for this role, you will collaborate with the founding team to expand the integration of our Big Data processing acceleration technology with Apache Spark to drive new optimizations and broader SQL operation coverage. Your contributions to our core solution will directly impact data infrastructure processing 10s … as batch processing code, data parsing, shuffling and data partitioning algorithms. Maintain the solution up to date and compatible with a variety of supported Apache Spark runtimes. Independently and diligently write, test and deploy production code driven by modern software engineering practices. Work with the internals of leading more »

Machine Learning Engineer - Zurich, Switzerland (Tech stack: Machine Learning Engineer, Python, TensorFlow, PyTorch, scikit-learn, Keras, Natural Language Processing (NLP), Computer Vision, Apache Spark, Jupyter Notebooks, Docker, AWS SageMaker, Azure Machine Learning, Google Cloud AI Platform, Pandas, NumPy, Matplotlib, Seaborn, XGBoost, LightGBM, OpenCV, Flask (for building APIs), FastAPI … training will be provided to fill any gaps in your skill set): Python, TensorFlow, PyTorch, scikit-learn, Keras, Natural Language Processing (NLP), Computer Vision, Apache Spark, Jupyter Notebooks, Docker, AWS SageMaker, Azure Machine Learning, Google Cloud AI Platform, Pandas, NumPy, Matplotlib, Seaborn, XGBoost, LightGBM, OpenCV, Flask (for building more »

of the company's data infrastructure. You will work with some of the most innovative tools in the market including Snowflake, AWS (Glue, S3), Apache Spark, Apache Airflow and DBT!! The role is hybrid, with 2 days in the office in central London and the company is more »

Data Engineer 6 Month Contract Inside IR35 £450/day Hiring Immediately Job Description (Apache Iceberg, Spark, Big Data) Job Details Overview: Overall IT experience of 5+ years of total experience with strong programming skills Excellent skill in Apache Iceberg, Spark, Big Data 3+ years of … Big Data project development experience Hands on experience in working areas like Apache Iceberg & Spark, Hadoop, Hive Must have knowledge in any Database Ex: Postgres, Oracle, MongoDB Excellent in SDLC Processes and DevOps knowledge (Jira, Jenkins pipeline) Working in Agile POD and with team collaboration Ability to participate more »

development (ideally AWS) and container technologies Strong communication and interpersonal skills Experience managing projects and working with external third party teams Ideally experience with Apache Spark or Apache Flink (but not essential) Please note, this role is unable to provide sponsorship. If this role sounds of interest more »

IaC), automation & configuration management; Ansible (plus Puppet, Saltstack), Terraform, CloudFormation; NodeJS, REACT/MaterialUI (plus Angular), Python, JavaScript; Big data processing and analysis, e.g. Apache Hadoop (CDH), Apache Spark; RedHat Enterprise Linux, CentOS, Debian or Ubuntu. Java 8, Spring framework (preferably Spring boot), AMQP RabbitMQ, Open source more »

Strong knowledge of SQL and experience with data analysis tools such as Excel, Tableau, and Power BI. Experience with data processing technologies such as Apache Spark and Apache Flink. Knowledge of statistical analysis techniques and experience with programming languages such as Python, R, and SAS. Strong problem more »

the UK). Role Overview: In this vital role, you will develop and maintain enterprise-grade software systems leveraging your expertise in Databricks, Python, Spark, R, and SQL. You will collaborate closely with our architecture team to design scalable, clean solutions that support continuous delivery and improvement. Your contributions more »

Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, Azure Stream Analytics Direct experience in building data pipelines using Azure Data Factory and Apache Spark (preferably Databricks). Experience building data warehouse solutions using ETL/ELT tools such as SQL Server Integration Services (SSIS), Oracle Data … Integrator (ODI), Talend, and Wherescape Red. Experience with Azure Event Hub, IOT Hub, Apache Kafka, Nifi for use with streaming data/event-based data Experience with other Open Source big data products eg Hadoop (incl. Hive, Pig, Impala) Experience with Open Source non-relational/NoSQL data repositories more »

automation. The Role As a Data Engineer with us, you will: Design, develop and maintain scalable and efficient data pipelines and ETL processes using Spark, ensuring high-quality data processing and integration. Collaborate with cross-functional teams to translate complex data requirements into actionable technical solutions. Utilize Microsoft Azure … highest standards in database design and operations. The Person We're looking for someone who is: Experienced as a Data Engineer, demonstrating proficiency in Apache Spark and cloud-based technologies, especially Microsoft Azure and Databricks. Skilled in programming, particularly Python, and familiar with data integration tools and ETL more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

develop, and maintain high volume Java and Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database Write efficient SQL for loading and querying data … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and open minded more »

powering machine-learning models. Have a strong background in developing distributed systems with experience in scalable data pipelines Familiar with big data technologies like Spark or Flink and comfortable in engineering data pipelines using big data technologies on financial datasets Experience with RESTful APIs and server-side APIs integration more »

and dealing with ad-hoc requests About You We’re looking for an experienced Data Engineer with excellent knowledge of Snowflake, AWS, Python, and Apache Airflow who is ready to lead by example and is used to rolling up their sleeves to get things done. The successful candidate must … 3NF and dimensional modelling, Kimball, DV 2.0 etc.) Strong experience in building robust and scalable ELT/ETL data pipelines Proficient coding in - python, Apache Spark and expert knowledge of SQL and good experience with shell-scripting languages Working knowledge of orchestration tools, e.g. Apache Airflow Experience … or consumer finance IaaC such as Terraform or AWS CloudFormation Knowledge of visualization tools, e.g. Tableau, Looker, Power BI, AWS QuickSight Exposure to streaming: Apache Kafka, AWS MSK Docker Understanding of SCRUM and Agile principles and collaboration tools like JIRA software and Confluence. What's in it for you more »

design, and ensuring low-latency, high-quality decisioning. Architect, deploy, and optimize Databricks solutions on AWS for streamlined data operations and enhanced deployments. Utilize Spark, Java, Python, Databricks, Tecton, and AWS services (S3, Redshift, EMR, Athena, Glue) for data processing implementations. Coach and mentor team members, fostering growth and … processing systems. Deep knowledge of data processing, comfortable with multi-terabyte datasets, skilled in high-scale data ingestion, transformation, and distributed processing, particularly with Apache Spark or Databricks. Proficient in building low-latency, high-availability data stores for real-time or near-real-time processing, using Python, Scala … Java, or JavaScript/TypeScript, as well as SQL and NoSQL for data retrieval. Hands-on expertise in data technologies including Spark, Airflow, Databricks, AWS services (SQS, Kinesis, etc.), and Kafka, with a knack for recommending tailored architectural solutions. Keen awareness of the latest trends in Data, ML, and more »

and product development, encompassing experience in both stream and batch processing. · Designing and deploying production data pipelines, utilizing languages such as Java, Python, Scala, Spark, and SQL. In addition, you should have proficiency or familiarity with: · Scripting and data extraction via APIs, along with composing SQL queries. · Integrating data more »

field (STEM) Technical proficiency in cloud-based data solutions (AWS, Azure or GCP), engineering languages including Python, SQL, Java, and pipeline management tools e.g., Apache Airflow. Familiarity with big data technologies, Hadoop, or Spark. If this opportunity is of interest, or you know anyone who would be interested in more »

develop, and maintain high volume Java or Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database. Excellent in writing efficient SQL for loading and … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing. Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Leader as team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and more »

develop, and maintain high volume Java or Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database. Excellent in writing efficient SQL for loading and … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing. Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Leader as team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and more »

develop, and maintain high volume Java or Scala based data processing jobs using industry standard tools and frameworks in the Hadoop ecosystem, such as Spark, Kafka, Hive, Impala, Avro, Flume, Oozie, and Sqoop Design and maintain schemas in our analytics database. Excellent in writing efficient SQL for loading and … technologies, languages, and techniques in the rapidly evolving world of high-volume data processing. Technologies We Use: Development languages/frameworks : Java/Scala, Apache Spark, Kafka, Vertica, JavaScript (React/Redux), MicroStrategy Amazon : EMR, Step Functions, SQS, LaMDA and AWS cloud-native architectures DevOps Tools : Terraform or … Cloud Formation, NewRelic, Jenkins, Grafana, PagerDuty, GitHub, GitHub Actions Database : MySQL, Vertica, DynamoDB Stream Processing : Kafka, Spark Streaming, Kinesis What We Look For: Ability to work within a dynamic team committed to excellence. Leader as team contributor to active discussion, meetings. Solid listening skills, ability to be flexible and more »