1 to 25 of 90 Apache Spark Jobs in England

Spark Scala Developer - Scala/Apache Spark - Hybrid/Leeds - £450-£550 Spark Scala Developer to join our client, one of the biggest financial services organizations in the world, with operations in more than 38 countries. It has an IT infrastructure of 200,000+ servers … 000+ database instances, and over 150 PB of data. As a Spark Scala Engineer, you will have the responsibility to refactor Legacy ETL code, for example DataStage into PySpark using Prophecy low-code no-code and available converters. Converted code is causing failures/performance issues. Your responsibilities; As … a Spark Scala Engineer you will be working for the GDT (Global Data Technology) Team, you will be responsible for: Designing, building, and maintaining data pipelines using Apache Spark and Scala Working on an Enterprise scale Cloud infrastructure and Cloud Services in one of the Clouds (GCP more »

Spark/PySpark Architect - 12 months+ -£Inside IR35- Hybrid working of 3 days on site in Leeds My client are a Global Consultancy who are looking for a number of Spark/PySpark Architects to join them on a Long term programme. As the Spark architect, you … Integration upgrade to PySpark Collaboration with multiple customer stakeholders Knowledge of working with Cloud Databases Excellent communication and solution presentation skills. Able to analyse Spark code failures through Spark Plans and make correcting recommendations Able to review PySpark and Spark SQL jobs and make performance improvement recommendations … Able to understand Data Frames/Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations Able to monitor Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures. As a Spark architect, who can demonstrate deep knowledge more »

stack is not essential, but it would make things easier. The main technologies we use include: Elixir, Erlang, Python, Terraform, Ansible, Packer, EMR/Spark, Apache Spark, Apache Druid, BigQuery and Redis Familiarity with cloud technologies. Ideally AWS and technologies such as EC2, ECS, EMR, AWS … Lambda, DynamoDB, S3, Kinesis, SQS, SES, Cloudwatch.... Knowledge or experience with big data technologies such as Spark, Hadoop, Redshift, Snowflake, Kafka, Flink, Druid, Clickhouse... is highly desirable. Benefits Flexible work arrangements to support a healthy work-life balance 25 vacation days (excluding bank holidays) and 5 days of carers more »

Data Scientists and Service Engineering teams Experience with design, development and operations that leverages deep knowledge in the use of services like Amazon Kinesis, Apache Kafka, Apache Spark, Amazon Sagemaker, Amazon EMR, NoSQL technologies and other 3rd parties Develop and define key business questions and to build … a related field Experience of Data platform implementation, including 3 years of hands-on experience in implementation and performance tuning Kinesis/Kafka/Spark/Storm implementations Experience with analytic solutions applied to the Marketing or Risk needs of enterprises Basic understanding of machine learning fundamentals Ability to … take Machine Learning models and implement them as part of data pipeline IT platform implementation experience Experience with one or more relevant tools ( Flink, Spark, Sqoop, Flume, Kafka, Amazon Kinesis) Experience developing software code in one or more programming languages (Java, JavaScript, Python, etc) Current hands-on implementation experience more »

to join Ordnance Survey's Data Derivation team in developing our next generation geospatial data. Using big data technologies (primarily Databricks, an implementation of Apache Spark) you will develop, support, and maintain automated processes that enrich and transform Ordnance Survey's large-scale data. Working in a forward … testing approaches and best practice Providing technical coaching and mentoring Advocating software engineering industry best practice Interpreting detailed specifications and translating into simple solutions Apache Spark/Databricks experience (desirable) Experience of working in Microsoft Azure (desirable) Experience of working with the Scala language (desirable) The rewards We more »

ML libraries (TensorFlow, pytorch, scikit-learn, transformers, XGBoost, ResNet), geospatial libraries (shapely, geopandas, rasterio), CV libraries (scikit-image, OpenCV, yolo, Detectron2). AWS, Postgres, Apache Airflow, Apache kafka, Apache Spark Mandatory requirements: You have at least 5 years of experience in the DS role, deploying models … You have expertise on applications focusing on geospatial data and mobility analytics (highly desirable). You have proven experience with big data technologies, specifically Spark and Kafka. You have experience working with state-of-art ML pipeline technologies (such as MLflow, Sagemaker...) or building a ML pipeline by yourself more »

development (ideally AWS) Knowledge and ideally hands-on experience with data streaming, event-based architectures and Kafka Strong communication and interpersonal skills Experience with Apache Spark or Apache Flink would be ideal, but not essential Please note, this role is unable to provide sponsorship. If this role more »

Spark Scala Developer - Scala/Apache Spark - Hybrid/Leeds - £450-£550 Spark Scala Developer to join our client, one of the biggest financial services organizations in the world, with operations in more than 38 countries. It has an IT infrastructure of 200,000+ servers … 000+ database instances, and over 150 PB of data. As a Spark Scala Engineer, you will have the responsibility to refactor L... more »

business value in collaboration with an Account Executive and a Senior Solutions Architect. Gain excitement from clients about Databricks through hands-on evaluation and Spark programming, integrating with the wider cloud ecosystem and 3rd party applications. Contribute to building the Databricks technical through engagement at workshops, seminars, and meet … data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter ,LinkedIn and Facebook . Our Commitment to Diversity and Inclusion At more »

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands more »

in backend development or data engineering. Hands-on experience with AWS services like S3, EC2, and EMR. Proficiency in SQL and experience with CDAP, Spark, and Kafka. Experience building scalable ETL processes and workflows. Strong programming ability with Python, Java, and unit testing. Infrastructure-as-code expertise with CI more »

DataBricks, Azure SQL (Indicative experience = 5yrs+) Build and test processes supporting data extraction, data transformation, data structures, metadata, dependency and workload management. Knowledge on Spark architecture and modern Datawarehouse/Data-Lake/Lakehouse techniques Build transformation tables using SQL. Moderate level knowledge of Python/PySpark or equivalent more »

Data mining, Data warehousing, ETL Experience in handling large volumes of data on SQL, NoSQL and Big Data databases Experience in Hadoop ecosystem: Hadoop, Spark, Hive, and/or Scala Experience in programming languages: PHP, Python, C++/Java Experience in Web development in Laravel MVC Framework Comfortable working more »

AWS Redshift, and Python Experience with ETL processes, data integration, and data warehousing. Strong SQL skills Experience with Big Data technologies such as Hadoop, Spark, and Kafka Familiarity with cloud platforms (AWS, Azure, Google Cloud) Working knowledge of data visualisation tools (PowerBI, Tableau, Qlik Sense) Additional Skills: Client-facing more »

platforms - preferably in GCP - and experience with container orchestration technologies such as Kubernetes. Strong background in distributed computing and familiarity with technologies like Hadoop, Spark, Kafka, and distributed cache systems (Hazelcast, Redis). Experience with database management and proficiency in SQL and NoSQL databases. Knowledge of monitoring and logging more »

Comfort with rapid prototyping and disciplined software development processes Experience with Python, ML libraries (e.g. spaCy, NumPy, SciPy, Transformers, etc.)data tools and technologies (Spark, Hadoop, Hive, Redshift, SQL), and toolkits for ML and deep learning (SparkML, Tensorflow, Keras) Demonstrated ability to work on multi-disciplinary teams with diverse more »

ADF, SSIS) Data governance (Purview, Unity Catalogue) Databricks Delta Lake Storage Azure Dev OPS DESIRED SKILLS Advanced Analytics Data Technologies Databricks, Delta Lake, Synapse Spark SQL, Pyspark Azure Data Explorer Logic Apps, Key Vault Semi structured data processing Integration Runtime Coding experience: Python, C#, Java for Data analysis purpose more »

data from a wide variety of corporate structured and unstructured sources, 3rd party providers and publicly available data. Leverage cloud data processing frameworks like Apache Spark, Databricks, or similar for handling, processing and analysing complex, high-volume data both structured and unstructured. Ability to script repetitive data tasks more »

environment. Experience in log management tools to troubleshoot issues as well as identify useful analytics data. Preferred Experience in Microsoft Azure services and Databricks Spark, Redshift, Hadoop Map-Reduce or other Big Data frameworks Code management tools (Git, Sbt, Maven) Pyspark, Scala or other functional programming languages Analytics tools more »

You are someone who should have good expertise in data on cloud (AWS Preferable), and someone who can demonstrate experience around distributed frameworks like APACHE SPARK & modern OLAP platforms like Snowflake. You will be looking after a team of 4, so experience of being able to motivate, manage … experience in modern OLAP technologies like Snowflake & Lake house implementation with AWS integrations. Good to have experience in the streaming side of technology like Apache Kafka, Kinesis, and no SQL databases like Dynamo DB, demonstrable experience in PYTHON with frameworks like SPARK & good understanding on SNOWPARK & SNOWSQL. Experience … around Scheduling tool like Apache AIRFLOW or AWS managed MWAA, is a plus. Proven Leadership and management experience; previously worked within financial services sector (preferred, not essential). What you’ll get for this role: Our purpose - with you today, for a better tomorrow – is a promise we make more »

the financial services or hedge fund industry.Technical Skills:Proficiency in Python and SQL.Experience with relational and NoSQL databases.Knowledge of big data frameworks (e.g., Hadoop, Spark, Kafka).Understanding of financial markets and trading systems.Strong analytical, problem-solving, and communication skills.Familiarity with DevOps tools and practices.This is an exciting opportunity for more »

data security principles. Desirable skillsets - Docker, Kubernetes - CI/CD pipelines & IaC. - Certified in one or more: Databricks or AWS. - Kafka and/or Spark more »

or hedge fund industry. Technical Skills: Proficiency in Python and SQL. Experience with relational and NoSQL databases. Knowledge of big data frameworks (e.g., Hadoop, Spark, Kafka). Understanding of financial markets and trading systems. Strong analytical, problem-solving, and communication skills. Familiarity with DevOps tools and practices. This is more »

tooling Scripting experience (Python, Perl, Bash, etc.) ELK (Elastic stack) JavaScript Cypress Linux experience Search engine technology (e.g., Elasticsearch) Big Data Technology experience (Hadoop, Spark, Kafka, etc.) Microservice and cloud native architecture Desirable Skills Able to demonstrate experience of troubleshooting and diagnosis of technical issues. Able to demonstrate excellent more »

or similar, with 6+ years of professional experience. A good understanding of modern lakehouse architectures and corresponding technologies, such as Dremio, Snowflake, Iceberg, (Py)Spark/Glue/EMR, dbt and Airflow/Dagster. Experience with Cloud providers. Familiarity with AWS S3, ECS and EC2/Fargate would be more »