151 to 175 of 223 Permanent PySpark Jobs

scikit-learn, standard libraries, etc.), Python packages that wrap Machine Learning (packages for NLP, Object Detection, etc.), Linux, AWS/C2S, Apache NiFi, Spark, pySpark, Hadoop, Kafka, ElasticSearch, Solr, Kibana, neo4J, MariaDB, Postgres, Docker, Puppet, and many others. Work on this program takes place in Chantilly, VA, McLean, VA … each project including ETL mappings, code use guide, code location and access instructions. Required Skills Extract, Transform and Load (ETL) tools and processes Python, Pyspark, Pytorch AWS SQL APIs Linux Geospatial tools/data Desired Skills (Optional) Knowledge of agile methodologies/experience delivering on agile teams (Participates in More ❯

Are you passionate about revolutionising engineering with AI? Here at Monolith AI we're on a mission to empower engineers to use AI to solve even their most intractable physics problems. We've doubled in size over the last four More ❯

Responsibilities: Analyse source data from source databases (PostgreSQL) to understand structures, relationships and semantics. Design and implement streaming data pipelines using AWS EMR and PySpark to generate real-time (fast-moving) features for the feature store. Develop and maintain batch processing pipelines using DBT and BigQuery to generate batch … ensure efficient integration into Feast feature store. Requirements Good knowledge of programming languages such as Python or Java. Strong experience with streaming technologies (Spark, PySpark, Flink, KSQL or similar) for developing data transformation pipelines. Solid understanding and practical experience with SQL and relational databases (PostgreSQL preferred). Proficiency with More ❯

Data Engineer Manager Department: Tech Hub Employment Type: Permanent Location: London Reporting To: Lead Data Engineer Description Contract type: Permanent Hours: Full time, 37.5 hours per week Salary: circa £80,000 depending on experience Location: Canary Wharf WFH policy: Employees More ❯

across clustering, propensity modelling, regression, and NLP Providing insights on their customers, pricing strategies, and the target audience YOUR SKILLS AND EXPERIENCE Python/PySpark experience is essential to create propensity models and clustering NLP experience is a plus Commercial awareness and insights experience is needed for this role More ❯

both short and long-term projects across clustering, propensity modelling, regression, and NLP Occasionally building dashboards for clients YOUR SKILLS AND EXPERIENCE Python/PySpark experience is essential to create propensity models and clustering NLP experience is a plus Commercial awareness and insights experience is needed for this role More ❯

or techniques (e.g. algorithm or development) to study issues and solve problems, engineering (electrical or computer), and/or high performance computing Preferred Python & pySpark experience More ❯

and supports the development of data pipelines to support model development. Proficient with software tools that develop data pipelines in a distributed computing environment (PySpark, GlueETL). Supports integration of model pipelines in a production environment. Develops understanding of SDLC for model production. Reviews pipeline designs, makes data model More ❯

The Software Engineer will run build and work on enterprise grade software systems using a modern tech stack including PySpark with Databricks for data engineering tasks, infrastructure as code with AWS CDK and GraphQL. As a Software Engineer, you are expected to work with architects to design clean decoupled … experience. Some level of professional working experience. More if no relevant degree. OO and functional programming experience, design patterns, SOLID principles. Experience in Python, PySpark and/or SQL is preferred. Experience with scrum, TDD, BDD, Pairing, Pull Requests, Continuous Integration & Delivery. Continuous Integration tools - Github, Azure DevOps, Jenkins More ❯

Date: 21 Mar 2025 Location: Edinburgh, GB Macclesfield, GB Glasgow, GB Company: Royal London Group Contract Type: Permanent Location: Wilmslow or Edinburgh or Glasgow Working style: Hybrid 50% home/office based The Group Data Office (GDO) is responsible for More ❯

Date: 21 Mar 2025 Location: Edinburgh, GB Macclesfield, GB Glasgow, GB Company: Royal London Group Contract Type: Permanent Location: Wilmslow or Edinburgh or Glasgow Working style: Hybrid 50% home/office based The Group Data Office (GDO) is responsible for More ❯

Date: 21 Mar 2025 Location: Edinburgh, GB Macclesfield, GB Glasgow, GB Company: Royal London Group Contract Type: Permanent Location: Wilmslow or Edinburgh or Glasgow Working style: Hybrid 50% home/office based The Group Data Office (GDO) is responsible for More ❯

Date: 21 Mar 2025 Location: Edinburgh, GB Macclesfield, GB Glasgow, GB Company: Royal London Group Contract Type: Permanent Location: Wilmslow or Edinburgh or Glasgow Working style: Hybrid 50% home/office based The Group Data Office (GDO) is responsible for More ❯

We're seeking a talented and motivated Lead Data Platform Administrator to join our Data Platform and Engineering team. Our aim is to deliver a robust platform that enables Machine Learning and analytics to drive indemnity savings, efficiency benefits and More ❯

Location: Guernsey, Haywards Heath, Home Office (Remote) or Manchester Salary: £85,000 - £100,000 - depending on experience Department: Technology and Data We're First Central Insurance & Technology Group (First Central for short), an innovative, market-leading insurance company. We protect More ❯

Location: Guernsey, Haywards Heath, Home Office (Remote) or Manchester Salary: £85,000 - £100,000 - depending on experience Department: Technology and Data We're First Central Insurance & Technology Group (First Central for short), an innovative, market-leading insurance company. We protect More ❯

Location: Guernsey, Haywards Heath, Home Office (Remote) or Manchester Salary: £85,000 - £100,000 - depending on experience Department: Technology and Data We're First Central Insurance & Technology Group (First Central for short), an innovative, market-leading insurance company. We protect More ❯

JSON and Spark), and schema management. Key Skills and Experience: Strong understanding of complex JSON manipulation Experience with Data Pipelines using custom Python/PySpark frameworks Knowledge of the 4 core Data categories (Reference, Master, Transactional, Freeform) and handling Reference Data Understanding of Data Security principles, access controls, GDPR … browser-based IDEs like Jupyter Notebooks Familiarity with Agile methodologies (SAFE, Scrum, JIRA) Languages and Frameworks: JSON YAML Python (advanced proficiency, Pydantic bonus) SQL PySpark Delta Lake Bash Git Markdown Scala (bonus) Azure SQL Server (bonus) Technologies: Azure Databricks Apache Spark Delta Tables Data processing with Python PowerBI (Data More ❯

Azure data engineering and cloud-native development. Showcase strong, hands-on experience with Microsoft Fabric, including Lakehouse (Delta format), OneLake, Pipelines & Dataflows Gen2, Notebooks (PySpark), Power BI & Semantic Models. Possess a solid understanding of data integration patterns, ETL/ELT, and modern data architectures. Be familiar with CI/… CD practices in a data engineering context. Have excellent SQL and Spark (PySpark) skills. Have obtained experience in working with large, complex datasets and performance optimization. Exhibit strong leadership and communication skills. Demonstrate previous experience in leading and mentoring a technical team. Be Microsoft Certified (Azure Data Engineer Associate More ❯

Data is the new Oil Are you passionate about data? Does the prospect of dealing with massive volumes of data excite you? Do you want to build big-data solutions that process billions of records a day in a scalable More ❯

Lead Data Engineer (Python, PySpark, DynamoDB) (Enterprise Platforms Technology)Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers More ❯

Lead Data Engineer (Python, PySpark, DynamoDB) (Enterprise Platforms Technology)Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers More ❯

Job Title : Pyspark Developer Location : Boston, MA Duration : Fulltime Infosys is seeking a Data Engineer- Pyspark Developer. In this role, you will enable digital transformation for our clients in a global delivery model, research on technologies independently, recommend appropriate solutions and contribute to technology-specific best practices and … encouraged to apply. At least 4 years of Information Technology experience At least 3+ years of experience in Hadoop platform 3+ years' experience in PySpark development 4 years minimum experience using Python 4+ years of Azure or AWS cloud platform experience At least 2 years of Hands-on experience More ❯

to London) Duration: 9Months+ Contract Inside IR35 Role Summary: We are seeking a versatile and experienced Data Engineer with a strong foundation in Python , PySpark , and modern data platforms. This role demands hands-on experience with CI/CD automation , unit testing , and working within Azure environments — both through … testing • OpenTelemetry (exposure) • Poetry • VS Code, Dev Containers • SQL Querying • CI/CD tools • ADO/GitLab • Pipelines for automation Data Engineering (Highly desirable) • PySpark • SparkSQL • Data file formats like Delta, parquet Fabric (Not absolutely required but desirable) • Fabric Notebooks • Data Factory pipelines • Kusto • Data Flow Gen 2 Generalist More ❯

and manage data solutions that align with business needs and industry standards. The ideal candidate will have expertise in Java, SQL, Python, and Spark (PySpark & SparkSQL) while also being comfortable working with Microsoft Power Platform. Experience with Microsoft Purview is a plus. The role requires strong communication skills to … data standards. Key Responsibilities: 1. Data Architecture & Engineering Design and implement scalable data architectures that align with business objectives. Work with Java, SQL, Python, PySpark, and SparkSQL to build robust data pipelines. Develop and maintain data models tailored to organizational needs. Reverse-engineer data models from existing live systems. More ❯