Lead Software Engineer - LLM Ops Platform Reliability

Job Description

hackajob is partnering directly with JPMorganChase to hire for this role. JOB DESCRIPTIONHelp shape how AI systems run reliably in production at scale. In this role, you'll build and operate large language model serving infrastructure, bringing strong engineering fundamentals and site reliability practices to cutting-edge AI platforms. You'll work hands-on with cloud and Kubernetes-based deployments, deep observability, and cost-aware performance tuning. If you enjoy solving hard production problems and making platforms measurably better, you'll find meaningful impact and growth here.

As a Lead Software Engineer at JPMorgan Chase in the AI and Machine Learning Platform team, you will build and scale AI infrastructure that modernizes traditional infrastructure management and site reliability engineering through applied AI. You will own the reliability, performance, and cost-efficiency of the LLM inference platform end to end. You will operate large language model serving stacks (such as vLLM and llm-d) in production at scale, with deep instrumentation and strong operational rigor. You will partner across engineering to deliver secure software, improve stability, and lead incident response and continuous improvement.

Job ResponsibilitiesDesign, develop, troubleshoot, and deliver secure, high-quality production software and services for AI infrastructureBuild backend services and APIs that enable reliable operation of AI infrastructure in productionOperate and scale LLM serving infrastructure (such as vLLM and llm-d), including model hosting, request routing, continuous batching, and KV-cache optimizationDeploy, host, and lifecycle-manage open-source and proprietary LLMs on Amazon EKS and Amazon SageMaker, as well as on-prem and local GPU clusters, using reproducible infrastructure as code and continuous delivery pipelinesImplement observability (logs, metrics, traces) with dashboards and actionable alerting, including Prometheus metrics and Grafana/Alertmanager integration for LLM and GPU workloadsTune GPU and accelerator capacity, autoscaling, and cost efficiency for LLM inference workloads using performance and optimization techniques (e.g., quantization, parallelism, speculative decoding)Lead reliability engineering for LLM endpoints through capacity planning, load/soak testing, safe rollouts (blue/green, canary), failover, and incident response for outages and model-quality regressionsParticipate in an on-call rotation, lead incident triage and mitigation, and produce clear post-incident root-cause analyses and follow-upsIdentify recurring operational issues and automate remediation to improve platform stability and developer experienceBuild and maintain multi-agent systems with strong orchestration (planning, coordination, tool-calling, state/memory, and workflow control) where appropriateContribute to an inclusive team culture grounded in diversity, opportunity, inclusion, and respect, and help drive adoption of leading-edge technologies through communities of practiceRequired Qualifications, Capabilities, and SkillsFormal training, certification, or equivalent practical experience in software engineering conceptsHands-on experience with system design, application development, testing, and operational stability in production environmentsAdvanced proficiency in Python for building production-grade services and toolingProficiency with automation and continuous delivery methodsHands-on experience with AWS and Terraform for infrastructure delivery and lifecycle managementStrong understanding of site reliability engineering practices, including incident management, root-cause analysis, runbooks, and reliability patternsPractical knowledge of observability and instrumentation across metrics, logs, and tracesComfort with on-call operations and production troubleshootingHands-on production experience operating LLM inference servers such as vLLM and llm-d (or directly equivalent serving stacks)Hands-on experience hosting and serving LLMs on Amazon EKS and/or Amazon SageMaker, and on local GPU infrastructureKnowledge of LLM reliability and risk considerations, including latency/throughput trade-offs, model and weight versioning, prompt/response logging, and safe rollout patternsPreferred Qualifications, Capabilities, and SkillsExperience developing generative AI applications, AI agents, vector search, and retrieval-augmented generation patternsExperience building AI agents using frameworks such as LangChain, CrewAI, LangGraph, or similar orchestration platformsExperience operating or integrating serving platforms such as KServe, Ray Serve, NVIDIA Triton Inference Server, Text Generation Inference (TGI), alongside vLLM/llm-dFamiliarity with Amazon SageMaker JumpStart, SageMaker Endpoints, and Amazon Bedrock for managed model hostingExperience with online LLM quality monitoring (e.g., hallucination, toxicity, drift detection) and tracing via OpenTelemetry conventionsContributions to open-source LLM serving or inference projects (e.g., vLLM, llm-d, Ray, KServe, Triton)ABOUT USJ.P. Morgan is a global leader in financial services, providing strategic advice and products to the world's most prominent corporations, governments, wealthy individuals and institutional investors. Our first-class business in a first-class way approach to serving clients drives everything we do. We strive to build trusted, long-term partnerships to help our clients achieve their business objectives.

We recognize that our people are our strength and the diverse talents they bring to our global workforce are directly linked to our success. We are an equal opportunity employer and place a high value on diversity and inclusion at our company. We do not discriminate on the basis of any protected attribute, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis protected under applicable law.

We also make reasonable accommodations for applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. Visit ourFAQsfor more information about requesting an accommodation. ABOUT THE TEAMOur professionals in our Corporate Functions cover a diverse range of areas from finance and risk to human resources and marketing. Our corporate teams are an essential part of our company, ensuring that we're setting our businesses, clients, customers and employees up for success.