ModelService is a Helm chart that simplifies LLM deployment on llm-d by declaratively managing Kubernetes resources for serving base models. It enables reproducible, scalable, and tunable model deployments through modular presets, and clean integration with llm-d ecosystem components (including vLLM, Gateway API Inference Extension, LeaderWorkerSet). It provides an opinionated but flexible path for deploying, benchmarking, and tuning LLM inference workloads.

The ModelService Helm Chart proposal is accepted on June 10, 2025. Read more about the roadmap, motivation, and other alternatives considered here.

TL;DR:

Active scearios supported:

• P/D disaggregation
• Multi-node inference, utilizing data parallelism
• One pod per node (see llm-d-infra for the ModelService values file)
• One pod per DP rank

Integration with llm-d components:

• Quickstart guide in llm-d-infra depends on ModelService
• Flexible configuration of llm-d-inference-scheduler for routing
• Features llm-d-routing-sidecar in P/D disaggregation
• Utilized in benchmarking experiments in llm-d-benchmark
• Effortless use of llm-d-inference-sim for CPU-only workloads

Add this repository to Helm.

helm repo add llm-d-modelservice https://llm-d-incubation.github.io/llm-d-modelservice/
helm repo update

ModelService operates under the assumption that llm-d-infra has been installed in a Kuberentes cluster, which installs the required prerequisites and CRDs. Read the llm-d-infra Quickstart for more information.

At a minimal, follow these steps to install the required external CRDs as the ModelService helm chart depends on them.

Note that in order to create HTTPRoute objects last, Helm hooks are used. As a consequence, these objects are not deleted when helm delete is executed. They should be manually deleted to avoid unexpected routing problems.

See examples for how to use this Helm chart. Some examples contain placeholders for components such as the gateway name. Use the --set flag to override placeholders. For example,

helm install cpu-only llm-d-modelservice -f examples/values-cpu.yaml --set prefill.replicas=0 --set "routing.parentRefs[0].name=MYGATEWAY"

Check Helm's official docs for more guidance.

Below are the values you can set.

We welcome contributions to llm-d-modelservice! Please see our Contributing Guide for detailed information on how to contribute to this project, including guidelines for submitting issues, pull requests, and development setup.

Please open a ticket if you see a gap in your use case as we continue to evolve this project.

Get involved or ask questions in the #sig-model-service channel in the llm-d Slack workspace! Details on how to join the workspace can be found here.