It would be better to write a test case for this URI to verify whether it meets expectations when it is genuinely available and when it is not.

Honestly, I tried, but I don't know how to simulate a failure that keeps the inference service down without affecting the web service to verify the healthcheck.

I tried causing exceptions manually, but the service quit immediately. Then I reverted the code to v0.2.5, then raise an Exception at the same line as #853, but it still quit immediately.

Maybe I need some help about the unittest.😢

Maybe you can use pytest.raises?

I needed this feature. We've been using POST /v1/completions {"model":" ","prompt":" ","max_tokens":1} for custom health check endpoint for autoscaler/autohealer at our infra. Once this merges, we can finally and reliably use GET /health (or /v1/health) for health checking.

One little concern is under high load, /health could timeout even if it is actually running. Maybe in the future, we can design sophisticated health check logic without needing to actually run the inference? but I think it's a good start.

One little concern is under high load, /health could timeout even if it is actually running.

@vhain Yeah, I have considered this. Maybe for healthcheck (livenessProbe in k8s), we could set a long timeout, and for load banlance ready check (readinessProbe in k8s), we could set a short timeout.

I thought that both healthcheck and ready check timeout should be determined based on various factors such as different model, devices and context lengths, which making it difficult to find a fixed value.

Could you please share some insights or experiences regarding timeout?

@LucienShui I think I need your insight as well 😅 I think we can set distinct timeout for each hardware config, and model settings as you mentioned.

In our case, we've been just constantly monitoring the latency for 1 token generation, and found the p95 to be around 5s, so we've set the timeout to 10s. However turned out sometimes it was not enough, so we just bumped up to 30s.

If it's taking more than 30s to generate 1 token, it's bad anyways, so we have the health check to trigger our autoscaler to scale out. If it's timing out for over threshold iterations (currently 2), we mark the instance unhealthy and replace it with the new one.

I think there is no right or wrong answer for LLM backend autoscaling/autohealing at the moment, as things are pretty in early stage.

@LucienShui I actually got an idea. Maybe we can add a periodic check (1 token generation; for every 10s or so - configurable by server args), running inside of the SGLang itself, cache the result, and use it for /health serving?

This way we can make sure that calling /health multiple times does not hurt, also it responds very quickly.

@vhain I have considered this approach, but it has the following issues:

1. The health check results might represent a historical health status under high load.
2. If let user control the internal loop interval, it would be better to let the user decide the health check interval directly.
3. Additionally, its implementation and runtime complexity are higher.

Great discussions! Can we merge this first and iterate on better designs in follow-up PRs?
For this PR, please create a separate endpoint and we can merge it.

@merrymercy Yeah, sure! I'm glad it can be merged.

I'll trying to add some test cases in another PR.

@LucienShui Thanks for your contribution!