chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM #1934

keivenchang · 2025-07-15T00:47:33Z

Overview:

Enabled /live and /health (when DYN_SYSTEM_ENABLED=true), and renamed from "HTTP_SERVER_" to "SYSTEM_" names as to not get confused with client HTTP requests.

Details:

Renamed constants: DYN_HTTP_SERVER_* → DYN_SYSTEM_*
Updated method: http_server_enabled() → system_server_enabled()
Updated all docs/comments to use "system server" terminology, removed mentions of "HTTP"
Added tests.

Where should the reviewer start?

lib/runtime/src/config.rs - Main changes
lib/runtime/src/distributed.rs - Updated references

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

DIS-239

Summary by CodeRabbit

New Features
- Added a new /live endpoint to the system server for liveness checks.
- Introduced a fallback route that returns a 404 status for unmatched paths.
Refactor
- Renamed HTTP server configuration and environment variables to use "system server" terminology.
- Updated related documentation and log messages to reflect the new naming.
- Improved server startup to provide clearer logging and asynchronous operation.
Tests
- Added tests for the new /live endpoint and fallback route behavior.

copy-pr-bot · 2025-07-15T00:47:37Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

coderabbitai · 2025-07-15T00:51:52Z

Walkthrough

The changes rename all HTTP server configuration and code references to "system server" throughout the runtime library, updating constants, struct fields, environment variables, and documentation accordingly. The HTTP server logic is refactored to use a new spawn_http_server function, which now spawns the server as a background task and returns its address and handle. Additional endpoints and improved logging are introduced. Tests and dependencies are updated to match the new naming and functionality.

Changes

File(s)	Change Summary
lib/runtime/Cargo.toml	Added `reqwest` as a dev-dependency with the "json" feature.
lib/runtime/src/config.rs	Renamed all HTTP server config fields, constants, env vars, and docs to "system server"; updated related tests.
lib/runtime/src/distributed.rs	Updated to use new system server config fields and `system_server_enabled`; refactored to use `spawn_http_server`.
lib/runtime/src/http_server.rs	Renamed and refactored server startup to `spawn_http_server`; added `/live` endpoint, fallback route, and improved logging; added async test for endpoints.

Sequence Diagram(s)

sequenceDiagram
    participant DistributedRuntime
    participant Config
    participant HTTPServer

    DistributedRuntime->>Config: Check system_server_enabled()
    alt Enabled
        DistributedRuntime->>HTTPServer: spawn_http_server(host, port, ...)
        HTTPServer->>DistributedRuntime: Return (address, join_handle)
        DistributedRuntime->>DistributedRuntime: Log server started
    else Disabled
        DistributedRuntime->>DistributedRuntime: Log server disabled
    end

Poem

A hop and a skip, the server’s renamed,
From HTTP to system, its purpose proclaimed.
New endpoints now live, with logging that sings,
Tests and configs updated—such wonderful things!
With spawn in its step, the server runs free,
A rabbit’s delight in pure clarity.
🐇✨

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 2

FIXED: 🧹 Nitpick comments (1)

lib/runtime/src/config.rs (1)
22-22: Minor: Inconsistent terminology in comment.

The comment mentions "system-service" while the rest of the codebase uses "system server" terminology.
-    /// Grace shutdown period for the system-service.
+    /// Grace shutdown period for the system server.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9e76590 and f3ff3fe.

⛔ Files ignored due to path filters (1)

Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

lib/runtime/Cargo.toml (1 hunks)
lib/runtime/src/config.rs (7 hunks)
lib/runtime/src/distributed.rs (1 hunks)
lib/runtime/src/http_server.rs (2 hunks)

🧰 Additional context used

🧠 Learnings (3)

lib/runtime/src/distributed.rs (1)

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

lib/runtime/src/http_server.rs (2)

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

Learnt from: oandreeva-nv
PR: ai-dynamo/dynamo#1195
File: lib/llm/tests/block_manager.rs:150-152
Timestamp: 2025-06-02T19:37:27.666Z
Learning: In Rust/Tokio applications, when background tasks use channels for communication, dropping the sender automatically signals task termination when the receiver gets `None`. The `start_batching_publisher` function in `lib/llm/tests/block_manager.rs` demonstrates this pattern: when the `KVBMDynamoRuntimeComponent` is dropped, its `batch_tx` sender is dropped, causing `rx.recv()` to return `None`, which triggers cleanup and task termination.

lib/runtime/src/config.rs (1)

Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.898Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.

🔇 Additional comments (5)

lib/runtime/src/distributed.rs (1)

81-103: LGTM! Server configuration and startup logic properly updated.

The changes correctly implement the renaming from HTTP server to system server, including:

Updated method calls and configuration field names

Improved async server management with the new spawn_http_server signature

Appropriate error handling and logging

lib/runtime/src/http_server.rs (3)

73-137: Well-structured refactoring to async server spawning.

The changes improve the HTTP server implementation:

Proper async task spawning pattern

Better error handling and logging

Returns both address and handle for improved control

92-97: Good addition of the /live endpoint.

Using the same handler for both /health and /live endpoints is a common pattern for Kubernetes readiness and liveness probes.

260-304: Comprehensive test coverage for HTTP endpoints.

The test properly validates:

All endpoint responses (/health, /live, and 404 fallback)

Server lifecycle management

Graceful shutdown handling

lib/runtime/src/config.rs (1)

128-144: Good implementation of environment variable mapping.

The mapping logic correctly transforms DYN_SYSTEM_* environment variables to the appropriate struct fields.

lib/runtime/Cargo.toml

lib/runtime/src/config.rs

…code behavior

nnshah1

@grahamking for viz

ZichengMa

May I know why we rename HTTP to System? System seems to be a bigger word.

keivenchang · 2025-07-15T03:59:47Z

@ZichengMa this was discussed in a meeting today, where people may confuse your DYN_HTTP_SERVER_* with the actual HTTP server that's used on the HTTP frontend. So to separate them out, we looked into: DYN_METRICS_AND_STATS_SERVER_THAT_IS_NOT_USED_IN_THE_FRONTEND and combinations of, and finally decided on DYN_SYSTEM_{PORT,HOST,ENABLED}.

grahamking · 2025-07-15T15:20:37Z

lib/runtime/Cargo.toml

+[patch.crates-io]
+tokio  = { version = "1.18.4" }   # addresses RUSTSEC-2023-0001
+h2     = { version = "0.4.4"   }   # addresses RUSTSEC-2024-0332
+rustls = { version = "0.23.18" }   # addresses RUSTSEC-2024-0399


warning: patch for the non root package will be ignored, specify patch at the workspace root

Introduced here: #1934 `cargo deny` normally prevents this but it wasn't being enforced last night. We avoid OpenSSL because: - It needs to be dynamically linked, so we have to build for a specific version. - We have to stay on top of CVEs. The Rust TLS is considered more secure. - It interferes with other libraries trying to do static linking. For example we cannot statically link `llama.cpp` if we dynamically link OpenSSL.

grahamking · 2025-07-15T15:58:15Z

lib/runtime/Cargo.toml

@@ -77,5 +77,13 @@ socket2 = { version = "0.5.8" }
 [dev-dependencies]
 assert_matches = { version = "1.5.0" }
 env_logger = { version = "0.11" }
+reqwest = { version = "0.12.22", features = ["json"] }


I'm trying to avoid us having a dependency on OpenSSL. Fix is here: #1945

grahamking · 2025-07-15T16:14:43Z

lib/runtime/src/http_server.rs

@@ -239,4 +263,64 @@ mod tests {
        assert!(response.contains("dynamo_runtime_uptime_seconds"));
        assert!(response.contains("Total uptime of the DistributedRuntime in seconds"));
    }
+
+    #[tokio::test]
+    async fn test_spawn_http_server_endpoints() {


---- http_server::tests::test_spawn_http_server_endpoints stdout ---- thread 'http_server::tests::test_spawn_http_server_endpoints' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.46.0/src/runtime/blocking/shutdown.rs:51:21: Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Could you take a look?

…1934) Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

commit e330d96 Author: Yan Ru Pei <yanrpei@gmail.com> Date: Fri Jul 18 13:40:54 2025 -0700 feat: enable / disable chunked prefill for mockers (#2015) Signed-off-by: Yan Ru Pei <yanrpei@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> commit 353146e Author: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Date: Fri Jul 18 13:33:36 2025 -0700 feat: add vLLM v1 multi-modal example. Add llama4 Maverick example (#1990) Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Co-authored-by: krishung5 <krish@nvidia.com> commit 1f07dab Author: Jacky <18255193+kthui@users.noreply.github.com> Date: Fri Jul 18 13:04:20 2025 -0700 feat: Add migration to LLM requests (#1930) commit 5f17918 Author: Tanmay Verma <tanmayv@nvidia.com> Date: Fri Jul 18 12:59:34 2025 -0700 refactor: Migrate to new UX2 for python launch (#2003) commit fc12436 Author: Graham King <grahamk@nvidia.com> Date: Fri Jul 18 14:52:57 2025 -0400 feat(frontend): router-mode settings (#2001) commit dc75cf1 Author: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Date: Fri Jul 18 18:47:28 2025 +0200 chore: Move NIXL repo clone to Dockerfiles (#2009) commit f6f392c Author: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Date: Thu Jul 17 18:44:17 2025 -0700 Remove link to the fix for disagg + eagle3 for TRT-LLM example (#2006) Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> commit cc90ca6 Author: atchernych <atchernych@nvidia.com> Date: Thu Jul 17 18:34:40 2025 -0700 feat: Create a convenience script to uninstall Dynamo Deploy CRDs (#1933) commit 267b422 Author: Greg Clark <grclark@nvidia.com> Date: Thu Jul 17 20:44:21 2025 -0400 chore: loosed python requirement versions (#1998) Signed-off-by: Greg Clark <grclark@nvidia.com> commit b8474e5 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Thu Jul 17 16:35:05 2025 -0700 chore: update cmake and gap installation and sgl in wideep container (#1991) commit 157a3b0 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 15:38:12 2025 -0700 fix: incorrect helm upgrade command (#2000) commit 0dfca2c Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 15:33:33 2025 -0700 ci: Update trtllm gitlab triggers for new components directory and test script (#1992) commit f3fb09e Author: Kris Hung <krish@nvidia.com> Date: Thu Jul 17 14:59:59 2025 -0700 fix: Fix syntax for tokio-console (#1997) commit dacffb8 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 14:57:10 2025 -0700 fix: use non-dev golang image for operator (#1993) commit 2b29a0a Author: zaristei <zaristei@berkeley.edu> Date: Thu Jul 17 13:10:42 2025 -0700 fix: Working Arm Build Dockerfile for Vllm_v1 (#1844) commit 2430d89 Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 12:57:46 2025 -0700 test: Add trtllm kv router tests (#1988) commit 1eadc01 Author: Graham King <grahamk@nvidia.com> Date: Thu Jul 17 15:07:41 2025 -0400 feat(runtime): Support tokio-console (#1986) commit b62e633 Author: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Date: Thu Jul 17 11:16:28 2025 -0700 feat: support separate chat_template.jinja file (#1853) commit 8ae3719 Author: Hongkuan Zhou <tedzhouhk@gmail.com> Date: Thu Jul 17 11:12:35 2025 -0700 chore: add some details to dynamo deploy quickstart and fix deploy.sh (#1978) Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com> Co-authored-by: julienmancuso <161955438+julienmancuso@users.noreply.github.com> commit 08891ff Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 10:57:42 2025 -0700 fix: Update trtllm tests to use new scripts instead of dynamo serve (#1979) commit 49b7a0d Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Thu Jul 17 08:35:04 2025 -0600 feat: record + analyze logprobs (#1957) commit 6d2be14 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 00:17:58 2025 -0700 refactor: replace vllm with vllm_v1 container (#1953) Co-authored-by: alec-flowers <aflowers@nvidia.com> commit 4d2a31a Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Wed Jul 16 18:04:09 2025 -0700 chore: add port reservation to utils (#1980) commit 1e3e4a0 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Wed Jul 16 15:54:04 2025 -0700 fix: port race condition through deterministic ports (#1937) commit 4ad281f Author: Tanmay Verma <tanmayv@nvidia.com> Date: Wed Jul 16 14:33:51 2025 -0700 refactor: Move TRTLLM example to the component/backends (#1976) commit 57d24a1 Author: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed Jul 16 14:10:24 2025 -0700 build: Removing shell configuration violations. It's bad practice to hardcod… (#1973) commit 182d3b5 Author: Graham King <grahamk@nvidia.com> Date: Wed Jul 16 16:12:40 2025 -0400 chore(bindings): Remove mistralrs / llama.cpp (#1970) commit def6eaa Author: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Date: Wed Jul 16 15:50:23 2025 -0400 feat: attributions for debian deps of sglang, trtllm, vllm runtime containers (#1971) commit f31732a Author: Yan Ru Pei <yanrpei@gmail.com> Date: Wed Jul 16 11:22:15 2025 -0700 feat: integrate mocker with dynamo-run and python cli (#1927) commit aba6099 Author: Graham King <grahamk@nvidia.com> Date: Wed Jul 16 12:26:32 2025 -0400 perf(router): Remove lock from router hot path (#1963) commit b212103 Author: Hongkuan Zhou <tedzhouhk@gmail.com> Date: Wed Jul 16 08:55:33 2025 -0700 docs: add notes in docs to deprecate local connector (#1959) commit 7b325ee Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 18:52:00 2025 -0700 fix: vllm router examples (#1942) commit a50be1a Author: hhzhang16 <54051230+hhzhang16@users.noreply.github.com> Date: Tue Jul 15 17:58:01 2025 -0700 feat: update CODEOWNERS (#1926) commit e260fdf Author: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Date: Tue Jul 15 18:49:21 2025 -0400 feat: add bitnami helm chart attribution (#1943) Signed-off-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> commit 1c03404 Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 14:26:24 2025 -0700 fix: update inference gateway deployment instructions (#1940) commit 5ca570f Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:54:03 2025 -0400 chore: Rename dynamo.ingress to dynamo.frontend (#1944) commit 7b9182f Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:33:07 2025 -0400 chore: Move examples/cli to lib/bindings/examples/cli (#1952) commit 40d40dd Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:02:19 2025 -0400 chore(multi-modal): Rename frontend.py to web.py (#1951) commit a9e0891 Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Tue Jul 15 12:30:30 2025 -0600 feat: adding http clients and recorded response stream (#1919) commit 4128d58 Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 10:30:47 2025 -0700 feat: allow helm upgrade using deploy script (#1936) commit 4da078b Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 12:57:38 2025 -0400 fix: Remove OpenSSL dependency, use Rust TLS (#1945) commit fc004d4 Author: jthomson04 <jwillthomson19@gmail.com> Date: Tue Jul 15 08:45:42 2025 -0700 fix: Fix TRT-LLM container build when using a custom pip wheel (#1825) commit 3c6fc6f Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 22:35:20 2025 -0700 chore: fix typo (#1938) commit de7fe38 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Mon Jul 14 21:47:12 2025 -0700 feat: add vllm e2e integration tests (#1935) commit 860f3f7 Author: Keiven C <213854356+keivenchang@users.noreply.github.com> Date: Mon Jul 14 21:44:19 2025 -0700 chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM (#1934) Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com> commit fc402a3 Author: Biswa Panda <biswa.panda@gmail.com> Date: Mon Jul 14 21:21:20 2025 -0700 feat: configurable namespace for vllm v1 example (#1909) commit df40d2c Author: ZichengMa <zichengma1225@gmail.com> Date: Mon Jul 14 21:11:29 2025 -0700 docs: fix typo and add mount-workspace to vllm doc (#1931) Signed-off-by: ZichengMa <zichengma1225@gmail.com> Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com> commit 901715b Author: Tanmay Verma <tanmayv@nvidia.com> Date: Mon Jul 14 20:14:51 2025 -0700 refactor: Refactor the TRTLLM examples remove dynamo SDK (#1884) commit 5bf23d5 Author: hhzhang16 <54051230+hhzhang16@users.noreply.github.com> Date: Mon Jul 14 18:29:19 2025 -0700 feat: update DynamoGraphDeployments for vllm_v1 (#1890) Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu> commit 9e76590 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 17:29:56 2025 -0700 docs: organize sglang readme (#1910) commit ef59ac8 Author: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com> Date: Mon Jul 14 16:16:44 2025 -0700 docs: TRTLLM Example of Llama4+Eagle3 (Speculative Decoding) (#1828) Signed-off-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com> Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> commit 053041e Author: Jorge António <matroid@outlook.com> Date: Tue Jul 15 00:06:38 2025 +0100 fix: resolve incorrect finish reason propagation (#1857) commit 3733f58 Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 19:04:22 2025 -0400 feat(backends): Python llama.cpp engine (#1925) commit 6a1350c Author: Tushar Sharma <tusharma@nvidia.com> Date: Mon Jul 14 14:56:36 2025 -0700 build: minor improvements to sglang dockerfile (#1917) commit e2a619b Author: Neelay Shah <neelays@nvidia.com> Date: Mon Jul 14 14:52:53 2025 -0700 fix: remove environment variable passing (#1911) Signed-off-by: Neelay Shah <neelays@nvidia.com> Co-authored-by: Neelay Shah <neelays@a4u8g-0057.ipp2u2.colossus.nvidia.com> commit 3d17a49 Author: Schwinn Saereesitthipitak <17022745+galletas1712@users.noreply.github.com> Date: Mon Jul 14 14:41:56 2025 -0700 refactor: remove dynamo build (#1778) Signed-off-by: Schwinn Saereesitthipitak <17022745+galletas1712@users.noreply.github.com> commit 3e0cb07 Author: Anant Sharma <anants@nvidia.com> Date: Mon Jul 14 15:43:48 2025 -0400 fix: copy attributions and license to trtllm runtime container (#1916) commit fc36bf5 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 12:31:49 2025 -0700 feat: receive kvmetrics from sglang scheduler (#1789) Co-authored-by: zixuanzhang226 <zixuanzhang@bytedance.com> commit df91fce Author: Yan Ru Pei <yanrpei@gmail.com> Date: Mon Jul 14 12:24:04 2025 -0700 feat: prefill aware routing (#1895) commit ad8ad66 Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 15:20:35 2025 -0400 feat: Shrink the ai-dynamo wheel by 35 MiB (#1918) Remove http and llmctl binaries. They have been unused for a while. commit 480b41d Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 15:06:45 2025 -0400 feat: Python frontend / ingress node (#1912)

commit cb6de94 Author: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Date: Sun Jul 20 22:34:50 2025 +0200 chore: Install vLLM and WideEP kernels in vLLM runtime container (#2010) Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com> Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com> Co-authored-by: alec-flowers <aflowers@nvidia.com> commit fe63c17 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Fri Jul 18 17:45:08 2025 -0700 fix: Revert "feat: add vLLM v1 multi-modal example. Add llama4 Maverick ex… (#2017) commit bf1998f Author: jthomson04 <jwillthomson19@gmail.com> Date: Fri Jul 18 17:23:50 2025 -0700 fix: Don't detokenize twice in TRT-LLM examples (#1955) commit 343a481 Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Fri Jul 18 16:22:43 2025 -0600 feat: http disconnects (#2014) commit e330d96 Author: Yan Ru Pei <yanrpei@gmail.com> Date: Fri Jul 18 13:40:54 2025 -0700 feat: enable / disable chunked prefill for mockers (#2015) Signed-off-by: Yan Ru Pei <yanrpei@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> commit 353146e Author: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Date: Fri Jul 18 13:33:36 2025 -0700 feat: add vLLM v1 multi-modal example. Add llama4 Maverick example (#1990) Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Co-authored-by: krishung5 <krish@nvidia.com> commit 1f07dab Author: Jacky <18255193+kthui@users.noreply.github.com> Date: Fri Jul 18 13:04:20 2025 -0700 feat: Add migration to LLM requests (#1930) commit 5f17918 Author: Tanmay Verma <tanmayv@nvidia.com> Date: Fri Jul 18 12:59:34 2025 -0700 refactor: Migrate to new UX2 for python launch (#2003) commit fc12436 Author: Graham King <grahamk@nvidia.com> Date: Fri Jul 18 14:52:57 2025 -0400 feat(frontend): router-mode settings (#2001) commit dc75cf1 Author: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Date: Fri Jul 18 18:47:28 2025 +0200 chore: Move NIXL repo clone to Dockerfiles (#2009) commit f6f392c Author: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Date: Thu Jul 17 18:44:17 2025 -0700 Remove link to the fix for disagg + eagle3 for TRT-LLM example (#2006) Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> commit cc90ca6 Author: atchernych <atchernych@nvidia.com> Date: Thu Jul 17 18:34:40 2025 -0700 feat: Create a convenience script to uninstall Dynamo Deploy CRDs (#1933) commit 267b422 Author: Greg Clark <grclark@nvidia.com> Date: Thu Jul 17 20:44:21 2025 -0400 chore: loosed python requirement versions (#1998) Signed-off-by: Greg Clark <grclark@nvidia.com> commit b8474e5 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Thu Jul 17 16:35:05 2025 -0700 chore: update cmake and gap installation and sgl in wideep container (#1991) commit 157a3b0 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 15:38:12 2025 -0700 fix: incorrect helm upgrade command (#2000) commit 0dfca2c Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 15:33:33 2025 -0700 ci: Update trtllm gitlab triggers for new components directory and test script (#1992) commit f3fb09e Author: Kris Hung <krish@nvidia.com> Date: Thu Jul 17 14:59:59 2025 -0700 fix: Fix syntax for tokio-console (#1997) commit dacffb8 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 14:57:10 2025 -0700 fix: use non-dev golang image for operator (#1993) commit 2b29a0a Author: zaristei <zaristei@berkeley.edu> Date: Thu Jul 17 13:10:42 2025 -0700 fix: Working Arm Build Dockerfile for Vllm_v1 (#1844) commit 2430d89 Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 12:57:46 2025 -0700 test: Add trtllm kv router tests (#1988) commit 1eadc01 Author: Graham King <grahamk@nvidia.com> Date: Thu Jul 17 15:07:41 2025 -0400 feat(runtime): Support tokio-console (#1986) commit b62e633 Author: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Date: Thu Jul 17 11:16:28 2025 -0700 feat: support separate chat_template.jinja file (#1853) commit 8ae3719 Author: Hongkuan Zhou <tedzhouhk@gmail.com> Date: Thu Jul 17 11:12:35 2025 -0700 chore: add some details to dynamo deploy quickstart and fix deploy.sh (#1978) Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com> Co-authored-by: julienmancuso <161955438+julienmancuso@users.noreply.github.com> commit 08891ff Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 10:57:42 2025 -0700 fix: Update trtllm tests to use new scripts instead of dynamo serve (#1979) commit 49b7a0d Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Thu Jul 17 08:35:04 2025 -0600 feat: record + analyze logprobs (#1957) commit 6d2be14 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 00:17:58 2025 -0700 refactor: replace vllm with vllm_v1 container (#1953) Co-authored-by: alec-flowers <aflowers@nvidia.com> commit 4d2a31a Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Wed Jul 16 18:04:09 2025 -0700 chore: add port reservation to utils (#1980) commit 1e3e4a0 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Wed Jul 16 15:54:04 2025 -0700 fix: port race condition through deterministic ports (#1937) commit 4ad281f Author: Tanmay Verma <tanmayv@nvidia.com> Date: Wed Jul 16 14:33:51 2025 -0700 refactor: Move TRTLLM example to the component/backends (#1976) commit 57d24a1 Author: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed Jul 16 14:10:24 2025 -0700 build: Removing shell configuration violations. It's bad practice to hardcod… (#1973) commit 182d3b5 Author: Graham King <grahamk@nvidia.com> Date: Wed Jul 16 16:12:40 2025 -0400 chore(bindings): Remove mistralrs / llama.cpp (#1970) commit def6eaa Author: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Date: Wed Jul 16 15:50:23 2025 -0400 feat: attributions for debian deps of sglang, trtllm, vllm runtime containers (#1971) commit f31732a Author: Yan Ru Pei <yanrpei@gmail.com> Date: Wed Jul 16 11:22:15 2025 -0700 feat: integrate mocker with dynamo-run and python cli (#1927) commit aba6099 Author: Graham King <grahamk@nvidia.com> Date: Wed Jul 16 12:26:32 2025 -0400 perf(router): Remove lock from router hot path (#1963) commit b212103 Author: Hongkuan Zhou <tedzhouhk@gmail.com> Date: Wed Jul 16 08:55:33 2025 -0700 docs: add notes in docs to deprecate local connector (#1959) commit 7b325ee Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 18:52:00 2025 -0700 fix: vllm router examples (#1942) commit a50be1a Author: hhzhang16 <54051230+hhzhang16@users.noreply.github.com> Date: Tue Jul 15 17:58:01 2025 -0700 feat: update CODEOWNERS (#1926) commit e260fdf Author: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Date: Tue Jul 15 18:49:21 2025 -0400 feat: add bitnami helm chart attribution (#1943) Signed-off-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> commit 1c03404 Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 14:26:24 2025 -0700 fix: update inference gateway deployment instructions (#1940) commit 5ca570f Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:54:03 2025 -0400 chore: Rename dynamo.ingress to dynamo.frontend (#1944) commit 7b9182f Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:33:07 2025 -0400 chore: Move examples/cli to lib/bindings/examples/cli (#1952) commit 40d40dd Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:02:19 2025 -0400 chore(multi-modal): Rename frontend.py to web.py (#1951) commit a9e0891 Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Tue Jul 15 12:30:30 2025 -0600 feat: adding http clients and recorded response stream (#1919) commit 4128d58 Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 10:30:47 2025 -0700 feat: allow helm upgrade using deploy script (#1936) commit 4da078b Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 12:57:38 2025 -0400 fix: Remove OpenSSL dependency, use Rust TLS (#1945) commit fc004d4 Author: jthomson04 <jwillthomson19@gmail.com> Date: Tue Jul 15 08:45:42 2025 -0700 fix: Fix TRT-LLM container build when using a custom pip wheel (#1825) commit 3c6fc6f Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 22:35:20 2025 -0700 chore: fix typo (#1938) commit de7fe38 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Mon Jul 14 21:47:12 2025 -0700 feat: add vllm e2e integration tests (#1935) commit 860f3f7 Author: Keiven C <213854356+keivenchang@users.noreply.github.com> Date: Mon Jul 14 21:44:19 2025 -0700 chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM (#1934) Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com> commit fc402a3 Author: Biswa Panda <biswa.panda@gmail.com> Date: Mon Jul 14 21:21:20 2025 -0700 feat: configurable namespace for vllm v1 example (#1909) commit df40d2c Author: ZichengMa <zichengma1225@gmail.com> Date: Mon Jul 14 21:11:29 2025 -0700 docs: fix typo and add mount-workspace to vllm doc (#1931) Signed-off-by: ZichengMa <zichengma1225@gmail.com> Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com> commit 901715b Author: Tanmay Verma <tanmayv@nvidia.com> Date: Mon Jul 14 20:14:51 2025 -0700 refactor: Refactor the TRTLLM examples remove dynamo SDK (#1884) commit 5bf23d5 Author: hhzhang16 <54051230+hhzhang16@users.noreply.github.com> Date: Mon Jul 14 18:29:19 2025 -0700 feat: update DynamoGraphDeployments for vllm_v1 (#1890) Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu> commit 9e76590 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 17:29:56 2025 -0700 docs: organize sglang readme (#1910) commit ef59ac8 Author: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com> Date: Mon Jul 14 16:16:44 2025 -0700 docs: TRTLLM Example of Llama4+Eagle3 (Speculative Decoding) (#1828) Signed-off-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com> Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> commit 053041e Author: Jorge António <matroid@outlook.com> Date: Tue Jul 15 00:06:38 2025 +0100 fix: resolve incorrect finish reason propagation (#1857) commit 3733f58 Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 19:04:22 2025 -0400 feat(backends): Python llama.cpp engine (#1925) commit 6a1350c Author: Tushar Sharma <tusharma@nvidia.com> Date: Mon Jul 14 14:56:36 2025 -0700 build: minor improvements to sglang dockerfile (#1917) commit e2a619b Author: Neelay Shah <neelays@nvidia.com> Date: Mon Jul 14 14:52:53 2025 -0700 fix: remove environment variable passing (#1911) Signed-off-by: Neelay Shah <neelays@nvidia.com> Co-authored-by: Neelay Shah <neelays@a4u8g-0057.ipp2u2.colossus.nvidia.com> commit 3d17a49 Author: Schwinn Saereesitthipitak <17022745+galletas1712@users.noreply.github.com> Date: Mon Jul 14 14:41:56 2025 -0700 refactor: remove dynamo build (#1778) Signed-off-by: Schwinn Saereesitthipitak <17022745+galletas1712@users.noreply.github.com> commit 3e0cb07 Author: Anant Sharma <anants@nvidia.com> Date: Mon Jul 14 15:43:48 2025 -0400 fix: copy attributions and license to trtllm runtime container (#1916) commit fc36bf5 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 12:31:49 2025 -0700 feat: receive kvmetrics from sglang scheduler (#1789) Co-authored-by: zixuanzhang226 <zixuanzhang@bytedance.com> commit df91fce Author: Yan Ru Pei <yanrpei@gmail.com> Date: Mon Jul 14 12:24:04 2025 -0700 feat: prefill aware routing (#1895) commit ad8ad66 Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 15:20:35 2025 -0400 feat: Shrink the ai-dynamo wheel by 35 MiB (#1918) Remove http and llmctl binaries. They have been unused for a while. commit 480b41d Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 15:06:45 2025 -0400 feat: Python frontend / ingress node (#1912)

commit d4b5414 Author: atchernych <atchernych@nvidia.com> Date: Mon Jul 21 13:10:24 2025 -0700 fix: mypy error (#2029) commit 79337c7 Author: Ryan McCormick <rmccormick@nvidia.com> Date: Mon Jul 21 12:12:16 2025 -0700 build: support custom TRTLLM build for commits not on main branch (#2021) commit 95dd942 Author: atchernych <atchernych@nvidia.com> Date: Mon Jul 21 12:09:33 2025 -0700 docs: Post-Merge cleanup of the deploy documentation (#1922) commit cb6de94 Author: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Date: Sun Jul 20 22:34:50 2025 +0200 chore: Install vLLM and WideEP kernels in vLLM runtime container (#2010) Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com> Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com> Co-authored-by: alec-flowers <aflowers@nvidia.com> commit fe63c17 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Fri Jul 18 17:45:08 2025 -0700 fix: Revert "feat: add vLLM v1 multi-modal example. Add llama4 Maverick ex… (#2017) commit bf1998f Author: jthomson04 <jwillthomson19@gmail.com> Date: Fri Jul 18 17:23:50 2025 -0700 fix: Don't detokenize twice in TRT-LLM examples (#1955) commit 343a481 Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Fri Jul 18 16:22:43 2025 -0600 feat: http disconnects (#2014) commit e330d96 Author: Yan Ru Pei <yanrpei@gmail.com> Date: Fri Jul 18 13:40:54 2025 -0700 feat: enable / disable chunked prefill for mockers (#2015) Signed-off-by: Yan Ru Pei <yanrpei@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> commit 353146e Author: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Date: Fri Jul 18 13:33:36 2025 -0700 feat: add vLLM v1 multi-modal example. Add llama4 Maverick example (#1990) Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Co-authored-by: krishung5 <krish@nvidia.com> commit 1f07dab Author: Jacky <18255193+kthui@users.noreply.github.com> Date: Fri Jul 18 13:04:20 2025 -0700 feat: Add migration to LLM requests (#1930) commit 5f17918 Author: Tanmay Verma <tanmayv@nvidia.com> Date: Fri Jul 18 12:59:34 2025 -0700 refactor: Migrate to new UX2 for python launch (#2003) commit fc12436 Author: Graham King <grahamk@nvidia.com> Date: Fri Jul 18 14:52:57 2025 -0400 feat(frontend): router-mode settings (#2001) commit dc75cf1 Author: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Date: Fri Jul 18 18:47:28 2025 +0200 chore: Move NIXL repo clone to Dockerfiles (#2009) commit f6f392c Author: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Date: Thu Jul 17 18:44:17 2025 -0700 Remove link to the fix for disagg + eagle3 for TRT-LLM example (#2006) Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> commit cc90ca6 Author: atchernych <atchernych@nvidia.com> Date: Thu Jul 17 18:34:40 2025 -0700 feat: Create a convenience script to uninstall Dynamo Deploy CRDs (#1933) commit 267b422 Author: Greg Clark <grclark@nvidia.com> Date: Thu Jul 17 20:44:21 2025 -0400 chore: loosed python requirement versions (#1998) Signed-off-by: Greg Clark <grclark@nvidia.com> commit b8474e5 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Thu Jul 17 16:35:05 2025 -0700 chore: update cmake and gap installation and sgl in wideep container (#1991) commit 157a3b0 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 15:38:12 2025 -0700 fix: incorrect helm upgrade command (#2000) commit 0dfca2c Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 15:33:33 2025 -0700 ci: Update trtllm gitlab triggers for new components directory and test script (#1992) commit f3fb09e Author: Kris Hung <krish@nvidia.com> Date: Thu Jul 17 14:59:59 2025 -0700 fix: Fix syntax for tokio-console (#1997) commit dacffb8 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 14:57:10 2025 -0700 fix: use non-dev golang image for operator (#1993) commit 2b29a0a Author: zaristei <zaristei@berkeley.edu> Date: Thu Jul 17 13:10:42 2025 -0700 fix: Working Arm Build Dockerfile for Vllm_v1 (#1844) commit 2430d89 Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 12:57:46 2025 -0700 test: Add trtllm kv router tests (#1988) commit 1eadc01 Author: Graham King <grahamk@nvidia.com> Date: Thu Jul 17 15:07:41 2025 -0400 feat(runtime): Support tokio-console (#1986) commit b62e633 Author: GuanLuo <41310872+GuanLuo@users.noreply.github.com> Date: Thu Jul 17 11:16:28 2025 -0700 feat: support separate chat_template.jinja file (#1853) commit 8ae3719 Author: Hongkuan Zhou <tedzhouhk@gmail.com> Date: Thu Jul 17 11:12:35 2025 -0700 chore: add some details to dynamo deploy quickstart and fix deploy.sh (#1978) Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com> Co-authored-by: julienmancuso <161955438+julienmancuso@users.noreply.github.com> commit 08891ff Author: Ryan McCormick <rmccormick@nvidia.com> Date: Thu Jul 17 10:57:42 2025 -0700 fix: Update trtllm tests to use new scripts instead of dynamo serve (#1979) commit 49b7a0d Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Thu Jul 17 08:35:04 2025 -0600 feat: record + analyze logprobs (#1957) commit 6d2be14 Author: Biswa Panda <biswa.panda@gmail.com> Date: Thu Jul 17 00:17:58 2025 -0700 refactor: replace vllm with vllm_v1 container (#1953) Co-authored-by: alec-flowers <aflowers@nvidia.com> commit 4d2a31a Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Wed Jul 16 18:04:09 2025 -0700 chore: add port reservation to utils (#1980) commit 1e3e4a0 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Wed Jul 16 15:54:04 2025 -0700 fix: port race condition through deterministic ports (#1937) commit 4ad281f Author: Tanmay Verma <tanmayv@nvidia.com> Date: Wed Jul 16 14:33:51 2025 -0700 refactor: Move TRTLLM example to the component/backends (#1976) commit 57d24a1 Author: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed Jul 16 14:10:24 2025 -0700 build: Removing shell configuration violations. It's bad practice to hardcod… (#1973) commit 182d3b5 Author: Graham King <grahamk@nvidia.com> Date: Wed Jul 16 16:12:40 2025 -0400 chore(bindings): Remove mistralrs / llama.cpp (#1970) commit def6eaa Author: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Date: Wed Jul 16 15:50:23 2025 -0400 feat: attributions for debian deps of sglang, trtllm, vllm runtime containers (#1971) commit f31732a Author: Yan Ru Pei <yanrpei@gmail.com> Date: Wed Jul 16 11:22:15 2025 -0700 feat: integrate mocker with dynamo-run and python cli (#1927) commit aba6099 Author: Graham King <grahamk@nvidia.com> Date: Wed Jul 16 12:26:32 2025 -0400 perf(router): Remove lock from router hot path (#1963) commit b212103 Author: Hongkuan Zhou <tedzhouhk@gmail.com> Date: Wed Jul 16 08:55:33 2025 -0700 docs: add notes in docs to deprecate local connector (#1959) commit 7b325ee Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 18:52:00 2025 -0700 fix: vllm router examples (#1942) commit a50be1a Author: hhzhang16 <54051230+hhzhang16@users.noreply.github.com> Date: Tue Jul 15 17:58:01 2025 -0700 feat: update CODEOWNERS (#1926) commit e260fdf Author: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Date: Tue Jul 15 18:49:21 2025 -0400 feat: add bitnami helm chart attribution (#1943) Signed-off-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> commit 1c03404 Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 14:26:24 2025 -0700 fix: update inference gateway deployment instructions (#1940) commit 5ca570f Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:54:03 2025 -0400 chore: Rename dynamo.ingress to dynamo.frontend (#1944) commit 7b9182f Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:33:07 2025 -0400 chore: Move examples/cli to lib/bindings/examples/cli (#1952) commit 40d40dd Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 16:02:19 2025 -0400 chore(multi-modal): Rename frontend.py to web.py (#1951) commit a9e0891 Author: Ryan Olson <ryanolson@users.noreply.github.com> Date: Tue Jul 15 12:30:30 2025 -0600 feat: adding http clients and recorded response stream (#1919) commit 4128d58 Author: Biswa Panda <biswa.panda@gmail.com> Date: Tue Jul 15 10:30:47 2025 -0700 feat: allow helm upgrade using deploy script (#1936) commit 4da078b Author: Graham King <grahamk@nvidia.com> Date: Tue Jul 15 12:57:38 2025 -0400 fix: Remove OpenSSL dependency, use Rust TLS (#1945) commit fc004d4 Author: jthomson04 <jwillthomson19@gmail.com> Date: Tue Jul 15 08:45:42 2025 -0700 fix: Fix TRT-LLM container build when using a custom pip wheel (#1825) commit 3c6fc6f Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 22:35:20 2025 -0700 chore: fix typo (#1938) commit de7fe38 Author: Alec <35311602+alec-flowers@users.noreply.github.com> Date: Mon Jul 14 21:47:12 2025 -0700 feat: add vllm e2e integration tests (#1935) commit 860f3f7 Author: Keiven C <213854356+keivenchang@users.noreply.github.com> Date: Mon Jul 14 21:44:19 2025 -0700 chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM (#1934) Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com> commit fc402a3 Author: Biswa Panda <biswa.panda@gmail.com> Date: Mon Jul 14 21:21:20 2025 -0700 feat: configurable namespace for vllm v1 example (#1909) commit df40d2c Author: ZichengMa <zichengma1225@gmail.com> Date: Mon Jul 14 21:11:29 2025 -0700 docs: fix typo and add mount-workspace to vllm doc (#1931) Signed-off-by: ZichengMa <zichengma1225@gmail.com> Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com> commit 901715b Author: Tanmay Verma <tanmayv@nvidia.com> Date: Mon Jul 14 20:14:51 2025 -0700 refactor: Refactor the TRTLLM examples remove dynamo SDK (#1884) commit 5bf23d5 Author: hhzhang16 <54051230+hhzhang16@users.noreply.github.com> Date: Mon Jul 14 18:29:19 2025 -0700 feat: update DynamoGraphDeployments for vllm_v1 (#1890) Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu> commit 9e76590 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 17:29:56 2025 -0700 docs: organize sglang readme (#1910) commit ef59ac8 Author: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com> Date: Mon Jul 14 16:16:44 2025 -0700 docs: TRTLLM Example of Llama4+Eagle3 (Speculative Decoding) (#1828) Signed-off-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com> Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> commit 053041e Author: Jorge António <matroid@outlook.com> Date: Tue Jul 15 00:06:38 2025 +0100 fix: resolve incorrect finish reason propagation (#1857) commit 3733f58 Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 19:04:22 2025 -0400 feat(backends): Python llama.cpp engine (#1925) commit 6a1350c Author: Tushar Sharma <tusharma@nvidia.com> Date: Mon Jul 14 14:56:36 2025 -0700 build: minor improvements to sglang dockerfile (#1917) commit e2a619b Author: Neelay Shah <neelays@nvidia.com> Date: Mon Jul 14 14:52:53 2025 -0700 fix: remove environment variable passing (#1911) Signed-off-by: Neelay Shah <neelays@nvidia.com> Co-authored-by: Neelay Shah <neelays@a4u8g-0057.ipp2u2.colossus.nvidia.com> commit 3d17a49 Author: Schwinn Saereesitthipitak <17022745+galletas1712@users.noreply.github.com> Date: Mon Jul 14 14:41:56 2025 -0700 refactor: remove dynamo build (#1778) Signed-off-by: Schwinn Saereesitthipitak <17022745+galletas1712@users.noreply.github.com> commit 3e0cb07 Author: Anant Sharma <anants@nvidia.com> Date: Mon Jul 14 15:43:48 2025 -0400 fix: copy attributions and license to trtllm runtime container (#1916) commit fc36bf5 Author: ishandhanani <82981111+ishandhanani@users.noreply.github.com> Date: Mon Jul 14 12:31:49 2025 -0700 feat: receive kvmetrics from sglang scheduler (#1789) Co-authored-by: zixuanzhang226 <zixuanzhang@bytedance.com> commit df91fce Author: Yan Ru Pei <yanrpei@gmail.com> Date: Mon Jul 14 12:24:04 2025 -0700 feat: prefill aware routing (#1895) commit ad8ad66 Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 15:20:35 2025 -0400 feat: Shrink the ai-dynamo wheel by 35 MiB (#1918) Remove http and llmctl binaries. They have been unused for a while. commit 480b41d Author: Graham King <grahamk@nvidia.com> Date: Mon Jul 14 15:06:45 2025 -0400 feat: Python frontend / ingress node (#1912)

chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM

f3ff3fe

keivenchang assigned nnshah1 and keivenchang Jul 15, 2025

keivenchang added the metrics label Jul 15, 2025

pull-request-size bot added the size/L label Jul 15, 2025

github-actions bot added the chore label Jul 15, 2025

coderabbitai bot reviewed Jul 15, 2025

View reviewed changes

lib/runtime/Cargo.toml Show resolved Hide resolved

lib/runtime/src/config.rs Outdated Show resolved Hide resolved

keivenchang added 2 commits July 14, 2025 18:09

Fix dependencies for security, and update comment to be in-sync with …

ce7b562

…code behavior

update nitpick noun "system server"

fc22d8d

nnshah1 approved these changes Jul 15, 2025

View reviewed changes

ZichengMa approved these changes Jul 15, 2025

View reviewed changes

cargo fmt

4dba63c

keivenchang enabled auto-merge (squash) July 15, 2025 04:05

keivenchang merged commit 860f3f7 into main Jul 15, 2025
8 of 9 checks passed

keivenchang deleted the keivenchang/Observability-enableHealthLiveEndpoints__DIS-239 branch July 15, 2025 04:44

grahamking reviewed Jul 15, 2025

View reviewed changes

grahamking mentioned this pull request Jul 15, 2025

fix: Remove OpenSSL dependency, use Rust TLS #1945

Merged

grahamking reviewed Jul 15, 2025

View reviewed changes

ZichengMa pushed a commit that referenced this pull request Jul 17, 2025

chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM (#…

9f6c0ed

…1934) Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

This was referenced Aug 5, 2025

feat(metrics): add NATS client metrics to prometheus_metrics_fmt #2292

Merged

refactor: Rename HTTP server to metrics server in worker process #2318

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM #1934

chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM #1934

Uh oh!

keivenchang commented Jul 15, 2025 •

edited

Loading

Uh oh!

copy-pr-bot bot commented Jul 15, 2025

Uh oh!

coderabbitai bot commented Jul 15, 2025

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

Uh oh!

coderabbitai bot left a comment •

edited by keivenchang

Loading

Uh oh!

Uh oh!

Uh oh!

nnshah1 left a comment

Uh oh!

ZichengMa left a comment

Uh oh!

keivenchang commented Jul 15, 2025

Uh oh!

Uh oh!

grahamking Jul 15, 2025

Uh oh!

grahamking Jul 15, 2025

Uh oh!

grahamking Jul 15, 2025

Uh oh!

Uh oh!

chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM #1934

chore: metrics endpoint variables renamed from HTTP_SERVER->SYSTEM #1934

Uh oh!

Conversation

keivenchang commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview:

Details:

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

Summary by CodeRabbit

Uh oh!

copy-pr-bot bot commented Jul 15, 2025

Uh oh!

coderabbitai bot commented Jul 15, 2025

Walkthrough

Changes

Sequence Diagram(s)

Poem

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

Uh oh!

coderabbitai bot left a comment • edited by keivenchang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nnshah1 left a comment

Choose a reason for hiding this comment

Uh oh!

ZichengMa left a comment

Choose a reason for hiding this comment

Uh oh!

keivenchang commented Jul 15, 2025

Uh oh!

Uh oh!

grahamking Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

grahamking Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

grahamking Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keivenchang commented Jul 15, 2025 •

edited

Loading

coderabbitai bot left a comment •

edited by keivenchang

Loading