Skip to content

fix: update inference gateway deployment instructions #1940 #1941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 15, 2025
Merged

fix: update inference gateway deployment instructions #1940 #1941

merged 1 commit into from
Jul 15, 2025

Conversation

biswapanda
Copy link
Contributor

@biswapanda biswapanda commented Jul 15, 2025 •
edited by coderabbitai bot
Loading

Overview:

Cherry-pick for #1940

  • Builderless k8s dyanmo graph deployment based example for inference gateway set up
    This is targeted at vllm_v0 examples specifically for release/0.3.2

fixes: https://nvbugs/5357930

closes: DEP-230

Details:

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #xxx

Summary by CodeRabbit

  • New Features

    • Added new Kubernetes deployment manifests for multimodal examples, enabling easy deployment of multimodal graph applications with dedicated services and GPU support.
    • Introduced a function to dynamically retrieve the C API library path in the SDK.
    • Added a deployment manifest for the hello-world example application.

  • Improvements

    • Updated container images and resource specifications across multiple deployment YAMLs to use fixed version tags and generic GPU keys.
    • Enhanced Dockerfiles and build scripts with updated dependency versions and improved build arguments for flexibility.
    • Refined Helm charts and templates for more flexible and explicit service configuration, health checks, and resource management.
    • Updated documentation for clearer deployment instructions, prerequisites, and customization options.
    • Streamlined deployment examples to use pre-built images and simplified configuration management.

  • Bug Fixes

    • Improved handling of environment variables and port assignments in test and deployment scripts to support multimodal deployments and dynamic configuration.

  • Removals

    • Removed MinIO, ETCD, and NATS Helm chart dependencies and associated configuration files.
    • Deleted legacy deployment scripts and redundant configuration blocks.

  • Dependency Updates

    • Upgraded versions of key dependencies such as NATS, ETCD, NIXL, and related Python/Rust packages.
    • Added the av Python package to requirements for video processing support.

  • Style/Refactor

    • Refactored import statements and deployment templates for better maintainability and to avoid runtime errors.

  • Documentation

    • Expanded and restructured documentation for deployment, configuration, and example usage, including new sections for Kubernetes deployment of multimodal examples.

@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jul 15, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@biswapanda biswapanda changed the base branch from main to release/0.3.2 July 15, 2025 08:57
@github-actions github-actions bot added the fix label Jul 15, 2025
@coderabbitai coderabbitai
Copy link
Contributor

coderabbitai bot commented Jul 15, 2025 •
edited
Loading

Caution

Review failed

Failed to post review comments.

Walkthrough

This update introduces extensive changes across container, deployment, example, and SDK files. Key changes include Dockerfile enhancements (notably for sglang), dependency upgrades (NATS, ETCD, NIXL), new and updated Kubernetes deployment manifests for multimodal and LLM examples, Helm chart restructuring, and improvements to test logic and SDK utility exports. Several configuration and documentation files are also updated or removed.

Changes

Files/Paths Change Summary
container/Dockerfile.sglang, Dockerfile.* Enhanced build with new args for sglang/sgl-kernel versions, full UCX EFA setup, NATS/ETCD/NIXL upgrades, explicit wheel builds, runtime env improvements, and attribution copying. Other Dockerfiles update NATS/ETCD versions and minor fixes.
container/build.sh NIXL_COMMIT hash updated for new version checkout.
container/deps/requirements.txt Added av==15.0.0 to dependencies.
deploy/cloud/helm/platform/Chart.yaml, values.yaml Removed MinIO dependency and configuration from Helm chart and values.
deploy/cloud/operator/internal/consts/consts.go, controller/dynamocomponentdeployment_controller_test.go, dynamo/graph_test.go Changed DynamoServicePort from 3000 to 8000 and updated all test code to use the constant instead of literals.
deploy/helm/README.md Expanded and restructured documentation for Helm deployment, adding prerequisites, install/customization instructions, and a config table.
deploy/helm/chart/Chart.yaml, templates/deployment.yaml, templates/service.yaml, values.yaml Chart renamed to "dynamo-graph", version bumped, deployment/service templates refactored for .Values.spec.services, service-specific config, health checks, and resource handling; imagePullSecrets pruned.
deploy/helm/dependencies/etcd-values.yaml, nats-values.yaml, deploy.sh Deleted default dependency values for etcd/NATS and removed Helm deploy script.
deploy/inference-gateway/example/README.md, resources/dynamo-epp.yaml, resources/inference-pool.yaml Deployment instructions revised for pre-built images; resource YAMLs updated with new image tags, port, selector, and extensionRef changes.
deploy/sdk/src/dynamo/sdk/init.py, lib/utils.py Exported and implemented get_capi_library_path() utility for C API library path resolution.
docs/support_matrix.md, lib/llm/Cargo.toml Bumped NIXL/nixl-sys versions to 0.4.0 in support matrix and Rust dependency.
examples/hello_world/config.yaml, deploy/hello_world.yaml Backend worker count reduced to 1; new deployment YAML added for hello-world example with three services.
examples/llm/deploy/.yaml, vllm_v0/deploy/.yaml, vllm_v1/deploy/*.yaml All LLM example deployments updated to use fixed image tag 0.3.1, switch GPU resource key from nvidia.com/gpu to gpu, add config file flags, and remove inline env configs.
examples/multimodal/README.md, deploy/k8s/*.yaml Added detailed K8s deployment instructions; new multimodal deployment manifests for agg/disagg video, phi3v, qwen, llava examples.
examples/multimodal/utils/model.py Made Worker import lazy inside function to avoid CUDA errors on non-GPU.
examples/sglang/*.md, slurm_jobs/job_script_template.j2 Updated multinode instructions for SGLang, fixed command flags, and changed enroot mount option.
examples/vllm_v0/components/worker.py Refactored metrics publishing to use structured metric objects.
hatch_build.py Added force_include entry for libdynamo_llm_capi.so in build hook.
launch/dynamo-run/src/subprocess/vllm_inc.py Switched to dynamic C API library path using new SDK utility.
tests/serve/test_dynamo_serve.py Conditional logic for --Frontend.port argument and DYNAMO_PORT env var based on deployment type (multimodal or not).

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant K8s
    participant Container
    participant SDK

    User->>K8s: Apply DynamoGraphDeployment manifest (e.g. multimodal/llm example)
    K8s->>Container: Launch container with specified image, resources, args
    Container->>SDK: Import and use get_capi_library_path()
    SDK->>Container: Return correct path for C API library
    Container->>K8s: Expose health/readiness endpoints (as per Helm template)
    User->>K8s: Interact via service endpoints
Loading

Possibly related PRs

  • ai-dynamo/dynamo#1865: Also updates NATS and ETCD versions in Dockerfiles, overlapping with this PR's container dependency upgrades.
  • ai-dynamo/dynamo#1854: Modifies the same examples/hello_world/deploy/hello_world.yaml deployment manifest as this PR.
  • ai-dynamo/dynamo#1845: Refactors metrics publishing in examples/vllm_v0/components/worker.py, directly related to similar changes here.

Poem

In the warren where containers brew,
New versions hop and configs renew.
Helm charts pruned, the graphs deploy,
Multimodal bunnies leap with joy!
SDKs now find their library path—
With every hop, we dodge old wrath.
🐇✨ The future’s bright, on Kubernetes night!

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@nv-anants nv-anants merged commit c433cd2 into release/0.3.2 Jul 15, 2025
5 checks passed
@nv-anants nv-anants deleted the bis/cp-dep-230 branch July 15, 2025 17:47
@coderabbitai coderabbitai bot mentioned this pull request Aug 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hhzhang16 hhzhang16 hhzhang16 approved these changes

@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV

@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk

@PeaBrane PeaBrane Awaiting requested review from PeaBrane

@alec-flowers alec-flowers Awaiting requested review from alec-flowers

@ishandhanani ishandhanani Awaiting requested review from ishandhanani ishandhanani is a code owner

@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4

@grahamking grahamking Awaiting requested review from grahamking

@indrajit96 indrajit96 Awaiting requested review from indrajit96

@krishung5 krishung5 Awaiting requested review from krishung5

@whoisj whoisj Awaiting requested review from whoisj

@GuanLuo GuanLuo Awaiting requested review from GuanLuo

@julienmancuso julienmancuso Awaiting requested review from julienmancuso julienmancuso is a code owner

@kkranen kkranen Awaiting requested review from kkranen

@mohammedabdulwahhab mohammedabdulwahhab Awaiting requested review from mohammedabdulwahhab mohammedabdulwahhab is a code owner

@nnshah1 nnshah1 Awaiting requested review from nnshah1 nnshah1 is a code owner

@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia

@tanmayv25 tanmayv25 Awaiting requested review from tanmayv25

@hutm hutm Awaiting requested review from hutm hutm is a code owner

@richardhuo-nv richardhuo-nv Awaiting requested review from richardhuo-nv

@ryanolson ryanolson Awaiting requested review from ryanolson

@paulhendricks paulhendricks Awaiting requested review from paulhendricks

@tmonty12 tmonty12 Awaiting requested review from tmonty12

@jthomson04 jthomson04 Awaiting requested review from jthomson04

Assignees
No one assigned
Labels
fix size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@biswapanda @hhzhang16 @nv-anants