feat: add vLLM v1 multi-modal example. Add llama4 Maverick example #1990

GuanLuo · 2025-07-17T19:28:58Z

Overview:

Details:

Where should the reviewer start?

This branch is built on top of branch krish/multimodal/v1. The additional files are llama 4 related files and README.md. Code changes on existing files are:

encode_worker.py: image loading code is extracted to image_loader.py which will be reused in VllmPDWorker in the case where prefill worker is also responsible for encoding. VllmEncodeWorker will clear image_url of the request as a hint that the prefill worker should consume image embedding.
worker.py: VllmPDWorker will either load image from url or image embedding from NIXL based on whether image_url is set in the request.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

closes GitHub issue: #xxx

Summary by CodeRabbit

New Features
- Introduced a comprehensive suite of multimodal model serving examples, including support for aggregated and disaggregated workflows with text and image inputs.
- Added ready-to-use configuration files for popular models (LLaVA, Qwen, Phi3V, Llama 4).
- Provided example FastAPI-based web frontends and detailed deployment instructions for both local and Kubernetes environments.
- Enabled high-performance data transfer between distributed workers using RDMA and NATS queue integration.
Documentation
- Added extensive README files detailing deployment scenarios, workflows, and usage instructions for multimodal model serving.
- Included protocol specifications and utility documentation for advanced users.
Utilities
- Added image loading, argument parsing, logging, and model handling utilities to streamline multimodal deployment and inference.
- Provided a Python RDMA connector library for efficient GPU data transfer in distributed setups.

copy-pr-bot · 2025-07-17T19:29:01Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

coderabbitai · 2025-07-17T19:32:17Z

Walkthrough

This change introduces a comprehensive multimodal example under examples/multimodal_v1, including detailed documentation, configuration files for various model setups, Python modules for component services (workers, processor, frontend), utility modules for image loading, RDMA data transfer, protocol definitions, and deployment graphs. The example supports both aggregated and disaggregated multimodal serving using the Dynamo framework and vLLM, with support for text and image input, streaming responses, and GPU-accelerated distributed inference.

Changes

File(s)	Change Summary
`examples/multimodal_v1/README.md`, `backup_README.md`, `connect/README.md`	Added comprehensive documentation for multimodal deployment, RDMA library, and Kubernetes usage.
`examples/multimodal_v1/configs/*.yaml`	Added configuration files for different multimodal models and serving setups (LLaVA, Qwen, Phi3V, Llama).
`examples/multimodal_v1/graphs/agg.py`, `graphs/disagg.py`	Defined component linking pipelines for aggregated and disaggregated serving graphs.
`examples/multimodal_v1/components/encode_worker.py`, `components/worker.py`	Introduced GPU-enabled encode, decode, and prefill worker service classes for multimodal inference.
`examples/multimodal_v1/components/processor.py`, `components/direct_processor.py`	Added processor service classes for pre/post-processing and orchestration of multimodal requests.
`examples/multimodal_v1/components/web.py`	Added FastAPI-based frontend service for handling multimodal chat completion requests.
`examples/multimodal_v1/connect/__init__.py`	Implemented RDMA Connector library for zero-copy data transfer between distributed workers.
`examples/multimodal_v1/utils/args.py`	Added utility for parsing vLLM engine arguments with multimodal-specific options.
`examples/multimodal_v1/utils/chat_processor.py`	Added utilities for chat/completion request preprocessing and streaming response postprocessing.
`examples/multimodal_v1/utils/image_loader.py`	Introduced async image loader with caching for HTTP and data URLs.
`examples/multimodal_v1/utils/logging.py`	Added async utility to monitor and log worker readiness.
`examples/multimodal_v1/utils/model.py`	Added utilities for loading vision models and constructing multimodal data.
`examples/multimodal_v1/utils/nats_queue.py`	Added async singleton wrapper for NATS-based task queues.
`examples/multimodal_v1/utils/nixl.py`	Added utilities for handling and storing NixlMetadata in distributed environments.
`examples/multimodal_v1/utils/protocol.py`	Defined Pydantic models and protocol for multimodal request/response serialization and validation.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Frontend
    participant Processor
    participant EncodeWorker
    participant PDWorker
    participant DecodeWorker

    Client->>Frontend: POST /v1/chat/completions (with text + image URL)
    Frontend->>Processor: generate(MultiModalRequest)
    Processor->>EncodeWorker: encode(vLLMMultimodalRequest)
    EncodeWorker->>PDWorker: generate(vLLMMultimodalRequest with embeddings)
    alt Disaggregated
        PDWorker->>DecodeWorker: generate(prefilled request)
        DecodeWorker-->>PDWorker: stream generation responses
        PDWorker-->>Processor: stream responses
    else Aggregated
        PDWorker-->>Processor: stream generation responses
    end
    Processor-->>Frontend: stream JSON responses
    Frontend-->>Client: stream JSON responses

Possibly related PRs

ai-dynamo/dynamo#1366: Adds prompt-template settings in multimodal config files and enhances deployment docs, directly relating to configuration and documentation aspects of the multimodal example.
ai-dynamo/dynamo#1882: Fixes a missing "av" package dependency in the container for multimodal examples, related due to deployment support for multimodal models.
ai-dynamo/dynamo#1388: Generalizes vision-language model embedding extraction and adds support for multiple models, overlapping with the new multimodal example in model handling utilities.

Poem

In the land of code, where models abound,
A rabbit hopped in and looked all around.
With images and text, it built a new show,
Streaming responses in a multimodal flow.
Workers and graphs, all linked with great care—
Now inference is faster, with RDMA flair!
🐰✨

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

GuanLuo · 2025-07-17T19:44:05Z

@krishung5 for you and others to review. This branch is built on top of branch krish/multimodal/v1. The additional files are llama 4 related files and README.md. Code changes on existing files are:

encode_worker.py: image loading code is extracted to image_loader.py which will be reused in VllmPDWorker in the case where prefill worker is also responsible for encoding. VllmEncodeWorker will clear image_url of the request as a hint that the prefill worker should consume image embedding.
worker.py: VllmPDWorker will either load image from url or image embedding from NIXL based on whether image_url is set in the request.

coderabbitai

Actionable comments posted: 36

🔭 Outside diff range comments (1)

examples/multimodal_v1/utils/protocol.py (1)

177-177: Add newline at end of file.

The file should end with a single newline character to comply with Black formatting requirements.

Add a newline after line 177.

🧹 Nitpick comments (27)

examples/multimodal_v1/utils/logging.py (1)
44-44: Use consistent logging approach.

The function uses both logger.info() and print() for output. Consider using the logger consistently for better log management and formatting.
-    print(f"Workers ready: {worker_ids}")
+    logger.info(f"Workers ready: {worker_ids}")
examples/multimodal_v1/utils/image_loader.py (1)

1-88: Apply Black formatting

The pipeline indicates Black formatting issues. Please run black on this file to fix formatting.
examples/multimodal_v1/utils/args.py (2)
16-17: Address TODO comment about naming ambiguity

The TODO comment indicates potential naming conflicts with the vllm package. Consider renaming this module to something more specific like multimodal_args.py or vllm_config_parser.py.

Would you like me to open an issue to track this renaming task?

23-23: Add type hints for function parameters

The function parameters lack type annotations for better code clarity and type safety.
-def parse_vllm_args(service_name, prefix) -> AsyncEngineArgs:
+def parse_vllm_args(service_name: str, prefix: str) -> AsyncEngineArgs:
examples/multimodal_v1/components/web.py (2)
27-27: Remove unused logger

The logger is defined but never used in this module.

Either remove the unused logger or add appropriate logging statements for debugging and monitoring.

62-62: Consider proper SSE formatting for streaming responses

The response uses text/event-stream media type but doesn't format the output as Server-Sent Events (SSE). For proper SSE streaming, each message should be prefixed with data: .
-                    yield s
+                    yield f"data: {json.dumps(s)}\n\n"
examples/multimodal_v1/configs/agg-phi3v.yaml (1)
33-34: Document the mm-processor-kwargs configuration

The mm-processor-kwargs with num_crops: 16 is model-specific configuration. Consider adding a comment explaining what this parameter controls for better maintainability.
   mm-processor-kwargs:
+    # Number of image crops for Phi3V's visual processing
     num_crops: 16
examples/multimodal_v1/utils/nats_queue.py (2)
81-81: Consider using more descriptive method names

The TODO raises a good point. Since this is a general-purpose queue, consider renaming to enqueue_message/dequeue_message or put/get to be more generic and align with common queue interfaces.

91-103: Add configurable timeout and consider async iteration

The hardcoded 0.1s timeout might not be suitable for all use cases. Also, the polling approach could be improved.

Consider this implementation:
-async def clear_queue(self) -> int:
+async def clear_queue(self, dequeue_timeout: float = 0.1) -> int:
     try:
         cleared_count = 0
         # Continue until we can't dequeue any more messages
         while True:
-            # use a small timeout
-            message = await self.dequeue_task(timeout=0.1)
+            message = await self.dequeue_task(timeout=dequeue_timeout)
             if message is None:
                 break
             cleared_count += 1
         return cleared_count
     except Exception as e:
         raise RuntimeError(f"Failed to clear queue: {e}")
For future enhancement, consider adding an async iterator interface for more Pythonic queue consumption.
examples/multimodal_v1/components/direct_processor.py (2)

171-173: Implement completion request handling

The completion request type is defined in the enum but not implemented. This could cause runtime errors if completion requests are sent to this processor.

Would you like me to help implement the completion request handling or create an issue to track this TODO?

99-103: Uncomment or remove etcd_kv_cache initialization

The etcd cache initialization is commented out. If it's not needed, remove it entirely. If it's planned for future use, add a TODO comment explaining when it will be enabled.
examples/multimodal_v1/utils/model.py (1)
51-58: Clean up or document commented code

The commented code shows alternative model loading methods but provides no context. Either remove it or add a comment explaining why these alternatives exist.

Consider adding a comment like:
# Alternative loading methods for specific model types:
# - Direct transformers loading (for debugging or when vLLM loading fails)
# model = LlavaForConditionalGeneration.from_pretrained(...)
# model = Qwen2_5_VLForConditionalGeneration.from_pretrained(...)
examples/multimodal_v1/connect/README.md (3)
26-44: Fix list indentation for markdown consistency

The unordered lists have 2-space indentation but markdown linters expect 0-space indentation.
-  - **Register local readable memory**:
-
-    Register local memory buffer(s) with the RDMA subsystem to enable a remote worker to read from.
-
-  - **Register local writable memory**:
-
-    Register local memory buffer(s) with the RDMA subsystem to enable a remote worker to write to.
-
-  - **Read from registered, remote memory**:
-
-    Read remote memory buffer(s), registered by a remote worker to be readable, into local memory buffer(s).
-
-  - **Write to registered, remote memory**:
-
-    Write local memory buffer(s) to remote memory buffer(s) registered by a remote worker to writable.
+- **Register local readable memory**:
+
+  Register local memory buffer(s) with the RDMA subsystem to enable a remote worker to read from.
+
+- **Register local writable memory**:
+
+  Register local memory buffer(s) with the RDMA subsystem to enable a remote worker to write to.
+
+- **Read from registered, remote memory**:
+
+  Read remote memory buffer(s), registered by a remote worker to be readable, into local memory buffer(s).
+
+- **Write to registered, remote memory**:
+
+  Write local memory buffer(s) to remote memory buffer(s) registered by a remote worker to writable.
52-52: Fix typo in "requested"
-Local then sends metadata about the requuested RDMA operation to Remote.
+Local then sends metadata about the requested RDMA operation to Remote.
163-163: Consider replacing "prior to" with "before"

For more concise writing, consider replacing "prior to" with "before" throughout the document.

Also applies to: 176-176, 189-189, 202-202, 245-245, 269-269, 293-293, 317-317
examples/multimodal_v1/components/encode_worker.py (2)
68-68: Clean up commented code blocks

Large sections of commented code make the file harder to read. Consider:

Remove if no longer needed

Add explanatory comments if kept for reference

Move to documentation if showing alternative approaches

The commented code appears to show:

Alternative vision model loading (line 68)

Different embedding extraction approach (lines 98-133)

Either remove or add a comment explaining why these alternatives exist.

Also applies to: 98-133

69-71: Consider using the load_vision_model utility

You have a commented line using load_vision_model from the utils module, but instead directly load the model. Consider using the utility function for consistency across the codebase.
-# self.vision_model = load_vision_model(self.MODEL_ID)
-self.vision_model = LlavaForConditionalGeneration.from_pretrained(
-    self.MODEL_ID, device_map="auto", torch_dtype=torch.float16
-).eval()
+from utils.model import load_vision_model
+self.vision_model = load_vision_model(self.MODEL_ID)
examples/multimodal_v1/connect/__init__.py (3)
80-86: Simplify redundant conditional assignment.

The notification_key is guaranteed to be non-None after line 80, making the conditional assignment on line 86 redundant.
-        notification_key = str(uuid.uuid4()) if notification_key is None else notification_key
-        if not isinstance(notification_key, str):
-            raise TypeError("Argument `notification_key` must be `str` or `None`.")
-        if len(notification_key) == 0:
-            raise ValueError("Argument `notification_key` must not be an empty string.")
-
-        self._notification_key: str = "" if notification_key is None else notification_key
+        notification_key = str(uuid.uuid4()) if notification_key is None else notification_key
+        if not isinstance(notification_key, str):
+            raise TypeError("Argument `notification_key` must be `str` or `None`.")
+        if len(notification_key) == 0:
+            raise ValueError("Argument `notification_key` must not be an empty string.")
+
+        self._notification_key: str = notification_key
860-860: Remove empty line with indentation.

Line 860 contains an empty line with indentation that should be completely empty.

Remove the indentation from the empty line 860.

727-727: Add space around comparison operator for consistency.
-            self._data_ref = data[3] if len(data) >=4 else None
+            self._data_ref = data[3] if len(data) >= 4 else None
examples/multimodal_v1/utils/protocol.py (2)
61-68: Document potential risks of monkey-patching.

While the monkey-patching is necessary for Pydantic compatibility, it could break if vLLM changes its internal structure. Consider adding a more detailed comment about this risk and potential version compatibility issues.
 # Monkey-patch the SamplingParams type to add a dummy core schema so pydantic can validate it
 # Sampling params is a mspspec struct
 # SamplingParams is defined here: https://github.com/vllm-project/vllm/blob/a4c402a756fa3213caf9d2cde0e4ceb2d57727f2/vllm/sampling_params.py#L88
+# WARNING: This monkey-patch may break if vLLM changes the SamplingParams implementation.
+# Consider pinning the vLLM version or implementing a wrapper class instead.
 
 SamplingParams.__get_pydantic_core_schema__ = classmethod(
     lambda cls, source, handler: core_schema.any_schema()
 )
150-150: Track the TODO for removing image_features.

The comment indicates that image_features field will be removed once NIXL support is added. This should be tracked properly.

Would you like me to create an issue to track the removal of the image_features field once NIXL support is fully implemented?
examples/multimodal_v1/backup_README.md (1)
89-89: Improve sentence variety to enhance readability

The static analysis correctly identifies three consecutive sentences starting with "If serving...". Consider rephrasing for better flow.
-If serving the example Qwen model, replace `"llava-hf/llava-1.5-7b-hf"` in the `"model"` field with `"Qwen/Qwen2.5-VL-7B-Instruct"`. If serving the example Phi3V model, replace `"llava-hf/llava-1.5-7b-hf"` in the `"model"` field with `"microsoft/Phi-3.5-vision-instruct"`.
+For the Qwen model example, replace `"llava-hf/llava-1.5-7b-hf"` in the `"model"` field with `"Qwen/Qwen2.5-VL-7B-Instruct"`. When using the Phi3V model example, replace it with `"microsoft/Phi-3.5-vision-instruct"`.
examples/multimodal_v1/components/processor.py (2)
97-101: Remove commented code

The commented etcd_kv_cache initialization should be removed if not needed, or implemented if required.
-        # self.etcd_kv_cache = await EtcdKvCache.create(
-        #     runtime.etcd_client(),
-        #     "/dynamo/processor/",
-        #     {"router": self.engine_args.router},
-        # )
198-206: Improve image URL extraction logic

The current implementation only extracts the first image URL found. Consider handling multiple images or validating the message structure more thoroughly.
         image_url = None
-
         for message in raw_request.messages:
             for item in message.content:
                 if item.type == "image_url":
                     image_url = item.image_url.url
+                    break  # Exit inner loop once found
+            if image_url:
+                break  # Exit outer loop once found
+        
         if image_url is None:
             raise ValueError("Image URL is required")
examples/multimodal_v1/utils/nixl.py (2)
25-25: Consider making metadata directory configurable

The hardcoded /tmp/nixl directory could cause issues in containerized environments or when running multiple instances.
-METADATA_DIR = "/tmp/nixl"
+METADATA_DIR = os.environ.get("NIXL_METADATA_DIR", "/tmp/nixl")
84-92: Optimize etcd lookup by using single-key get instead of prefix scan

Since you already know the exact key ({self._key_prefix}/{engine_id}), replace the prefix scan and loop with a direct kv_get call:
             key = "/".join([self._key_prefix, engine_id])
-            key_values = await self._client.kv_get_prefix(key)
-            deserialized_metadata = None
-
-            for item in key_values:
-                deserialized_metadata = msgspec.msgpack.decode(
-                    item["value"], type=NixlMetadata
-                )
-                break
+            # If the client supports direct key lookup, prefer this for performance
+            value = await self._client.kv_get(key)
+            deserialized_metadata = None
+
+            if value:
+                deserialized_metadata = msgspec.msgpack.decode(
+                    value, type=NixlMetadata
+                )
• Please verify that the etcd client returned by runtime.etcd_client() exposes a kv_get(key) method that returns the raw bytes (not a key/value list).
• If kv_get isn’t available or returns a different structure, continue using kv_get_prefix as-is.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1eadc01 and d4c6518.

📒 Files selected for processing (24)

examples/multimodal_v1/README.md (1 hunks)
examples/multimodal_v1/backup_README.md (1 hunks)
examples/multimodal_v1/components/direct_processor.py (1 hunks)
examples/multimodal_v1/components/encode_worker.py (1 hunks)
examples/multimodal_v1/components/processor.py (1 hunks)
examples/multimodal_v1/components/web.py (1 hunks)
examples/multimodal_v1/components/worker.py (1 hunks)
examples/multimodal_v1/configs/agg-llava.yaml (1 hunks)
examples/multimodal_v1/configs/agg-phi3v.yaml (1 hunks)
examples/multimodal_v1/configs/agg-qwen.yaml (1 hunks)
examples/multimodal_v1/configs/disagg.yaml (1 hunks)
examples/multimodal_v1/configs/llama.yaml (1 hunks)
examples/multimodal_v1/connect/README.md (1 hunks)
examples/multimodal_v1/connect/__init__.py (1 hunks)
examples/multimodal_v1/graphs/agg.py (1 hunks)
examples/multimodal_v1/graphs/disagg.py (1 hunks)
examples/multimodal_v1/utils/args.py (1 hunks)
examples/multimodal_v1/utils/chat_processor.py (1 hunks)
examples/multimodal_v1/utils/image_loader.py (1 hunks)
examples/multimodal_v1/utils/logging.py (1 hunks)
examples/multimodal_v1/utils/model.py (1 hunks)
examples/multimodal_v1/utils/nats_queue.py (1 hunks)
examples/multimodal_v1/utils/nixl.py (1 hunks)
examples/multimodal_v1/utils/protocol.py (1 hunks)

🧰 Additional context used

🧠 Learnings (10)

examples/multimodal_v1/configs/agg-qwen.yaml (2)

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: The current get_vision_embeddings_size() function in examples/multimodal/utils/model.py uses a hardcoded fallback of 4096 for hidden_size, which assumes all VLMs follow the LLaVA architecture pattern. This is problematic because different VLMs like Qwen2-VL, MiniCPM-V, and others have different hidden dimensions, making the fallback unreliable and potentially causing tensor shape mismatches.

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: Different Vision-Language Models (VLMs) have different hidden dimensions, so using a single hardcoded fallback value of 4096 in get_vision_embeddings_size() is problematic and can cause tensor shape mismatches. The current code uses getattr(config, "hidden_size", 4096) where 4096 is a fallback default based on LLaVA models, but other VLMs like Qwen2-VL may have different dimensions.

examples/multimodal_v1/configs/agg-phi3v.yaml (2)

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: The current get_vision_embeddings_size() function in examples/multimodal/utils/model.py uses a hardcoded fallback of 4096 for hidden_size, which assumes all VLMs follow the LLaVA architecture pattern. This is problematic because different VLMs like Qwen2-VL, MiniCPM-V, and others have different hidden dimensions, making the fallback unreliable and potentially causing tensor shape mismatches.

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: Different Vision-Language Models (VLMs) have different hidden dimensions, so using a single hardcoded fallback value of 4096 in get_vision_embeddings_size() is problematic and can cause tensor shape mismatches. The current code uses getattr(config, "hidden_size", 4096) where 4096 is a fallback default based on LLaVA models, but other VLMs like Qwen2-VL may have different dimensions.

examples/multimodal_v1/README.md (1)

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: The current get_vision_embeddings_size() function in examples/multimodal/utils/model.py uses a hardcoded fallback of 4096 for hidden_size, which assumes all VLMs follow the LLaVA architecture pattern. This is problematic because different VLMs like Qwen2-VL, MiniCPM-V, and others have different hidden dimensions, making the fallback unreliable and potentially causing tensor shape mismatches.

examples/multimodal_v1/configs/agg-llava.yaml (4)

Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/configs/agg_tp_1_dp_8.yaml:31-38
Timestamp: 2025-07-01T15:33:53.262Z
Learning: In fault tolerance test configurations, the `resources` section under `ServiceArgs` specifies resources per individual worker, not total resources for all workers. So `workers: 8` with `gpu: '1'` means 8 workers × 1 GPU each = 8 GPUs total.

Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/job_script_template.j2:59-59
Timestamp: 2025-07-02T13:20:28.800Z
Learning: In the SLURM job script template at examples/sglang/slurm_jobs/job_script_template.j2, the `--total_nodes` parameter represents the total nodes per worker type (prefill or decode), not the total nodes in the entire cluster. Each worker type needs to know its own group size for distributed coordination.

Learnt from: biswapanda
PR: ai-dynamo/dynamo#1890
File: examples/vllm/deploy/agg.yaml:63-70
Timestamp: 2025-07-14T23:01:16.218Z
Learning: In vLLM worker deployments, grep-based log checks for "VllmWorker.*has been initialized" are appropriate for readiness probes to verify worker startup, but should not be used for liveness probes which need to detect ongoing worker health.

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: The current get_vision_embeddings_size() function in examples/multimodal/utils/model.py uses a hardcoded fallback of 4096 for hidden_size, which assumes all VLMs follow the LLaVA architecture pattern. This is problematic because different VLMs like Qwen2-VL, MiniCPM-V, and others have different hidden dimensions, making the fallback unreliable and potentially causing tensor shape mismatches.

examples/multimodal_v1/configs/disagg.yaml (5)

Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/configs/agg_tp_1_dp_8.yaml:31-38
Timestamp: 2025-07-01T15:33:53.262Z
Learning: In fault tolerance test configurations, the `resources` section under `ServiceArgs` specifies resources per individual worker, not total resources for all workers. So `workers: 8` with `gpu: '1'` means 8 workers × 1 GPU each = 8 GPUs total.

Learnt from: GuanLuo
PR: ai-dynamo/dynamo#1371
File: examples/llm/benchmarks/vllm_multinode_setup.sh:18-25
Timestamp: 2025-06-05T01:46:15.509Z
Learning: In multi-node setups with head/worker architecture, the head node typically doesn't need environment variables pointing to its own services (like NATS_SERVER, ETCD_ENDPOINTS) because local processes can access them via localhost. Only worker nodes need these environment variables to connect to the head node's external IP address.

Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/job_script_template.j2:59-59
Timestamp: 2025-07-02T13:20:28.800Z
Learning: In the SLURM job script template at examples/sglang/slurm_jobs/job_script_template.j2, the `--total_nodes` parameter represents the total nodes per worker type (prefill or decode), not the total nodes in the entire cluster. Each worker type needs to know its own group size for distributed coordination.

Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:113-116
Timestamp: 2025-07-03T09:44:41.470Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, logging the full configuration file content is acceptable because the config file is public, contains only placeholder replacements (no sensitive data), and provides debugging benefits for users who may want to create configurations based on the logged output.

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: Different Vision-Language Models (VLMs) have different hidden dimensions, so using a single hardcoded fallback value of 4096 in get_vision_embeddings_size() is problematic and can cause tensor shape mismatches. The current code uses getattr(config, "hidden_size", 4096) where 4096 is a fallback default based on LLaVA models, but other VLMs like Qwen2-VL may have different dimensions.

examples/multimodal_v1/configs/llama.yaml (2)

Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/configs/agg_tp_1_dp_8.yaml:31-38
Timestamp: 2025-07-01T15:33:53.262Z
Learning: In fault tolerance test configurations, the `resources` section under `ServiceArgs` specifies resources per individual worker, not total resources for all workers. So `workers: 8` with `gpu: '1'` means 8 workers × 1 GPU each = 8 GPUs total.

Learnt from: biswapanda
PR: ai-dynamo/dynamo#1890
File: examples/vllm/deploy/agg.yaml:63-70
Timestamp: 2025-07-14T23:01:16.218Z
Learning: In vLLM worker deployments, grep-based log checks for "VllmWorker.*has been initialized" are appropriate for readiness probes to verify worker startup, but should not be used for liveness probes which need to detect ongoing worker health.

examples/multimodal_v1/utils/nats_queue.py (1)

Learnt from: kthui
PR: ai-dynamo/dynamo#1424
File: lib/runtime/src/pipeline/network/egress/push_router.rs:204-209
Timestamp: 2025-06-13T22:07:24.843Z
Learning: The codebase uses async-nats version 0.40, not the older nats crate. Error handling should use async_nats::error::Error variants, not nats::Error variants.

examples/multimodal_v1/components/worker.py (3)

Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1392
File: launch/dynamo-run/src/subprocess/vllm_v1_inc.py:71-71
Timestamp: 2025-06-05T01:04:24.775Z
Learning: The `create_endpoint` method in `WorkerMetricsPublisher` has backward compatibility maintained through pyo3 signature annotation `#[pyo3(signature = (component, dp_rank = None))]`, making the `dp_rank` parameter optional with a default value of `None`.

Learnt from: biswapanda
PR: ai-dynamo/dynamo#1890
File: examples/vllm/deploy/agg.yaml:63-70
Timestamp: 2025-07-14T23:01:16.218Z
Learning: In vLLM worker deployments, grep-based log checks for "VllmWorker.*has been initialized" are appropriate for readiness probes to verify worker startup, but should not be used for liveness probes which need to detect ongoing worker health.

Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.

examples/multimodal_v1/utils/model.py (2)

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: The current get_vision_embeddings_size() function in examples/multimodal/utils/model.py uses a hardcoded fallback of 4096 for hidden_size, which assumes all VLMs follow the LLaVA architecture pattern. This is problematic because different VLMs like Qwen2-VL, MiniCPM-V, and others have different hidden dimensions, making the fallback unreliable and potentially causing tensor shape mismatches.

Learnt from: krishung5
PR: ai-dynamo/dynamo#1388
File: examples/multimodal/utils/model.py:47-53
Timestamp: 2025-06-09T17:52:06.761Z
Learning: Different Vision-Language Models (VLMs) have different hidden dimensions, so using a single hardcoded fallback value of 4096 in get_vision_embeddings_size() is problematic and can cause tensor shape mismatches. The current code uses getattr(config, "hidden_size", 4096) where 4096 is a fallback default based on LLaVA models, but other VLMs like Qwen2-VL may have different dimensions.

examples/multimodal_v1/utils/chat_processor.py (1)

Learnt from: ishandhanani
PR: ai-dynamo/dynamo#1626
File: lib/llm/src/preprocessor.rs:238-239
Timestamp: 2025-06-24T20:59:35.725Z
Learning: In lib/llm/src/preprocessor.rs, the `sampling_options` call in the `preprocess_request` method is placed in the common section after the match statement on `request.prompt_input_type()`, meaning it applies to both `PromptInput::Tokens` and `PromptInput::Text` request types.

🧬 Code Graph Analysis (1)

examples/multimodal_v1/utils/protocol.py (2)

examples/multimodal_v1/components/encode_worker.py (1)

encode (76-170)

examples/multimodal_v1/connect/__init__.py (1)

SerializedRequest (1316-1344)

🪛 GitHub Actions: Copyright Checks

examples/multimodal_v1/utils/image_loader.py

[error] 1-1: Invalid or missing copyright header detected. Please add the required SPDX copyright and license header.

🪛 GitHub Actions: Pre Merge Validation of (ai-dynamo/dynamo/refs/pull/1990/merge) by GuanLuo.

examples/multimodal_v1/utils/image_loader.py

[error] Black formatting check failed. Files were reformatted by the black hook.

examples/multimodal_v1/components/worker.py

[error] Black formatting check failed. Files were reformatted by the black hook.

examples/multimodal_v1/components/direct_processor.py

[error] 153-153: Ruff: Local variable prompt_idx is assigned to but never used. (F841)

[error] isort formatting check failed. Files were modified by the isort hook.

[error] Black formatting check failed. Files were reformatted by the black hook.

examples/multimodal_v1/utils/protocol.py

[error] Black formatting check failed. Files were reformatted by the black hook.

🪛 markdownlint-cli2 (0.17.2)

examples/multimodal_v1/README.md

264-264: Multiple headings with the same content

(MD024, no-duplicate-heading)

examples/multimodal_v1/connect/README.md

26-26: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

30-30: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

34-34: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

38-38: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

138-138: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

140-140: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

339-339: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

340-340: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

341-341: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

342-342: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

🪛 LanguageTool

examples/multimodal_v1/connect/README.md

[style] ~30-~30: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...le a remote worker to read from. - Register local writable memory: Register ...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

[style] ~145-~145: ‘prior to’ might be wordy. Consider a shorter alternative.
Context: ... been completed or cancel the operation prior to completion. The RDMA metadata must be ...