[Docs] Update EAGLE example #20375

NickLucche · 2025-07-02T13:40:05Z

Copy pasting the example after changes to spec decoding conf yields:

  File "/home/nicolo/llmd/vllm/vllm/engine/arg_utils.py", line 1039, in create_speculative_config
    speculative_config = SpeculativeConfig.from_dict(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nicolo/llmd/vllm/vllm/config.py", line 2543, in from_dict
    return cls(**dict_value)
           ^^^^^^^^^^^^^^^^^
  File "/home/nicolo/llmd/.venv/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py", line 123, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
pydantic_core._pydantic_core.ValidationError: 1 validation error for SpeculativeConfig
  Value error, num_speculative_tokens must be provided with speculative model unless the draft model config contains an n_predict parameter. [type=value_error, input_value=ArgsKwargs((), {'model': ...sable_log_stats': True}), input_type=ArgsKwargs]

Signed-off-by: NickLucche <nlucches@redhat.com>

gemini-code-assist

Summary of Changes

Hello @NickLucche, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on improving the accuracy and usability of the documentation by correcting a non-functional code example related to speculative decoding. The update ensures that users can directly copy and run the provided configuration without encountering validation errors, thereby enhancing the developer experience.

Highlights

Documentation Update: I've updated the EAGLE speculative decoding example in docs/features/spec_decode.md to include the num_speculative_tokens parameter within the speculative_config.
Bug Fix for Example: This change resolves a ValidationError that occurred when attempting to use the previous example, specifically addressing the requirement for num_speculative_tokens when a speculative model is provided.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request fixes a broken example for EAGLE speculative decoding in the documentation. The change correctly adds the missing num_speculative_tokens parameter, which resolves the ValidationError.

I've added one suggestion to include a comment in the code example. This comment clarifies how the value for num_speculative_tokens is determined, which should be helpful for users configuring other EAGLE models in the future.

gemini-code-assist · 2025-07-02T13:41:00Z

docs/features/spec_decode.md

@@ -201,6 +201,7 @@ an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https
        speculative_config={
            "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
            "draft_tensor_parallel_size": 1,
+            "num_speculative_tokens": 2,


This change correctly fixes the example. To improve clarity for users, it would be beneficial to add a comment explaining how the value for num_speculative_tokens is determined for EAGLE models. This will help them configure other EAGLE models correctly.

For yuhuili/EAGLE-LLaMA3-Instruct-8B, the num_heads in its config.json is 2, so this value is correct.

Suggested change

"num_speculative_tokens": 2,

# For EAGLE models, this should match the `num_heads` in the

# draft model's config.json.

"num_speculative_tokens": 2,

github-actions · 2025-07-02T13:42:57Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: NickLucche <nlucches@redhat.com>

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

Signed-off-by: NickLucche <nlucches@redhat.com>

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

Signed-off-by: NickLucche <nlucches@redhat.com>

update docs

246b6fd

Signed-off-by: NickLucche <nlucches@redhat.com>

NickLucche requested a review from hmellor as a code owner July 2, 2025 13:40

gemini-code-assist bot reviewed Jul 2, 2025

View reviewed changes

mergify bot added the documentation Improvements or additions to documentation label Jul 2, 2025

gemini-code-assist bot reviewed Jul 2, 2025

View reviewed changes

aarnphm approved these changes Jul 2, 2025

View reviewed changes

aarnphm enabled auto-merge (squash) July 2, 2025 15:14

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 2, 2025

simon-mo disabled auto-merge July 3, 2025 00:13

simon-mo merged commit 3dd3591 into vllm-project:main Jul 3, 2025
52 of 55 checks passed

huydhn pushed a commit to huydhn/vllm that referenced this pull request Jul 8, 2025

[Docs] Update EAGLE example (vllm-project#20375)

7b72dc9

Signed-off-by: NickLucche <nlucches@redhat.com>

LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025

[Docs] Update EAGLE example (vllm-project#20375)

f30e765

Signed-off-by: NickLucche <nlucches@redhat.com>

avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025

[Docs] Update EAGLE example (vllm-project#20375)

aa34341

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

[Docs] Update EAGLE example (vllm-project#20375)

df926dc

Signed-off-by: NickLucche <nlucches@redhat.com>

jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025

[Docs] Update EAGLE example (vllm-project#20375)

3e7f4e4

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[Docs] Update EAGLE example (vllm-project#20375)

012f398

Signed-off-by: NickLucche <nlucches@redhat.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Docs] Update EAGLE example #20375

[Docs] Update EAGLE example #20375

Uh oh!

NickLucche commented Jul 2, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 2, 2025

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Docs] Update EAGLE example #20375

[Docs] Update EAGLE example #20375

Uh oh!

Conversation

NickLucche commented Jul 2, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!