Skip to content

[Feature] Support EAGLE 3 #4247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
Mar 18, 2025
Merged

[Feature] Support EAGLE 3 #4247

merged 31 commits into from
Mar 18, 2025

Conversation

chromecast56
Copy link
Contributor

Motivation

Add support for EAGLE-3: https://arxiv.org/abs/2503.01840

Modifications

  • Add EAGLE3 speculative method to server args
  • Refactor llama.py, logits_processor.py to support capturing auxiliary hidden states
  • Modify eagle_worker.py to support EAGLE-3 token map + untied LM head
  • Add llama_eagle3.py model
  • Tests and Documentation

Checklist

@zhyncs
Copy link
Member

zhyncs commented Mar 10, 2025

cc @Liyuhui-12 @hongyanz

@chromecast56
Copy link
Contributor Author

Benchmarks on MT-Bench (bsz 1):

Autoregressive:


python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000 --cuda-graph-max-bs 1

#questions: 1, Throughput: 147.05 token/s, Acceptance length: 1.00

EAGLE-3:

python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algo EAGLE3 \
    --speculative-draft jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 \
    --cuda-graph-max-bs 1 --mem-fraction 0.7 --dtype float16 --port 30000

#questions: 80, Throughput: 336.61 token/s, Acceptance length: 4.29

@zhyncs
Copy link
Member

zhyncs commented Mar 10, 2025

@chromecast56 Can we use --speculative-num-steps 8

@chromecast56
Copy link
Contributor Author

chromecast56 commented Mar 11, 2025 •
edited
Loading

Acceptance length is calculated as num_output_tokens / num_verify_ct over the entire dataset so it should take it into account. @hongyanz Do you normalize acceptance length per prompt, or is it over the entire dataset?

@hongyanz
Copy link

hongyanz commented Mar 12, 2025 •
edited
Loading

@chromecast56 Ours is in the round level. We calculate the number of accepted tokens in each speculative decoding round, add it by 1 (because the very last token in each round will always be accepted as it is from the target model), and average the numbers across all rounds.

ispobock
ispobock approved these changes Mar 12, 2025
View reviewed changes
Copy link
Collaborator

@ispobock ispobock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhaochenyang20
Copy link
Collaborator

@chromecast56 could you fix the conflicts of the docs? @zhyncs @merrymercy could we merge it?

@zhyncs
Copy link
Member

zhyncs commented Mar 13, 2025

@merrymercy @Ying1123 reminder

@zhyncs
Copy link
Member

zhyncs commented Mar 17, 2025

Hi @chromecast56 May you help fix the conflicts

merrymercy
merrymercy approved these changes Mar 18, 2025
View reviewed changes
Copy link
Contributor

@merrymercy merrymercy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the great work.

@zhyncs zhyncs merged commit 9e0186f into sgl-project:main Mar 18, 2025
34 of 36 checks passed
@zhaochenyang20
Copy link
Collaborator

cc @simveit hey simon. Eagle 3 is merged into sglang now, yienng @zhyncs will profiling it today. Could you help to update the docs https://docs.sglang.ai/backend/speculative_decoding.html after yineng provides the performance? thanks so much!

@simveit
Copy link
Contributor

simveit commented Mar 18, 2025

@zhaochenyang20 Yes. Let me read the paper in the next days.

@simveit simveit mentioned this pull request Mar 19, 2025
Closed
This was referenced Mar 19, 2025
Closed
Closed
@finger92
Copy link
Contributor

should we change
"You can enable EAGLE-3 decoding by setting --speculative_draft_model_path: EAGLE3:"
to
"You can enable EAGLE-3 decoding by setting --speculative-algorithm EAGLE3:"
?

@zhyncs zhyncs mentioned this pull request Mar 22, 2025
Merged
6 tasks
@merrymercy
Copy link
Contributor

@ispobock can you share the exact commands to launch the servers for eagle2 and eagle3?
We do not need to benchmark on MT-bench, we can just use python3 -m sglang.test.send_one

@ispobock
Copy link
Collaborator

@merrymercy For bs=1, the launch command is here:

# EAGLE2
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algo EAGLE \
    --speculative-draft jamesliu1/sglang-EAGLE-Llama-3.1-Instruct-8B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 \
    --cuda-graph-max-bs 1 --dtype float16 --port 30000 --tp 1 --disable-radix --mem-frac 0.7

# EAGLE3
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algo EAGLE3 \
    --speculative-draft jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 8 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 \
    --cuda-graph-max-bs 1 --dtype float16 --port 30000 --tp 1 --disable-radix --mem-frac 0.7

MT-bench is the default bench dataset in EAGLE's evaluation code. It's used to keep align with the setting in the paper.
python3 -m sglang.test.send_one also works for benchmark.

@ryang-max
Copy link
Contributor

I'll also work on the docs about this feature these days.
cc @zhaochenyang20 @simveit

@weedge weedge mentioned this pull request May 13, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rkooo567 rkooo567 rkooo567 left review comments

@merrymercy merrymercy merrymercy approved these changes

@ispobock ispobock ispobock approved these changes

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@kssteven418 kssteven418 Awaiting requested review from kssteven418 kssteven418 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@zhaochenyang20 zhaochenyang20 Awaiting requested review from zhaochenyang20

Assignees

@Ying1123 Ying1123

@merrymercy merrymercy

@ispobock ispobock

Labels
high priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.