Skip to content

Eagle speculative decoding part 1: Support target model verification in the attention backend #2678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 31, 2024
Merged

Eagle speculative decoding part 1: Support target model verification in the attention backend #2678

merged 1 commit into from
Dec 31, 2024

Conversation

merrymercy
Copy link
Contributor

@merrymercy merrymercy commented Dec 31, 2024 •
edited
Loading

This is used for speculative decoding in #2150

Co-authored-by: yukavio <kavioyu@gmail.com>

@merrymercy merrymercy changed the title Support target verify in the attention backend Support target model verification in the attention backend Dec 31, 2024
@merrymercy merrymercy merged commit f44d143 into main Dec 31, 2024
16 of 17 checks passed
@merrymercy merrymercy deleted the pr-attn branch December 31, 2024 06:58
@zhyncs
Copy link
Member

zhyncs commented Dec 31, 2024

@merrymercy test_dp_attention failed after this PR

python3 test_dp_attention.py

@zhyncs
Copy link
Member

zhyncs commented Dec 31, 2024

FYI After I revert git revert f44d143949f6c6fbca6cb96c52381b8bc1769a87, it works well cc @ispobock

@merrymercy merrymercy changed the title Support target model verification in the attention backend Eagle speculative decoding part 1. Support target model verification in the attention backend Dec 31, 2024
@merrymercy merrymercy changed the title Eagle speculative decoding part 1. Support target model verification in the attention backend Eagle speculative decoding part 1: Support target model verification in the attention backend Dec 31, 2024
@merrymercy merrymercy mentioned this pull request Dec 31, 2024
Merged
@merrymercy
Copy link
Contributor Author

The hanging issue is fixed by #2684

@merrymercy merrymercy mentioned this pull request Jan 2, 2025
Merged
XiaotongJiang pushed a commit to XiaotongJiang/sglang that referenced this pull request Jan 3, 2025
@merrymercy @yukavio @XiaotongJiang
Support target model verification in the attention backend (sgl-proje…
61fbf28 
…ct#2678)


Co-authored-by: yukavio <kavioyu@gmail.com>
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
@merrymercy @yukavio
Support target model verification in the attention backend (sgl-proje…
e96e62e 
…ct#2678)


Co-authored-by: yukavio <kavioyu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@merrymercy @zhyncs