Skip to content

fix: apply cache size limit of attention mask for VisionAttention #3657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 19, 2025
Merged

fix: apply cache size limit of attention mask for VisionAttention #3657

merged 2 commits into from
Feb 19, 2025

Conversation

mickqian
Copy link
Collaborator

@mickqian mickqian commented Feb 18, 2025 •
edited
Loading

Motivation

enforce an upper-bound limit for VisionAttention mask cache size.

This is included in #3203, and moved to here.

Modifications

Checklist

@mickqian
Copy link
Collaborator Author

mickqian commented Feb 18, 2025 •
edited
Loading

ref #3651

@mickqian mickqian mentioned this pull request Feb 18, 2025
Closed
5 tasks
@yizhang2077 yizhang2077 self-assigned this Feb 19, 2025
@yizhang2077 yizhang2077 self-requested a review February 19, 2025 06:26
yizhang2077
yizhang2077 reviewed Feb 19, 2025
View reviewed changes
Copy link
Collaborator

@yizhang2077 yizhang2077 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pls run benckmark like mmmu to verify accuracy and build an OOM case to check if this pr has solved oom problem. Thanks! cc @zhaochenyang20

@mickqian mickqian changed the title fix: apply cache size limit for VisionAttention fix: apply cache size limit of attention mask for VisionAttention Feb 19, 2025
@zhaochenyang20
Copy link
Collaborator

@yizhang2077 Will merge it after the CI.

@zhyncs zhyncs merged commit 99c1b9d into sgl-project:main Feb 19, 2025
17 of 19 checks passed
@Lzhang-hub
Copy link
Contributor

@mickqian I use the latest version run qwen2.5-vl-7b model with command

python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --port 8080  --chat-template qwen2-vl --chunked-prefill-size -1 --disable-radix-cache --mm-attention-backend fa3 --attention-backend fa3  --enable-torch-compile --cuda-graph-bs 80 --torch-compile-max-bs 80

then benchmark server with concurrency=80, after run sometime, server got OOM error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yizhang2077 yizhang2077 yizhang2077 left review comments

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

Assignees

@yizhang2077 yizhang2077

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mickqian @zhaochenyang20 @Lzhang-hub @yizhang2077 @zhyncs