Add Flashmask #65409
Merged
Add Flashmask #65409
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
|
Sorry to inform you that 9c65da5's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually. |
|
Sorry to inform you that efc5ce0's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually. |
…ashmask_merging
GuoxiaWang
previously approved these changes
Aug 16, 2024
| ): | ||
| is_unpad = False | ||
| else: | ||
| is_unpad = bool( |
There was a problem hiding this comment.
LGTM, 当 flashmask 识别到是 varlen 的模式时，启用 varlen 的方式执行。
| """ | ||
|
|
||
| if window_size is not None: |
There was a problem hiding this comment.
LGTM，当设置了 window_size 时，用 flashmask 直接构造 startend_row_indices，而不需要用户自己去构造。
tianshuo78520a
approved these changes
Aug 16, 2024
SigureMo
previously approved these changes
Aug 16, 2024
Comment on lines
1099
to
1114
| def flashmask_attention( | ||
| query, | ||
| key, | ||
| value, | ||
| startend_row_indices=None, | ||
| *, | ||
| dropout=0.0, | ||
| causal=False, | ||
| window_size=None, | ||
| return_softmax_lse=False, | ||
| return_seed_offset=False, | ||
| fixed_seed_offset=None, | ||
| rng_name="", | ||
| training=True, | ||
| name=None, | ||
| ): |
wanghuancoder
previously approved these changes
Aug 19, 2024
ff33521
SigureMo
previously approved these changes
Aug 19, 2024
zyfncg
previously approved these changes
Aug 20, 2024
afcfcee
raindrops2sea
approved these changes
Aug 25, 2024
SigureMo
approved these changes
Aug 25, 2024
wanghuancoder
approved these changes
Aug 26, 2024
YuanRisheng
approved these changes
Aug 26, 2024
jeff41404
approved these changes
Aug 26, 2024
sneaxiy
approved these changes
Aug 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Category
Performance Optimization
PR Types
New features
Description
Pcard-70459
重命名
flash_attention_with_sparse_mask为flashmask_attention, 重命名参数并支持更复杂的mask跳过masked block的计算以改善性能