Skip to content

Flashinfer 0.2.5 bump with torch 2.6.0 #5538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Flashinfer 0.2.5 bump with torch 2.6.0 #5538

wants to merge 2 commits into from

Conversation

AkazaAkane
Copy link
Contributor

Motivation

Modifications

Updated references to flashinfer_python in:

  • docs/start/install.md
  • python/pyproject.toml
  • python/sglang/srt/entrypoints/engine.py
  • scripts/ci_install_dependency.sh

Update BatchDecodeWithPagedKVCacheWrapper Plan (API Document) and remove cuda stream sync in:

  • python/sglang/srt/layers/attention/flashinfer_backend.py
  • python/sglang/srt/layers/attention/flashinfer_mla_backend.py

Checklist

@AkazaAkane
Copy link
Contributor Author

@sleepcoo Seems to have the same packages version conflict as urs. Any ideas on how to address that?

@Fridge003 Fridge003 mentioned this pull request Apr 10, 2025
Closed
5 tasks
@sleepcoo
Copy link
Collaborator

@sleepcoo Seems to have the same packages version conflict as urs. Any ideas on how to address that?

Here's the issue flashinfer-ai/flashinfer#911 , the flashinfer API is incompatible with the PyTorch distribution. I'm not sure what the best solution is. @zhyncs , do you have any suggestions?

@merrymercy
Copy link
Contributor

can you fix the conflicts?

@AkazaAkane
Copy link
Contributor Author

can you fix the conflicts?
The error is related to pytorch 2.6. I am able to adopt flashinfer to 0.2.5 with a lower version of torch, but I am not able to do 2.6 right now.

@zhyncs zhyncs mentioned this pull request Apr 29, 2025
Closed
2 tasks
@Fridge003 Fridge003 mentioned this pull request Apr 29, 2025
Merged
6 tasks
@zhyncs zhyncs closed this Apr 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@HandH1998 HandH1998 Awaiting requested review from HandH1998

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002

@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner

@zhaochenyang20 zhaochenyang20 Awaiting requested review from zhaochenyang20

Assignees

@zhyncs zhyncs

@Fridge003 Fridge003

Labels
high priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@AkazaAkane @sleepcoo @merrymercy @zhyncs @Fridge003