Skip to content

[WIP][Feautre, Hardware] add initial suport for ascend npu #3782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[WIP][Feautre, Hardware] add initial suport for ascend npu #3782

wants to merge 1 commit into from

Conversation

22dimensions
Copy link
Contributor

@22dimensions 22dimensions commented Feb 22, 2025 •
edited
Loading

Motivation

see issue #3781

Modifications

  1. add new device args 'npu'
  2. import torch_npu and adapt torch related code
  3. add usage document

Checklist

Examples

python -m sglang.launch_server --model-path LLM-Research/Llama-3.2-1B-Instruct  --device npu --grammar-backend outlines --attention-backend torch_native  --port 30000 --host 0.0.0.0

image

@22dimensions 22dimensions mentioned this pull request Feb 22, 2025
Closed
2 tasks
@22dimensions 22dimensions changed the title [WIP][Feautre, Hardware] add suport for ascend npu [WIP][Feautre, Hardware] add initial suport for ascend npu Feb 22, 2025
@botieking98 botieking98 mentioned this pull request Feb 25, 2025
Merged
@22dimensions
Copy link
Contributor Author

#3853 is merged, so i close this pr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy will be requested when the pull request is marked ready for review merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 will be requested when the pull request is marked ready for review Ying1123 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 will be requested when the pull request is marked ready for review hnyls2002 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs will be requested when the pull request is marked ready for review zhyncs is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock will be requested when the pull request is marked ready for review ispobock is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@22dimensions