Skip to content

Add Windows CUDA 12.9.1 build #155748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 9 commits into from
Closed

Add Windows CUDA 12.9.1 build #155748

wants to merge 9 commits into from

Conversation

tinglvv
Copy link
Collaborator

@tinglvv tinglvv commented Jun 11, 2025 •
edited
Loading

Adding build script, after Windows AMI is deployed

Issue - #155196

cc @atalman @ptrblck @nWEIdia

@tinglvv tinglvv requested a review from a team as a code owner June 11, 2025 22:19
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 11, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155748

Note: Links to docs will display an error until the docs builds have been completed.

❌ 73 New Failures

As of commit 3ed6962 with merge base e9fdaf8 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jun 11, 2025
@tinglvv tinglvv mentioned this pull request Jun 11, 2025
Open
12 tasks
@atalman
Copy link
Contributor

atalman commented Jun 11, 2025

HI @tinglvv I believe build scripts need to be added as well: https://github.com/pytorch/pytorch/blob/main/.ci/pytorch/windows/cuda128.bat

@janeyx99 janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 13, 2025
@atalman
Copy link
Contributor

atalman commented Jun 16, 2025

@pytorchmergebot rebase -b main

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased win-cu129-build onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout win-cu129-build && git pull --rebase)

@atalman atalman added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Jun 16, 2025
@tinglvv
Copy link
Collaborator Author

tinglvv commented Jun 16, 2025 •
edited
Loading

Running into OOM error with

2025-06-16T22:12:59.7536205Z [6891/7628] Building CUDA object caffe2\CMakeFiles\torch_cuda.dir\__\aten\src\ATen\native\cuda\SegmentReduce.cu.obj
2025-06-16T22:12:59.7537559Z FAILED: caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/SegmentReduce.cu.obj 
2025-06-16T22:12:59.7576033Z C:\actions-runner\_work\pytorch\pytorch\pytorch\.ci\pytorch\windows\\tmp_bin\randomtemp.exe C:\actions-runner\_work\pytorch\pytorch\pytorch\.ci\pytorch\windows\\tmp_bin\sccache.exe C:\PROGRA~1\NVIDIA~2\CUDA\v12.9\bin\nvcc.exe -forward-unknown-to-host-compiler -DAT_PER_OPERATOR_HEADERS -DEXPORT_AOTI_FUNCTIONS -DFMT_HEADER_ONLY=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNOMINMAX -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_C10D_GLOO -DUSE_CUDA -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_MEM_EFF_ATTENTION -DUSE_MIMALLOC -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_DEPRECATE=1 -D_UCRT_LEGACY_INFINITY -Dtorch_cuda_EXPORTS -IC:\actions-runner\_work\pytorch\pytorch\pytorch\build\aten\src -IC:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src -IC:\actions-runner\_work\pytorch\pytorch\pytorch\build -IC:\actions-runner\_work\pytorch\pytorch\pytorch -IC:\actions-runner\_work\pytorch\pytorch\pytorch\nlohmann -IC:\actions-runner\_work\pytorch\pytorch\pytorch\moodycamel -IC:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\mimalloc\include -IC:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\THC -IC:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\cuda -IC:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\fmt\include -IC:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\..\..\..\third_party\cutlass\include -IC:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\..\..\..\third_party\cutlass\tools\util\include -IC:\actions-runner\_work\pytorch\pytorch\pytorch\build\caffe2\aten\src -IC:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\.. -IC:\actions-runner\_work\pytorch\pytorch\pytorch\c10\cuda\..\.. -IC:\actions-runner\_work\pytorch\pytorch\pytorch\c10\.. -IC:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\api -IC:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\api\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\build\third_party\gloo -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\cmake\..\third_party\gloo -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\cmake\..\third_party\googletest\googlemock\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\cmake\..\third_party\googletest\googletest\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\protobuf\src -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\.ci\pytorch\windows\Python\Library\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\XNNPACK\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\ittapi\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\cmake\..\third_party\eigen -isystem "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\include" -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\ideep\mkl-dnn\include\oneapi\dnnl -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\ideep\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\INTERFACE -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\nlohmann\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\concurrentqueue -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\third_party\NVTX\c\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\cmake\..\third_party\cudnn_frontend\include -isystem C:\actions-runner\_work\pytorch\pytorch\pytorch\.ci\pytorch\windows\magma_cuda129_release\include -DLIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS -Xcompiler  /Zc:__cplusplus -Xcompiler /w -w -Xcompiler /FS -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch --use-local-env -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_100,code=sm_100 -gencode arch=compute_120,code=sm_120 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=bad_friend_decl --Werror cross-execution-space-call --no-host-device-move-forward --expt-relaxed-constexpr --expt-extended-lambda -Xfatbin -compress-all -Xcompiler=/wd4819,/wd4503,/wd4190,/wd4244,/wd4251,/wd4275,/wd4522 -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUB_WRAPPED_NAMESPACE=at_cuda_detail -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -Xcompiler="-O2 -Ob2" -DNDEBUG -Xcompiler /MD -std=c++17 -Xcompiler=-MD -Xcompiler=-Z7 -DMKL_HAS_SBGEMM -DMKL_HAS_SHGEMM -DCAFFE2_USE_GLOO -MD -MT caffe2\CMakeFiles\torch_cuda.dir\__\aten\src\ATen\native\cuda\SegmentReduce.cu.obj -MF caffe2\CMakeFiles\torch_cuda.dir\__\aten\src\ATen\native\cuda\SegmentReduce.cu.obj.d -x cu -c C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\cuda\SegmentReduce.cu -o caffe2\CMakeFiles\torch_cuda.dir\__\aten\src\ATen\native\cuda\SegmentReduce.cu.obj -Xcompiler=-Fdcaffe2\CMakeFiles\torch_cuda.dir\,-FS
2025-06-16T22:12:59.7613574Z LLVM ERROR: out of memory
2025-06-16T22:12:59.7614018Z SegmentReduce.cu
2025-06-16T22:12:59.7614525Z nvcc error   : '""%CICC_PATH%\cicc"' died with status 0xC0000409

https://github.com/pytorch/pytorch/actions/runs/15687710761/job/44203981629?pr=155748

@atalman atalman mentioned this pull request Jun 17, 2025
Closed
@tinglvv tinglvv requested review from eqy and syed-ahmed as code owners June 17, 2025 22:06
atalman
atalman reviewed Jun 19, 2025
View reviewed changes
@tinglvv tinglvv closed this Aug 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@atalman atalman atalman left review comments

@eqy eqy Awaiting requested review from eqy eqy is a code owner

@syed-ahmed syed-ahmed Awaiting requested review from syed-ahmed syed-ahmed is a code owner

Assignees
No one assigned
Labels
ciflow/binaries Trigger all binary build and upload jobs on the PR open source topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@tinglvv @atalman @pytorchmergebot @pytorchbot @janeyx99