[ROCm] Update CUDAPluggableAllocator.h (#1984) #150010

amd-sriram · 2025-03-26T08:25:21Z

Altering the flag to use the correct streamType in CUDAPluggableAllocator class for ROCm gpu. The flag TORCH_HIP_VERSION does not work for ROCm as intended. This flag is replaced with USE_ROCM. This is impacting Distributed Fused Adam in Rocm/APEX when using nccl_ub feature. This has been tested with rocm/apex.

See PR ROCm/apex#184

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

) Altering the flag to use the correct streamType for CUDAPluggableAllocator. This is impacting Distributed Fused Adam in Rocm/APEX. See PR ROCm/apex#184 Related Issue : https://ontrack-internal.amd.com/browse/SWDEV-519796

pytorch-bot · 2025-03-26T08:25:24Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150010

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 26 Pending, 5 Unrelated Failures

As of commit ea4e08c with merge base de68ddc ():

NEW FAILURES - The following jobs have failed:

linux-binary-libtorch-release / libtorch-cpu-shared-with-deps-release-build / build (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 10, 10, linux.s390x) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_scatter2_cpu_cpp_wrapper
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 2, 10, linux.s390x) (gh)
RuntimeError: export/test_export_nonstrict 1/1 failed!
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 4, 10, linux.s390x) (gh)
dynamo/test_utils.py::TestDynamoTimed::test_ir_count
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 5, 10, linux.s390x) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_relu_cpu_cpp_wrapper
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 7, 10, linux.s390x) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_add_complex4_cpu_cpp_wrapper
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 8, 10, linux.s390x) (gh)
inductor/test_cpu_cpp_wrapper.py::TestCppWrapper::test_add_complex_cpu_cpp_wrapper
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 9, 10, linux.s390x) (gh)
RuntimeError: onnx/internal/test_diagnostics 1/1 failed!

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

periodic / linux-focal-cuda12.6-py3.10-gcc11 / test (nogpu_AVX512, 1, 2, lf.linux.4xlarge) (gh) (similar failure)
RuntimeError: test_overrides 1/1 failed!
periodic / linux-focal-cuda12.6-py3.10-gcc11 / test (nogpu_NO_AVX2, 1, 2, lf.linux.4xlarge) (gh) (similar failure)
RuntimeError: test_overrides 1/1 failed!

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 3, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (trunk failure)
'test/profiler/test_profiler.py::TestProfiler::test_profile_all_threads'
periodic / linux-focal-cuda12.6-py3-gcc11-slow-gradcheck / test (default, 7, 8, linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (trunk failure)
functorch/test_eager_transforms.py::TestLinearizeCUDA::test_linearize_composition_grad_cuda_float32
periodic / linux-focal-cuda12.6-py3.10-gcc11-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
/var/lib/jenkins/workspace/BUILD.bazel:434:11: Compiling aten/src/ATen/native/cuda/RowwiseScaledMM.cu failed: (Exit 1): nvcc failed: error executing command (from target //:aten_cuda) external/local_cuda/cuda/bin/nvcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++17' -MD -MF ... (remaining 320 arguments skipped)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2025-03-26T08:25:27Z

The committers listed above are authorized under a signed CLA.

✅ login: amd-sriram / name: Sriram Kumar (ea4e08c)

amd-sriram · 2025-03-26T08:40:52Z

@pytorchbot label "release notes: rocm"

pytorch-bot · 2025-03-26T10:19:56Z

To add the ciflow label ciflow/rocm please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-03-26T10:19:57Z

To add the ciflow label ciflow/periodic please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-03-26T10:19:57Z

To add the ciflow label ciflow/inductor-rocm please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

amd-sriram · 2025-04-01T16:09:17Z

@pytorchbot merge

pytorchmergebot · 2025-04-01T16:11:21Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-04-01T16:11:54Z

Merge failed

Reason: 7 jobs have failed, first few of them are: s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 2, 10, linux.s390x), s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 4, 10, linux.s390x), s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 5, 10, linux.s390x), s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 7, 10, linux.s390x), s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 8, 10, linux.s390x)

Details for Dev Infra team

Raised by workflow job

jeffdaily · 2025-04-01T16:46:55Z

@pytorchbot merge -f "unrelated CI failures"

pytorchmergebot · 2025-04-01T16:48:35Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Altering the flag to use the correct streamType in CUDAPluggableAllocator class for ROCm gpu. The flag TORCH_HIP_VERSION does not work for ROCm as intended. This flag is replaced with USE_ROCM. This is impacting Distributed Fused Adam in Rocm/APEX when using nccl_ub feature. This has been tested with rocm/apex. See PR ROCm/apex#184 Pull Request resolved: pytorch#150010 Approved by: https://github.com/jeffdaily

jithunnair-amd · 2025-05-20T19:44:15Z

@pytorchbot cherry-pick --onto release/2.7

pytorch-bot · 2025-05-20T19:44:17Z

❌ 🤖 pytorchbot command failed:

@pytorchbot cherry-pick: error: the following arguments are required: -c/--classification

usage: @pytorchbot cherry-pick --onto ONTO [--fixes FIXES] -c
                               {regression,critical,fixnewfeature,docs,release}

Try @pytorchbot --help for more info.

jithunnair-amd · 2025-05-20T19:44:58Z

@pytorchbot cherry-pick --onto release/2.7 -c fixnewfeature

Altering the flag to use the correct streamType in CUDAPluggableAllocator class for ROCm gpu. The flag TORCH_HIP_VERSION does not work for ROCm as intended. This flag is replaced with USE_ROCM. This is impacting Distributed Fused Adam in Rocm/APEX when using nccl_ub feature. This has been tested with rocm/apex. See PR ROCm/apex#184 Pull Request resolved: #150010 Approved by: https://github.com/jeffdaily (cherry picked from commit a19b667)

pytorchbot · 2025-05-20T19:49:56Z

Cherry picking #150010

The cherry pick PR is at #153974 and it is recommended to link a fixnewfeature cherry pick PR with an issue. The following tracker issues are updated:

[v2.7.1] Release Tracker #152627 (comment)

Details for Dev Infra team

Raised by workflow job

[ROCm] Update CUDAPluggableAllocator.h (#1984) (#150010) Altering the flag to use the correct streamType in CUDAPluggableAllocator class for ROCm gpu. The flag TORCH_HIP_VERSION does not work for ROCm as intended. This flag is replaced with USE_ROCM. This is impacting Distributed Fused Adam in Rocm/APEX when using nccl_ub feature. This has been tested with rocm/apex. See PR ROCm/apex#184 Pull Request resolved: #150010 Approved by: https://github.com/jeffdaily (cherry picked from commit a19b667) Co-authored-by: Sriram Kumar <skishore@amd.com>

pytorch-bot bot added the module: rocm AMD GPU support for Pytorch label Mar 26, 2025

pytorchbot added the open source label Mar 26, 2025

amd-sriram marked this pull request as ready for review March 26, 2025 08:32

amd-sriram requested review from eqy and syed-ahmed as code owners March 26, 2025 08:32

pytorch-bot bot added the release notes: rocm mandatorylabel label Mar 26, 2025

jataylo added ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/rocm Trigger "default" config CI on ROCm ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Mar 26, 2025

pytorch-bot bot removed ciflow/rocm Trigger "default" config CI on ROCm ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Mar 26, 2025

jataylo added ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/rocm Trigger "default" config CI on ROCm ciflow/inductor-rocm Trigger "inductor" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Mar 26, 2025

soulitzer requested a review from jeffdaily March 26, 2025 22:45

soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Mar 26, 2025

jeffdaily approved these changes Mar 27, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 1, 2025

pytorchmergebot added the merging label Apr 1, 2025

pytorchmergebot removed the merging label Apr 1, 2025

pytorchmergebot added the merging label Apr 1, 2025

pytorchmergebot closed this in a19b667 Apr 1, 2025

pytorchmergebot added Merged and removed merging labels Apr 1, 2025

pytorchbot mentioned this pull request May 20, 2025

[v2.7.1] Release Tracker #152627

Closed

jithunnair-amd requested a review from atalman May 20, 2025 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ROCm] Update CUDAPluggableAllocator.h (#1984) #150010

[ROCm] Update CUDAPluggableAllocator.h (#1984) #150010

Uh oh!

amd-sriram commented Mar 26, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Mar 26, 2025 •

edited

Loading

Uh oh!

linux-foundation-easycla bot commented Mar 26, 2025 •

edited

Loading

Uh oh!

amd-sriram commented Mar 26, 2025

Uh oh!

pytorch-bot bot commented Mar 26, 2025

Uh oh!

pytorch-bot bot commented Mar 26, 2025

Uh oh!

pytorch-bot bot commented Mar 26, 2025

Uh oh!

amd-sriram commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Uh oh!

jeffdaily commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Uh oh!

jithunnair-amd commented May 20, 2025

Uh oh!

pytorch-bot bot commented May 20, 2025

Uh oh!

jithunnair-amd commented May 20, 2025

Uh oh!

pytorchbot commented May 20, 2025

Uh oh!

Uh oh!

[ROCm] Update CUDAPluggableAllocator.h (#1984) #150010

[ROCm] Update CUDAPluggableAllocator.h (#1984) #150010

Uh oh!

Conversation

amd-sriram commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150010

❌ 8 New Failures, 26 Pending, 5 Unrelated Failures

Uh oh!

linux-foundation-easycla bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amd-sriram commented Mar 26, 2025

Uh oh!

pytorch-bot bot commented Mar 26, 2025

Uh oh!

pytorch-bot bot commented Mar 26, 2025

Uh oh!

pytorch-bot bot commented Mar 26, 2025

Uh oh!

amd-sriram commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Merge started

Uh oh!

pytorchmergebot commented Apr 1, 2025

Merge failed

Uh oh!

jeffdaily commented Apr 1, 2025

Uh oh!

pytorchmergebot commented Apr 1, 2025

Merge started

Uh oh!

jithunnair-amd commented May 20, 2025

Uh oh!

pytorch-bot bot commented May 20, 2025

Uh oh!

jithunnair-amd commented May 20, 2025

Uh oh!

pytorchbot commented May 20, 2025

Cherry picking #150010

Uh oh!

Uh oh!

amd-sriram commented Mar 26, 2025 •

edited

Loading

pytorch-bot bot commented Mar 26, 2025 •

edited

Loading

linux-foundation-easycla bot commented Mar 26, 2025 •

edited

Loading