permute + conv2d crash/memory corruption on CPU in torch 2.2 #120211
Description
🐛 Describe the bug
The following code should finish in a few seconds after printing 0 through 127:
import torch
import torch.nn.functional
def reproducer(radius: int):
image = torch.rand([1, 1024, 1024, 3], dtype=torch.float32, device=torch.device('cpu'))
image = image.permute(0, 3, 1, 2)
# creating the tensor above with the appropriate shape directly does not reproduce the issue
#image = torch.rand([1, 3, 1024, 1024])
kernel_x = torch.zeros([3, 1, 1, radius * 2 + 1], device=image.device)
image = torch.nn.functional.conv2d(image, kernel_x, groups=image.shape[-3])
for i in range(0, 128):
print(i)
reproducer(radius=i)but (at least in certain environments) it crashes after printing 7. Starting at other numbers can result in different failure modes (such as burning CPU for minutes or longer).
Best guess so far is that it's connected to the CPU in use, with 3 out of 3 reporters who encountered the issue and who posted their CPU model used Ryzen 9 7000 series CPUs (7950X, 7950X, 7900X).
The attached version info is from WSL1, but I could reproduce on:
- WSL1 / Python 3.10 / torch 2.2.0
- WSL1 / Python 3.10 /
torch==2.3.0.dev20240219+cpu - Windows (native) / Python 3.10 / torch 2.2.0
- Windows (native) / Python 3.12 / torch 2.2.0
Here's the CPU section from collect_env.py on the native Windows side:
CPU:
Architecture=9
CurrentClockSpeed=4501
DeviceID=CPU0
Family=107
L2CacheSize=16384
L2CacheSpeed=
Manufacturer=AuthenticAMD
MaxClockSpeed=4501
Name=AMD Ryzen 9 7950X 16-Core Processor
ProcessorType=3
Revision=24834
The issue does not occur on torch 2.1.2.
It was originally reported in:
- Inpaint doesnt work Acly/krita-ai-diffusion#401
- Stuck at BlurMask Node Acly/comfyui-inpaint-nodes#15
Versions
PyTorch version: 2.2.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 36 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7950X 16-Core Processor
CPU family: 25
Model: 97
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
CPU max MHz: 4501.0000
CPU min MHz: 0.0000
BogoMIPS: 9002.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave osxsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl avx512vbmi umip pku avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid
Virtualization: AMD-V
Hypervisor vendor: Windows Subsystem for Linux
Virtualization type: container
Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] torch==2.2.0+cpu
[pip3] torchaudio==2.2.0+cpu
[pip3] torchvision==0.17.0+cpu
[conda] Could not collect
cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Metadata
Metadata
Assignees
Labels
Problems related to convolutions (THNN, THCUNN, CuDNN)Problem manifests as a hard crash, as opposed to a RuntimeErrorRelated to torch.nnIt used to work, and now it doesn'tEnsure you have actionable steps to reproduce the issue. Someone else needs to confirm the repro.This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Type
Projects
Status