Skip to content

CUDA Error: Invalid Argument When Using Interactive Prompts in Docker Container #37

@nilsrehtanz

Description

@nilsrehtanz

Thank you very much for this amazing tool! It really helps a lot! In the newest version of the Docker setup, I have this error:

When attempting to use point or scribble interactions on the SlicerNNInteractive server, operations fail with a CUDA error when creating tensors using the Docker setup.

Environment Information

  • Docker Image: coendevente/nninteractive-slicer-server:latest
  • CUDA Version: 12.1.0
  • Python Version: 3.12 (conda environment: nnInteractive)
  • GPU: Full GPU access (not virtualized/partial)
  • Input Image Dimensions: 797×512×512 (from server logs)
  • Operating System: Linux (Jetstream cloud instance)

Detailed Error

The server fails with RuntimeError: CUDA error: invalid argument when trying to create a CUDA tensor with torch.zeros() in the _initialize_interactions method. This happens after successfully uploading an image and attempting to add interactive prompts.

Steps to Reproduce

  1. Start the Docker container: docker run --gpus all --rm -it -p 1527:1527 coendevente/nninteractive-slicer-server:latest
  2. Connect to the server from 3D Slicer using the nnInteractive extension
  3. Upload an image (in my case, with dimensions 797×512×512)
  4. Try to add a point or scribble interaction
  5. Server crashes with the CUDA error shown in the stack trace

Error Stack Trace

RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

The error specifically occurs in:

File "/opt/conda/envs/nnInteractive/lib/python3.12/site-packages/nnInteractive/inference/inference_session.py", line 140, in _initialize_interactions
  self.interactions = torch.zeros(

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions