Skip to content

Bump nvidia driver for CUDA 12.1 support #16557

@ddelange

Description

@ddelange

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

Bump the nvidia driver for CUDA 12.1 support.

We are currently running our 1.26 cluster configured with DriverPackage nvidia-driver-535 (with CUDA 12.0 support).

Note that we moved away from nvidia-headless-XXX-server because it does not install nvidia-smi and some other binaries on the host system, which caused issues for us with some cuda docker images that rely on the host having them available. It's barely any additional disk space used, so that was a quick and effective fix.

When we tried bumping to nvidia-driver-550 on EC2, the nodes stopped registering in the cluster. It was reverted before I could pull logs from the host system I'm afraid. I hope that your CI catches this.

2. Feel free to provide a design supporting your feature request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions