Refactor CUDA Container file #994

tiran · 2024-04-25T07:27:06Z

Changes

Which issue is resolved by this Pull Request:
Resolves #899
Resolves #932

Description of your changes:

Modify the Containerfile for NVIDIA CUDA to:

use multi-stage build with CUDA runtime image for the final stage
update to CUDA 12.4.1
install the Python packages in a virtualenv that is copied in the final container
mimic the layout and behavior of ubi9/python-311 container
make the final container rootless
install flash-attn and bitesandbytes

Propose a shell alias that should simplify the user experience when using the container image, avoiding using long podman run commands.

Modify the containerization documentation to explain this and provide steps for the user.

Update the cuda target in the Makefile to reflect the changes above.

The changeset is co-authored by Fabien Dupont and is built on top of previous work by Charlie Doern and Fabien Dupont.

containers/cuda/ilab-cuda.alias

docs/containerization.md

containers/cuda/ilab-cuda.alias

Makefile

cdoern

nice cleanup! One thing I am unsure of is the new flow in which you podman run for each command you want to run. I would be ok with this as long as there was another build target that installed ilab but the entrypoint was just bin/bash

Makefile

containers/cuda/Containerfile

docs/containerization.md

containers/cuda/Containerfile

docs/containerization.md

luis5tb

Other than the couple of comments, this works well

docs/containerization.md

stefwalter · 2024-04-26T18:33:58Z

@tiran FYI, I also needed this package nvidia-driver-cuda-libs ... not sure if that's accounted for here.

tiran · 2024-04-26T18:38:07Z

@stefwalter In the final image or on the host system?

stefwalter · 2024-04-26T18:47:25Z

@stefwalter In the final image or on the host system?

In the image. Without nvidia-driver-cuda-libs

[root@8a706bc26838 instructlab]# podman3.11
bash: podman3.11: command not found
[root@8a706bc26838 instructlab]# python3.11
Python 3.11.5 (main, Sep  7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
torc>>> torch.cuda.device_count()
1
>>> torch.cuda.is_available()
False

With the library installed then is_available() returns True.

docs/containerization.md

tiran · 2024-04-28T13:26:08Z

I have simplified the CUDA container file even more. It's no longer using rootless. While rootless is more secure, it's harder to use, too. Let's keep it simple for now and improve it later. You can fetch a ready-to-use image from quay.io/tiran/instructlab-containers:cuda-ubi9.

mergify · 2024-05-02T14:41:52Z

This pull request has merge conflicts that must be resolved before it can be
merged. @tiran please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Christian Heimes <cheimes@redhat.com>

mergify · 2024-05-07T00:02:33Z

This pull request has merge conflicts that must be resolved before it can be
merged. @tiran please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Add some flags taken from instructlab/instructlab#994

cdoern · 2024-05-16T14:42:49Z

@tiran does this still only have the capability to only run 1 cmd per container run? if so, I would really vote to add back the old workflow of exec'ing into a running ctr with ilab installed.

I may have lost context here so feel free to correct me :)

mergify · 2024-05-16T15:03:05Z

This pull request has merge conflicts that must be resolved before it can be
merged. @tiran please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

nathan-weinberg · 2024-05-28T18:23:56Z

@tiran any update on this?

tiran · 2024-07-09T06:28:27Z

Fabien is taking care of CUDA container. Closing this stale PR.

This change modifies the Containerfile for NVIDIA CUDA to: - Use CentOS Stream 9 as the default base image. UBI9 also works - Install the NVIDIA packages from NVIDIA repository for exact CUDA version - Install the Python packages in a Python 3.11 virtualenv - Install instructlab package from PyPI - Allow specifying the instructlab version at build time - Make the final container rootless Resolves instructlab#899 Replaces instructlab#992 Replaces instructlab#994 Signed-off-by: Fabien Dupont <fdupont@redhat.com>

This change modifies the Containerfile for NVIDIA CUDA to: - Use CentOS Stream 9 as the default base image. UBI9 also works - Install the NVIDIA packages from NVIDIA repository for exact CUDA version - Add NVIDIA scripts from `nvcr.io/nvidia/cuda` image for copyrights... - Install the Python packages in a Python 3.11 virtualenv - Install instructlab package from PyPI - Allow specifying the instructlab version at build time - Make the final container rootless Resolves instructlab#899 Replaces instructlab#992 Replaces instructlab#994 Signed-off-by: Fabien Dupont <fdupont@redhat.com>

This change modifies the Containerfile for NVIDIA CUDA to: - Use CentOS Stream 9 as the default base image. UBI9 also works - Install the NVIDIA packages from NVIDIA repository for exact CUDA version - Add NVIDIA scripts from `nvcr.io/nvidia/cuda` image for copyrights... - Install the Python packages in a Python 3.11 virtualenv - Install instructlab package from PyPI - Allow specifying the instructlab version at build time - Make the final container rootless Resolves #899 Replaces #932 Replaces #994 Approved-by: nathan-weinberg Approved-by: markstur

github-actions bot added the documentation Improvements or additions to documentation label Apr 25, 2024

tiran changed the title ~~Refactor cuda~~ Refactor CUDA Container file Apr 25, 2024

tiran force-pushed the refactor-cuda branch from 8698588 to 09bba7b Compare April 25, 2024 07:39

leseb reviewed Apr 25, 2024

View reviewed changes

tiran force-pushed the refactor-cuda branch 2 times, most recently from 33599f0 to eea9eab Compare April 25, 2024 10:08

tiran marked this pull request as ready for review April 25, 2024 10:08

leseb reviewed Apr 25, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

tiran force-pushed the refactor-cuda branch from eea9eab to 24cea95 Compare April 25, 2024 12:34

cdoern reviewed Apr 25, 2024

View reviewed changes

cdoern added this to the Release - 5/30 milestone Apr 25, 2024

nathan-weinberg mentioned this pull request Apr 25, 2024

Container ships build dependencies #899

Closed

tiran force-pushed the refactor-cuda branch from d7f93c5 to 188fe29 Compare April 26, 2024 07:00

luis5tb reviewed Apr 26, 2024

View reviewed changes

docs/containerization.md Outdated Show resolved Hide resolved

luis5tb reviewed Apr 26, 2024

View reviewed changes

docs/containerization.md Outdated Show resolved Hide resolved

docs/containerization.md Show resolved Hide resolved

n1hility reviewed Apr 27, 2024

View reviewed changes

docs/containerization.md Outdated Show resolved Hide resolved

tiran force-pushed the refactor-cuda branch 2 times, most recently from f68ed6a to 6498f78 Compare April 28, 2024 12:35

github-actions bot added the testing Relates to testing label Apr 28, 2024

mergify bot added CI/CD Affects CI/CD configuration container Affects containization aspects labels Apr 29, 2024

tiran force-pushed the refactor-cuda branch from 6498f78 to 0a60bb1 Compare April 30, 2024 13:36

tiran mentioned this pull request Apr 30, 2024

Enable building the container on AArch64 also #1043

Closed

tiran force-pushed the refactor-cuda branch from 52b1734 to 4df5bbf Compare April 30, 2024 14:30

mergify bot added the needs-rebase This Pull Request needs to be rebased label May 2, 2024

tiran added 2 commits May 6, 2024 22:29

setsebool container_use_devices

d9791ec

Signed-off-by: Christian Heimes <cheimes@redhat.com>

PKG_CACHE, -DLLAMA_NATIVE=off

ad3e1c6

Signed-off-by: Christian Heimes <cheimes@redhat.com>

tiran force-pushed the refactor-cuda branch from 03daf78 to ad3e1c6 Compare May 6, 2024 20:29

mergify bot added needs-rebase This Pull Request needs to be rebased and removed needs-rebase This Pull Request needs to be rebased labels May 6, 2024

mergify bot removed the needs-rebase This Pull Request needs to be rebased label May 7, 2024

dhellmann added a commit to dhellmann/rebuilding-the-wheel that referenced this pull request May 15, 2024

add additional cuda flags for llama-cpp-python

992b7e8

Add some flags taken from instructlab/instructlab#994

cdoern added the hold In-progress PR. Tag should be removed before merge. label May 16, 2024

mergify bot added the needs-rebase This Pull Request needs to be rebased label May 16, 2024

nathan-weinberg removed this from the Release - 5/30 milestone May 29, 2024

tiran closed this Jul 9, 2024

fabiendupont mentioned this pull request Aug 2, 2024

Refactor the CUDA container spec #1961

Merged

Refactor CUDA Container file #994

Refactor CUDA Container file #994

Uh oh!

Conversation

tiran commented Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdoern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luis5tb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stefwalter commented Apr 26, 2024

Uh oh!

tiran commented Apr 26, 2024

Uh oh!

stefwalter commented Apr 26, 2024

Uh oh!

Uh oh!

tiran commented Apr 28, 2024

Uh oh!

mergify bot commented May 2, 2024

Uh oh!

mergify bot commented May 7, 2024

Uh oh!

cdoern commented May 16, 2024

Uh oh!

mergify bot commented May 16, 2024

Uh oh!

nathan-weinberg commented May 28, 2024

Uh oh!

tiran commented Jul 9, 2024

Uh oh!

Uh oh!

tiran commented Apr 25, 2024 •

edited

Loading