@rhatdan @jeremyeder @lmilbaum FYI

@lmilbaum @rhatdan These is an e2e test we'd run in https://github.com/containers/ai-lab-recipes/ ... related to containers/ai-lab-recipes#341

This work needs #1018 to fix the container first.

The work is incomplete right now, with lots of TODO statements indicating what remains to be done.

@russellb The Container file also needs this for cuda:

-RUN dnf install -y libcudnn8 nvidia-driver-NVML
+RUN dnf install -y libcudnn8 nvidia-driver-NVML nvidia-driver-cuda-libs

Is this ready for broader review and testing? Did a CI run work successfully in GitHub?

Is this ready for broader review and testing? Did a CI run work successfully in GitHub?

I don’t have the CI job passing yet. I’ll update again later today.

My last push included a rebase to be on top of a bunch of newer changes. Some outstanding issues:

• I defined an e2e CI job, but it doesn't work reliably. I don't think the github action runners have enough CPU to finish even the most minimal configuration. I'm going to switch to looking at running it on a GPU runner instead. The tradeoff will be that it won't run on every PR for now, since it's not free.
• Eventually, we'll squash all of this, but that's for when it's ready for review and merge ... I'd like to have a working CI job ready for this repo first though

The end-to-end CI job that does init/download/generate/train on Linux is passing. This is running on the host OS using a GPU worker available via github actions.

Next steps:

• I'm going to clean up this branch by squashing most of it together and putting it in a form ready for review and merging this first workflow
• Next I want to add a variant that instead of running it on the host, builds a cuda container using our containerfile and runs the same workflow using the container. I'd like to do that in a follow-up PR, though.

I applied the hold label because:

• This PR includes linux-train: Skip convert to gguf when --4-bit-quant is used #1058, so that should be reviewed first
• There is a commit on the branch that must be removed before merging (the change to make the job run automatically against this PR). The intent is to only merge it as a manually triggered workflow.
• I have a hack commit applied to avoid an error I'm hitting during ilab train, but I need to figure out the proper fix for it

Note that the workflow is currently limited and stops before ensuring you can run ilab serve with a model after ilab train because of #579

A data point on costs. The last two successful runs took 22 minutes. That costs us $1.54 each time ($0.07 per minute). If we run this only manually, I think the costs will stay under control.

I also think we could speed this up. A lot of time is spent on downloading and installing stuff. No caching is done in the workflow, yet.

For caching I recommend to use setup-python with cache pip and actions/cache to cache the Hugging Face models. We are using the same technique in test.yaml. You could also cache the APT packages with something like https://github.com/marketplace/actions/cache-apt-packages

      - name: Setup Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: py311
          cache: pip
          cache-dependency-path: |
            **/pyproject.toml
            **/requirements*.txt

      - name: Cache huggingface
        uses: actions/cache@v4
        with:
          path: ~/.cache/huggingface
          # config contains DEFAULT_MODEL
          key: huggingface-${{ hashFiles('src/instructlab/config.py') }}

Are you sure the PR uses a CUDA-accelerated build of llama-cpp-python? It downloads and builds llama-cpp-python two times:

Collecting llama-cpp-python
  Downloading llama_cpp_python-0.2.68.tar.gz (42.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.5/42.5 MB 170.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'

 Collecting llama_cpp_python[server]==0.2.55
  Downloading llama_cpp_python-0.2.55.tar.gz (36.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.8/36.8 MB 172.1 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'

I think you are overriding the CUDA build with a CPU-only build. Our installation instructions have sharp edges.

Please try this:

sed 's/\[.*\]//' requirements.txt constraints.txt
python3 -m pip cache remove llama_cpp_python
python3 -m pip install --no-binary llama_cpp_python -c constraints.txt llama_cpp_python

Are you sure the PR uses a CUDA-accelerated build of llama-cpp-python?

I guess I'm not! I was going off of the output from ilab train confirming that it was using the GPU, but I wasn't thinking that ilab serve might be different. Thank you.

LGTM - I'd love if we could incorporate this into the release workflow somehow - maybe having an automic trigger when we cut a new release branch for Y-streams or for Backport PRs to an existing release branch (to ensure Z-streams have no regressions)

Yeah, that's a good idea. We could definitely trigger it on tag creation. At the moment, it would be triggering it manually, and using the input field that lets you specify which branch or tag to run against. You't put in release-v0.15 or v0.15.0 or whatever you want and it'll run for you.