Document Model-As-OCI-Artifacts feature and reorganize proposals #1356
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: chewong <10557231+chewong@users.noreply.github.com>
… page Co-authored-by: chewong <10557231+chewong@users.noreply.github.com>
Copilot finished work on behalf of
chewong
August 5, 2025 17:54
chewong
requested changes
Aug 5, 2025
|
|
||
| ## Overview | ||
|
|
||
| Currently, KAITO employs a solution where the runtime library and model files are packaged within a single image. This method ensures a reliable and self-contained environment, particularly effective for distributing small models. However, as large language models grow, bundling them within containerized images becomes impractical. |
There was a problem hiding this comment.
Suggested change
| Currently, KAITO employs a solution where the runtime library and model files are packaged within a single image. This method ensures a reliable and self-contained environment, particularly effective for distributing small models. However, as large language models grow, bundling them within containerized images becomes impractical. | |
| Currently, KAITO employs a solution where the runtime library and model files are packaged within a single container image. This method ensures a reliable and self-contained environment, particularly effective for distributing small models. However, as large language models grow, bundling them within containerized images becomes impractical. |
|
|
||
| The system uses an initContainer to download model files as OCI artifacts using ORAS: | ||
|
|
||
| ```mermaid |
There was a problem hiding this comment.
Need this to render mermaid diagram in website:
export default {
markdown: {
mermaid: true,
},
themes: ['@docusaurus/theme-mermaid'],
};
|
|
||
| Traditional containerized model distribution faces several challenges: | ||
|
|
||
| - **Build Time**: With KAITO hosting multiple models, base images are frequently updated due to vulnerability fixes and feature requests. Each time the base image is updated, every model image needs to be rebuilt. |
There was a problem hiding this comment.
Suggested change
| - **Build Time**: With KAITO hosting multiple models, base images are frequently updated due to vulnerability fixes and feature requests. Each time the base image is updated, every model image needs to be rebuilt. | |
| - **Build Time**: With KAITO hosting multiple preset models, base images are frequently updated due to vulnerability fixes and feature requests. Each time the base image is updated, every model image needs to be rebuilt. |
|
|
||
| ### 1. Build Image Using ORAS Push | ||
|
|
||
| Instead of sending large model files to docker builder context, KAITO uses ORAS push to add model files to OCI layout assembly. This achieves the same result as Docker build but is much more efficient. |
There was a problem hiding this comment.
Suggested change
| Instead of sending large model files to docker builder context, KAITO uses ORAS push to add model files to OCI layout assembly. This achieves the same result as Docker build but is much more efficient. | |
| Instead of sending large model files to docker builder context, KAITO uses [ORAS](https://github.com/oras-project/oras) to add model weights and configuration files to OCI layout assembly. This achieves the same result as `docker build` but is much more efficient. |
|
|
||
| The containerized solution is split into two parts: | ||
|
|
||
| - **Base Image**: Contains the runtime and dependencies |
There was a problem hiding this comment.
Suggested change
| - **Base Image**: Contains the runtime and dependencies | |
| - **Base Image**: Contains the inference runtime and dependencies |
| The containerized solution is split into two parts: | ||
|
|
||
| - **Base Image**: Contains the runtime and dependencies | ||
| - **OCI Artifacts**: Contains the model files |
There was a problem hiding this comment.
Suggested change
| - **OCI Artifacts**: Contains the model files | |
| - **OCI Artifacts**: Contains the model weights and configuration files |
|
|
||
| ### Model Files Download Process | ||
|
|
||
| The system uses an initContainer to download model files as OCI artifacts using ORAS: |
There was a problem hiding this comment.
Suggested change
| The system uses an initContainer to download model files as OCI artifacts using ORAS: | |
| The system uses an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) to download model files as OCI artifacts using ORAS: |
There was a problem hiding this comment.
Also mentions that for super large model, we allow direct download from huggingface instead of packaging them as OCI artifacts
|
@copilot could you address the comments |
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #1356 +/- ##
=======================================
Coverage 56.60% 56.60%
=======================================
Files 75 75
Lines 8252 8252
=======================================
Hits 4671 4671
Misses 3359 3359
Partials 222 222
🚀 New features to boost your workflow:
|
TitleDocument Model-As-OCI-Artifacts feature and reorganize proposals Description
Changes walkthrough 📝
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
Copilot AI
added a commit
that referenced
this pull request
Aug 5, 2025
Co-authored-by: chewong <10557231+chewong@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the documentation reorganization and feature documentation requirements outlined in the issue:
Changes Made
1. Proposal Reorganization
website/docs/proposals/todocs/proposals/at the repository root to prevent them from being rendered as website pageswebsite/versioned_docs/version-v0.5.0/proposals/andwebsite/versioned_docs/version-v0.5.1/proposals/proposals.mdto use GitHub URLs pointing to the new location (https://github.com/kaito-project/kaito/blob/main/docs/proposals/)2. Model As OCI Artifacts Feature Documentation
Created a comprehensive new feature page at
website/docs/model-as-oci-artifacts.mdbased on the proposaldocs/proposals/20250609-model-as-oci-artifacts.mdthat includes:3. Navigation and Versioning
Screenshot
The new feature page integrates seamlessly into the documentation with comprehensive content and proper navigation:
The page includes detailed sections covering all aspects of the OCI Artifacts approach, from technical challenges to implementation benefits, with a professional layout and table of contents for easy navigation.
Fixes #1355.
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.