runtime-rs: Propagate k8s configs correctly when sharedfs is disabled #11240

Apokleos · 2025-05-07T08:39:56Z

runtime-rs: Propagate k8s configs correctly when sharedfs is disabled

In Kubernetes (k8s), while Kata Pods often use virtiofs for injecting
Service Accounts, Secrets, and ConfigMaps, security-sensitive
environments like CoCo disable host-guest sharing. Consequently, when
SharedFs is disabled, we propagate these configurations into the guest
via file copy and bind mount for correct container access.

Fixes #11237

Signed-off-by: alex.lyn alex.lyn@antgroup.com

stevenhorsman · 2025-05-07T10:41:17Z

Hi @Apokleos - thanks for this update. Just to check I understand the scope of this

this PR covers doing the initial recursive copy of directories when shared_fs = none
It doesn't set up a watcher to continue to propogate those changes to the guest if they change on the host side?
Is it filtering to only do this copying for Kubernetes directories, like https://github.com/stevenhorsman/kata-containers/blob/021e40781607aae0e5c3d64a32d10fb9863a984f/src/runtime/virtcontainers/fs_share_linux.go#L96 does with the regex matchers: https://github.com/stevenhorsman/kata-containers/blob/021e40781607aae0e5c3d64a32d10fb9863a984f/src/runtime/virtcontainers/fs_share_linux.go#L43-L55?

I'm just trying to confirm how much of the gap this closes, if the watcher is a separate PR, then that's not a problem for me. Thanks!

Apokleos · 2025-05-07T11:35:33Z

Hi @Apokleos - thanks for this update. Just to check I understand the scope of this

this PR covers doing the initial recursive copy of directories when shared_fs = none

yes

It doesn't set up a watcher to continue to propogate those changes to the guest if they change on the host side?

Yes, you're right. And I'm also thinking this point to do watch on the files/dir changes, as you have pointed it out, I believe I have to make it work with one more commit which belongs to this PR.

Is it filtering to only do this copying for Kubernetes directories, like https://github.com/stevenhorsman/kata-containers/blob/021e40781607aae0e5c3d64a32d10fb9863a984f/src/runtime/virtcontainers/fs_share_linux.go#L96 does with the regex matchers: https://github.com/stevenhorsman/kata-containers/blob/021e40781607aae0e5c3d64a32d10fb9863a984f/src/runtime/virtcontainers/fs_share_linux.go#L43-L55?

I think I'm working on the same functions as the code you list. More comments will be added to make it more clear.

I'm just trying to confirm how much of the gap this closes, if the watcher is a separate PR, then that's not a problem for me. Thanks!

Appreciated Steven @stevenhorsman I tend to introduce one more commits to make it closer to runtime-go.

stevenhorsman · 2025-05-07T11:49:39Z

Appreciated Steven @stevenhorsman I tend to introduce one more commits to make it closer to runtime-go.

Great - thanks for the clarification. I think this PR is still good, so am happy to see it go in, I just wanted to check I'd understand the limits/scope of it.

Apokleos · 2025-05-12T02:32:56Z

Appreciated Steven @stevenhorsman I tend to introduce one more commits to make it closer to runtime-go.

Great - thanks for the clarification. I think this PR is still good, so am happy to see it go in, I just wanted to check I'd understand the limits/scope of it.

Hi @stevenhorsman PR updated, PTAL! Thx.

stevenhorsman · 2025-05-12T15:54:33Z

src/runtime-rs/crates/resource/src/volume/share_fs_volume.rs

+                // multiple times in a short period; we only execute the last one.
+                if let Some(t) = last_event_time {
+                    if Instant::now().duration_since(t) > DEBOUNCE_TIME && *need_sync.lock().await {
+                        info!(sl!(), "debonce handle copyfile {:?} -> {:?}", &src, &dst);


nit: debonce -> debounce

stevenhorsman · 2025-05-12T15:54:45Z

src/runtime-rs/crates/resource/src/volume/share_fs_volume.rs

+                        {
+                            error!(
+                                sl!(),
+                                "debonce handle copyfile {:?} -> {:?} failed with error: {:?}",


stevenhorsman · 2025-05-12T15:57:22Z

Hi @stevenhorsman PR updated, PTAL! Thx.

Thanks for the update. Is it just the identification of the Kubernetes files we want to propogate left now for this feature?

Apokleos · 2025-05-15T09:46:32Z

Hi @stevenhorsman PR updated, PTAL! Thx.

Thanks for the update. Is it just the identification of the Kubernetes files we want to propogate left now for this feature?

Appreciate @stevenhorsman I have updated it with volume copy whitelist check added to restrict directory copying limited in /var/lib/kubelet/pods/<uid>/volumes/{kubernetes.io~configmap, kubernetes.io~secret, kubernetes.io~downward-api, kubernetes.io~projected}

stevenhorsman · 2025-05-15T10:57:07Z

src/runtime-rs/crates/resource/src/volume/share_fs_volume.rs

+/// This function is used to check whether a given volume is in the allowed copy whitelist.
+/// More specifically, it determines whether the volume's path is located under a predefined
+/// list of allowed copy directories.
+pub(crate) fn is_whitelisted_copy_volume(source_path: &PathBuf) -> bool {


To try and comply with the Inclusive Naming Initiative which the CNCF created, could we rename whitelist to allowlist here? https://inclusivenaming.org/word-lists/tier-1/whitelist/

Cool! definitly agree

Fixed, PTAL

stevenhorsman

The code looks good to me, I don't think we have a way to test that here, but that's true for the go-runtime as well (until we implement #8015 that I've just remembered), so I don't think that should be a blocker. Thanks for the changes @Apokleos

Add async directory traversal using BFS algorithm: (1) Support file type handling: Regular files (S_IFREG) with content streaming; Directories (S_IFDIR) with mode preservation; Symbolic links (S_IFLNK) with target recreation; (2) Maintain POSIX metadata: UID/GID preservation,File mode bits, and Directory permissions (3) Implement async I/O operations for: Directory enumeration, file reading, symlink target resolution Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

In Kubernetes (k8s), while Kata Pods often use virtiofs for injecting Service Accounts, Secrets, and ConfigMaps, security-sensitive environments like CoCo disable host-guest sharing. Consequently, when SharedFs is disabled, we propagate these configurations into the guest via file copy and bind mount for correct container access. Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Introduce event-driven file sync mechanism between host and guest when sharedfs is disabled, which will help monitor the host path in time and do sync files changes: 1. Introduce FsWatcher to monitor directory changes via inotify; 2. Support recursive watching with configurable filters; 3. Add debounce logic (default 500ms cooldown) to handle burst events; 4. Trigger `copy_dir_recursively` on stable state; 5. Handle CREATE/MODIFY/DELETE/MOVED/CLOSE_WRITE events; Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

When synchronizing file changes on the host, a "symlink AlreadyExists" issue occurs, primarily due to improper handling of symbolic links (symlinks). Additionally, there are other related problems. This patch will try to address these problems. (1) Handle symlink target existence (files, dirs, symlinks) during host file sync. Use appropriate removal methods (unlink, remove_file, remove_dir_all). (2) Enhance temporary file handling for safer operations and implement truncate only at offset 0 for resume support. (3) Set permissions and ownership for parent directories. (4) Check and clean target path for regular files before rename. Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

For security reasons, we have restricted directory copying. Introduces the `is_allowlisted_copy_volume` function to verify if a given volume path is present in an allowed copy directory. This enhances security by ensuring only permitted volumes are copied Currently, only directories under the path `/var/lib/kubelet/pods/<uid>/volumes/{kubernetes.io~configmap, kubernetes.io~secret, kubernetes.io~downward-api, kubernetes.io~projected}` are allowed to be copied into the guest. Copying of other directories will be prohibited. Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

fidencio

let's get this one in.
thanks @Apokleos!

Apokleos self-assigned this May 7, 2025

Apokleos added the ok-to-test label May 7, 2025

Apokleos requested a review from justxuewei May 7, 2025 08:40

katacontainersbot added the size/large Task of significant size label May 7, 2025

Apokleos requested review from lifupan, stevenhorsman, fidencio and bpradipt May 7, 2025 08:40

Apokleos changed the title ~~Copydir~~ runtime-rs: Propagate k8s configs correctly when sharedfs is disabled May 7, 2025

katacontainersbot added size/huge Largest and most complex task (probably needs breaking into small pieces) and removed size/large Task of significant size labels May 9, 2025

stevenhorsman reviewed May 12, 2025

View reviewed changes

Apokleos force-pushed the copydir branch 2 times, most recently from 92a1015 to 371dc34 Compare May 15, 2025 09:43

stevenhorsman reviewed May 15, 2025

View reviewed changes

Apokleos requested a review from stevenhorsman May 15, 2025 11:10

Apokleos force-pushed the copydir branch from 371dc34 to f53458c Compare May 15, 2025 12:17

stevenhorsman approved these changes May 15, 2025

View reviewed changes

Apokleos added 6 commits May 20, 2025 16:55

kata-types: Introduce k8s special volumes for projected and downward-api

8910bdd

Fixes kata-containers#11237 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Apokleos force-pushed the copydir branch from f53458c to 4b27ca9 Compare May 20, 2025 08:57

Apokleos mentioned this pull request May 26, 2025

A comparison of feature differences between Kata 3.0/runtime-rs and Kata 2.x and alignment status #8702

Open

fidencio approved these changes May 27, 2025

View reviewed changes

fidencio merged commit d3f81ec into kata-containers:main May 27, 2025
331 of 351 checks passed

Apokleos mentioned this pull request May 30, 2025

runtime-rs: Support shared_fs = "none" for CoCo #10697

Merged

Apokleos mentioned this pull request Jun 9, 2025

Kata Runtime-rs with CoCo Support confidential-containers/confidential-containers#308

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

runtime-rs: Propagate k8s configs correctly when sharedfs is disabled #11240

runtime-rs: Propagate k8s configs correctly when sharedfs is disabled #11240

Uh oh!

Apokleos commented May 7, 2025

Uh oh!

stevenhorsman commented May 7, 2025

Uh oh!

Apokleos commented May 7, 2025

Uh oh!

stevenhorsman commented May 7, 2025

Uh oh!

Apokleos commented May 12, 2025

Uh oh!

stevenhorsman May 12, 2025

Uh oh!

stevenhorsman May 12, 2025

Uh oh!

stevenhorsman commented May 12, 2025

Uh oh!

Apokleos commented May 15, 2025

Uh oh!

stevenhorsman May 15, 2025

Uh oh!

Apokleos May 15, 2025

Uh oh!

Apokleos May 15, 2025

Uh oh!

stevenhorsman left a comment

Uh oh!

fidencio left a comment

Uh oh!

Uh oh!

Uh oh!

runtime-rs: Propagate k8s configs correctly when sharedfs is disabled #11240

runtime-rs: Propagate k8s configs correctly when sharedfs is disabled #11240

Uh oh!

Conversation

Apokleos commented May 7, 2025

Uh oh!

stevenhorsman commented May 7, 2025

Uh oh!

Apokleos commented May 7, 2025

Uh oh!

stevenhorsman commented May 7, 2025

Uh oh!

Apokleos commented May 12, 2025

Uh oh!

stevenhorsman May 12, 2025

Choose a reason for hiding this comment

Uh oh!

stevenhorsman May 12, 2025

Choose a reason for hiding this comment

Uh oh!

stevenhorsman commented May 12, 2025

Uh oh!

Apokleos commented May 15, 2025

Uh oh!

stevenhorsman May 15, 2025

Choose a reason for hiding this comment

Uh oh!

Apokleos May 15, 2025

Choose a reason for hiding this comment

Uh oh!

Apokleos May 15, 2025

Choose a reason for hiding this comment

Uh oh!

stevenhorsman left a comment

Choose a reason for hiding this comment

Uh oh!

fidencio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!