clustermesh: fix config watcher missed events with fsnotify 1.8.0 #35770
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The clustermesh configuration watcher needs to explicitly watch the config files to receive a notification when the underlying file gets updated, if the path points to a symbolic link (e.g., when the folder is mounted from a Kubernetes ConfigMap/Secret). However, the fsnotify library slightly changed its behavior in the latest release [1], and no longer propagates a remove event in case the parent directory is watched as well. The reason being to prevent duplicate events, as it would be generated both by the specific watcher and the parent directory watcher. Still, the parent watcher doesn't emit the remove event in this context, given that the symbolic link is not actually removed. In turn, opening the possibility for race conditions in the current event processing logic, which depends on always correctly re-establishing the watcher. Let's address this by using two separate watchers, one for the config directory itself and one for the individual config files, to ensure that the remove event is correctly emitted when the symbolic link starts pointing to a different file (hence breaking the existing watcher), so that we can re-establish it. [1]: fsnotify/fsnotify#620 Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
|
/test |
1 task
|
Ci is reasonably green (failures are unrelated flakes). Dropping the extra commit. |
c2c6c38 to
5792b46
Compare
|
/test |
1 similar comment
|
/test |
|
/test |
aanm
approved these changes
Nov 6, 2024
|
@jrajahalme Gentle ping 🙏 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The clustermesh configuration watcher needs to explicitly watch the config files to receive a notification when the underlying file gets updated, if the path points to a symbolic link (e.g., when the folder is mounted from a Kubernetes ConfigMap/Secret). However, the fsnotify library slightly changed its behavior in the latest release [1], and no longer propagates a remove event in case the parent directory is watched as well. The reason being to prevent duplicate events, as it would be generated both by the specific watcher and the parent directory watcher. Still, the parent watcher doesn't emit the remove event in this context, given that the symbolic link is not actually removed. In turn, opening the possibility for race conditions in the current event processing logic, which depends on always correctly re-establishing the watcher.
Let's address this by using two separate watchers, one for the config directory itself and one for the individual config files, to ensure that the remove event is correctly emitted when the symbolic link starts pointing to a different file (hence breaking the existing watcher), so that we can re-establish it.
[1]: fsnotify/fsnotify#620