Skip to content

fix: skip removing layer's link file when '--dry-run' option specified #4425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 5, 2024

Conversation

microyahoo
Copy link
Contributor

@microyahoo microyahoo commented Jul 31, 2024

fix: #4344 (comment)

Signed-off-by: Liang Zheng zhengliang0901@gmail.com

Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
Copy link
Member

@milosgajdos milosgajdos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL @squizzi

@milosgajdos milosgajdos requested a review from squizzi July 31, 2024 15:45
@microyahoo microyahoo changed the title fix: skip removing layer's link file when '--dry-run' option spcified fix: skip removing layer's link file when '--dry-run' option specified Jul 31, 2024
@@ -169,6 +169,10 @@ func MarkAndSweep(ctx context.Context, storageDriver driver.StorageDriver, regis

for repo, dgsts := range deleteLayerSet {
for _, dgst := range dgsts {
emit("%s: layer link eligible for deletion: %s", repo, dgst)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a test we could add that would catch this in the future so we don't have someone encountering this when testing an actual release?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's rather "complicated" because the GC code has been largely unmaintained. At some point, I suggested we move it to an internal package until there is some plan to address a lot of the issues the users report. Fixing it up would be a serious effort.

Copy link
Contributor Author

@microyahoo microyahoo Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the GC be refactored or redesigned? Currently, it only supports offline GC, which I think is unacceptable in a production environment. Additionally, the lookup performance is relatively low, and for the S3 driver, it will make a large number of S3 requests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. It's a nontrivial amount of work to make things work online, especially in high volume traffic / large storage environments the current design doesn't scale.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, online GC needs to be redesigned, and it requires a significant amount of work. Is there a plan to start with some research? If there is any work in this area in the future, please include me in the relevant loop. Thanks in advance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no plan at the moment I'm afraid. This project suffers from zero funding and thus, a general lack of interest on the current maintainers' side bar few exceptions. I am not sure if anything beyond the v3 release will happen in this project.

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah
Copy link
Member

(but a test would be nice indeed, if we have tests that could be expanded for this scenario)

@milosgajdos milosgajdos merged commit ad73793 into distribution:main Aug 5, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants