Skip to content

Conversation

jsternberg
Copy link
Collaborator

@jsternberg jsternberg commented Sep 26, 2023

This updates #4270 to add an integration test and also merge some of the
logic for how the selectors are created. Now, toSelectors will perform
the root path detection instead of some custom logic in getMountDeps.

dedupePaths has also been updated to check if the number of paths is 1
or less so it can avoid an allocation when the function is a no-op.

@@ -468,3 +470,54 @@ COPY --from=base /tmpfssize /
require.NoError(t, err)
require.Contains(t, string(dt), `size=131072k`)
}

func testMountDuplicate(t *testing.T, sb integration.Sandbox) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave a comment // moby/buildkit#4123 here so it is clear that this is a regression test.

@jsternberg jsternberg changed the title solver: add integration test for multiple bind mount issue solver: use toSelectors to filter root paths instead of custom logic Sep 27, 2023
@jsternberg
Copy link
Collaborator Author

I was looking at adding a better unit test for NewExecOp that didn't test the private method and ended up finding a way to simplify the logic so the filtering is only handled in one place. I think this PR is probably better and easier to maintain than the original.

I've removed the unit test as it no longer serves a purpose and the integration test now exists.

@jsternberg jsternberg force-pushed the duplicate-mount-integration-test branch 2 times, most recently from 13caaaf to ff1afcc Compare September 27, 2023 20:54
Comment on lines +212 to +229
func forceTrailingSlash(s string) string {
if strings.HasSuffix(s, "/") {
return s
}
return s + "/"
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to handle the situation where / is the prefix path. Unfortunately, this doesn't work if dedupePaths is called with []string{"", "foo"}.

@jsternberg jsternberg marked this pull request as draft September 27, 2023 21:39
@jsternberg
Copy link
Collaborator Author

I want to also fix up dedupePaths as part of this and need a bit more time to figure out how that affects other areas of the system so I'm converting this to a draft for now.

@jsternberg jsternberg force-pushed the duplicate-mount-integration-test branch from ff1afcc to a22902a Compare November 15, 2023 19:27
@jsternberg jsternberg marked this pull request as ready for review November 15, 2023 20:35
Copy link
Member

@tonistiigi tonistiigi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the digest computation:

cm.Deps[i].Selector = digest.FromBytes(bytes.Join(dgsts, []byte{0}))

Also does not do dedupe (correctly) although I think it should.

This can be follow-up as not directly the same issue.

@jsternberg
Copy link
Collaborator Author

It looks like a potential correctness issue. Since the dedupe happens in getMountDeps presently and it now happens only for the content hash, I think it changes the behavior.

Should be an easy fix though.

@tonistiigi
Copy link
Member

Also does not do dedupe (correctly) although I think it should.

I think I was wrong in here. Selectors digest should not be deduplicated. Eg. if you have two mounts and one is inside another, if you switch their order, that should change the definition-based cache key because the result of the op changes.

@jsternberg
Copy link
Collaborator Author

I think I was wrong in here. Selectors digest should not be deduplicated. Eg. if you have two mounts and one is inside another, if you switch their order, that should change the definition-based cache key because the result of the op changes.

Ok I'll revert that and keep the selectors digest as not being deduplicated at all.

@jsternberg jsternberg force-pushed the duplicate-mount-integration-test branch from 70e30cd to 5cec511 Compare December 4, 2023 16:18
@jsternberg
Copy link
Collaborator Author

Ok I've reverted the behavior. The main change is that the definition based key will most likely include the root binding in the definition while it previously omitted it. This is just due to the check being done after the selectors are added rather than before.

I don't think that should end up mattering though.

This updates moby#4270 to add an integration test and also merge some of the
logic for how the selectors are created. Now, `toSelectors` will perform
the root path detection instead of some custom logic in `getMountDeps`.

`dedupePaths` has also been updated to check if the number of paths is 1
or less so it can avoid an allocation when the function is a no-op.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
@jsternberg jsternberg force-pushed the duplicate-mount-integration-test branch from 5cec511 to 091fb2c Compare December 4, 2023 20:02
@tonistiigi tonistiigi merged commit 6997850 into moby:master Dec 6, 2023
@jsternberg jsternberg deleted the duplicate-mount-integration-test branch December 6, 2023 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants