Skip to content

Conversation

alex-matei
Copy link
Contributor

LOOP_CONFIGURE is a new ioctl that is a lot faster than the LOOP_SET_FD+LOOP_SET_STATUS64 calls.

@k8s-ci-robot
Copy link

Hi @alex-matei. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dcantah
Copy link
Member

dcantah commented Oct 10, 2023

Cool! Do you have any benchmarks for this change that are enticing?

@alex-matei
Copy link
Contributor Author

alex-matei commented Oct 11, 2023

@dcantah I ran a few benchmarks:

Before

Kubernetes - blockfile snapshotter - 28 pods started in parallel:

time="2023-10-11T09:46:27.010767269Z" level=info msg="setupLoopDev took 14.483755ms"
time="2023-10-11T09:46:27.033784630Z" level=info msg="setupLoopDev took 11.940871ms"
time="2023-10-11T09:46:27.354981328Z" level=info msg="setupLoopDev took 9.885198ms"
time="2023-10-11T09:46:27.365784435Z" level=info msg="setupLoopDev took 7.743621ms"
time="2023-10-11T09:46:27.389949744Z" level=info msg="setupLoopDev took 13.282718ms"
time="2023-10-11T09:46:27.397778928Z" level=info msg="setupLoopDev took 11.252592ms"
time="2023-10-11T09:46:27.397793627Z" level=info msg="setupLoopDev took 11.360679ms"
time="2023-10-11T09:46:27.418040444Z" level=info msg="setupLoopDev took 15.678135ms"
time="2023-10-11T09:46:27.428771692Z" level=info msg="setupLoopDev took 4.627347ms"
time="2023-10-11T09:46:27.432758467Z" level=info msg="setupLoopDev took 10.72055ms"
time="2023-10-11T09:46:27.437740971Z" level=info msg="setupLoopDev took 4.945076ms"
time="2023-10-11T09:46:27.442812458Z" level=info msg="setupLoopDev took 9.061582ms"
time="2023-10-11T09:46:27.449803206Z" level=info msg="setupLoopDev took 9.102664ms"
time="2023-10-11T09:46:27.449808011Z" level=info msg="setupLoopDev took 6.282685ms"
time="2023-10-11T09:46:27.465754342Z" level=info msg="setupLoopDev took 10.950488ms"
time="2023-10-11T09:46:27.465794992Z" level=info msg="setupLoopDev took 9.249423ms"
time="2023-10-11T09:46:27.469806875Z" level=info msg="setupLoopDev took 6.200937ms"
time="2023-10-11T09:46:27.476759654Z" level=info msg="setupLoopDev took 7.763639ms"
time="2023-10-11T09:46:27.491769872Z" level=info msg="setupLoopDev took 12.715394ms"
time="2023-10-11T09:46:27.491797957Z" level=info msg="setupLoopDev took 11.605995ms"
time="2023-10-11T09:46:27.496772331Z" level=info msg="setupLoopDev took 8.555749ms"
time="2023-10-11T09:46:27.500756890Z" level=info msg="setupLoopDev took 10.984229ms"
time="2023-10-11T09:46:27.512774014Z" level=info msg="setupLoopDev took 12.701026ms"
time="2023-10-11T09:46:27.516798014Z" level=info msg="setupLoopDev took 12.954558ms"
time="2023-10-11T09:46:27.518756322Z" level=info msg="setupLoopDev took 9.17108ms"
time="2023-10-11T09:46:27.518946624Z" level=info msg="setupLoopDev took 6.55703ms"
time="2023-10-11T09:46:27.525779359Z" level=info msg="setupLoopDev took 7.238755ms"
time="2023-10-11T09:46:27.531789033Z" level=info msg="setupLoopDev took 5.534472ms"
time="2023-10-11T09:46:27.534743476Z" level=info msg="setupLoopDev took 6.516747ms"
time="2023-10-11T09:46:27.534852579Z" level=info msg="setupLoopDev took 4.293444ms"
time="2023-10-11T09:46:27.547774899Z" level=info msg="setupLoopDev took 10.011644ms"
time="2023-10-11T09:46:27.547787068Z" level=info msg="setupLoopDev took 10.693978ms"
time="2023-10-11T09:46:27.552768657Z" level=info msg="setupLoopDev took 6.002761ms"
time="2023-10-11T09:46:27.553807532Z" level=info msg="setupLoopDev took 7.240138ms"
time="2023-10-11T09:46:27.570749416Z" level=info msg="setupLoopDev took 12.751666ms"
time="2023-10-11T09:46:27.570792770Z" level=info msg="setupLoopDev took 10.716572ms"
time="2023-10-11T09:46:27.573753654Z" level=info msg="setupLoopDev took 9.811716ms"
time="2023-10-11T09:46:27.576790833Z" level=info msg="setupLoopDev took 8.300473ms"
time="2023-10-11T09:46:27.587764488Z" level=info msg="setupLoopDev took 9.107327ms"
time="2023-10-11T09:46:27.592778248Z" level=info msg="setupLoopDev took 10.219173ms"
time="2023-10-11T09:46:27.596743632Z" level=info msg="setupLoopDev took 8.379233ms"
time="2023-10-11T09:46:27.597791319Z" level=info msg="setupLoopDev took 9.094688ms"
time="2023-10-11T09:46:27.606785947Z" level=info msg="setupLoopDev took 8.063436ms"
time="2023-10-11T09:46:27.606844419Z" level=info msg="setupLoopDev took 6.538077ms"
time="2023-10-11T09:46:27.615758978Z" level=info msg="setupLoopDev took 5.730009ms"
time="2023-10-11T09:46:27.616767754Z" level=info msg="setupLoopDev took 6.645103ms"
time="2023-10-11T09:46:27.626743188Z" level=info msg="setupLoopDev took 6.981432ms"
time="2023-10-11T09:46:27.626809802Z" level=info msg="setupLoopDev took 6.413695ms"
time="2023-10-11T09:46:27.636850482Z" level=info msg="setupLoopDev took 8.530203ms"
time="2023-10-11T09:46:27.640768452Z" level=info msg="setupLoopDev took 10.875889ms"
time="2023-10-11T09:46:27.643808016Z" level=info msg="setupLoopDev took 5.13629ms"
time="2023-10-11T09:46:27.651733519Z" level=info msg="setupLoopDev took 9.729163ms"
time="2023-10-11T09:46:27.663741566Z" level=info msg="setupLoopDev took 10.685392ms"
time="2023-10-11T09:46:27.663770391Z" level=info msg="setupLoopDev took 9.687092ms"
time="2023-10-11T09:46:27.673794894Z" level=info msg="setupLoopDev took 10.358696ms"
time="2023-10-11T09:46:27.687749543Z" level=info msg="setupLoopDev took 10.952831ms"

Ctr:

time ctr run --snapshotter blockfile --runtime io.containerd.run.runc.v2 -t --rm docker.io/library/busybox:latest hello true
INFO[0000] setupLoopDev took 16.233184ms

real    0m0.112s
user    0m0.018s
sys     0m0.010s

After

Kubernetes - blockfile snapshotter - 28 pods started in parallel:

time="2023-10-11T09:41:23.624380253Z" level=info msg="setupLoopDev took 56.765µs"
time="2023-10-11T09:41:23.641245633Z" level=info msg="setupLoopDev took 50.726µs"
time="2023-10-11T09:41:23.647818529Z" level=info msg="setupLoopDev took 53.787µs"
time="2023-10-11T09:41:23.653829910Z" level=info msg="setupLoopDev took 46.269µs"
time="2023-10-11T09:41:23.657532291Z" level=info msg="setupLoopDev took 34.803µs"
time="2023-10-11T09:41:23.666763527Z" level=info msg="setupLoopDev took 40.25µs"
time="2023-10-11T09:41:23.669420957Z" level=info msg="setupLoopDev took 40.914µs"
time="2023-10-11T09:41:23.678953346Z" level=info msg="setupLoopDev took 34.231µs"
time="2023-10-11T09:41:23.989306418Z" level=info msg="setupLoopDev took 108.188µs"
time="2023-10-11T09:41:23.992591234Z" level=info msg="setupLoopDev took 88.631µs"
time="2023-10-11T09:41:24.011625447Z" level=info msg="setupLoopDev took 64.312µs"
time="2023-10-11T09:41:24.013063031Z" level=info msg="setupLoopDev took 58.132µs"
time="2023-10-11T09:41:24.017356614Z" level=info msg="setupLoopDev took 57.474µs"
time="2023-10-11T09:41:24.025520141Z" level=info msg="setupLoopDev took 92.703µs"
time="2023-10-11T09:41:24.037555721Z" level=info msg="setupLoopDev took 45.669µs"
time="2023-10-11T09:41:24.041492968Z" level=info msg="setupLoopDev took 61.748µs"
time="2023-10-11T09:41:24.041792826Z" level=info msg="setupLoopDev took 44.754µs"
time="2023-10-11T09:41:24.050468068Z" level=info msg="setupLoopDev took 65.518µs"
time="2023-10-11T09:41:24.052625127Z" level=info msg="setupLoopDev took 56.547µs"
time="2023-10-11T09:41:24.062413990Z" level=info msg="setupLoopDev took 49.14µs"
time="2023-10-11T09:41:24.068191343Z" level=info msg="setupLoopDev took 85.174µs"
time="2023-10-11T09:41:24.072586255Z" level=info msg="setupLoopDev took 39.607µs"
time="2023-10-11T09:41:24.073893663Z" level=info msg="setupLoopDev took 53.048µs"
time="2023-10-11T09:41:24.082570242Z" level=info msg="setupLoopDev took 50.188µs"
time="2023-10-11T09:41:24.084557546Z" level=info msg="setupLoopDev took 43.807µs"
time="2023-10-11T09:41:24.090964937Z" level=info msg="setupLoopDev took 41.739µs"
time="2023-10-11T09:41:24.094441025Z" level=info msg="setupLoopDev took 64.712µs"
time="2023-10-11T09:41:24.102905027Z" level=info msg="setupLoopDev took 61.215µs"
time="2023-10-11T09:41:24.103839571Z" level=info msg="setupLoopDev took 43.506µs"
time="2023-10-11T09:41:24.112371719Z" level=info msg="setupLoopDev took 44.802µs"
time="2023-10-11T09:41:24.114499630Z" level=info msg="setupLoopDev took 43.072µs"
time="2023-10-11T09:41:24.121982579Z" level=info msg="setupLoopDev took 48.833µs"
time="2023-10-11T09:41:24.124941275Z" level=info msg="setupLoopDev took 75.522µs"
time="2023-10-11T09:41:24.134198844Z" level=info msg="setupLoopDev took 104.9µs"
time="2023-10-11T09:41:24.135181160Z" level=info msg="setupLoopDev took 46.111µs"
time="2023-10-11T09:41:24.145505713Z" level=info msg="setupLoopDev took 70.983µs"
time="2023-10-11T09:41:24.148041192Z" level=info msg="setupLoopDev took 56.234µs"
time="2023-10-11T09:41:24.155549921Z" level=info msg="setupLoopDev took 66.976µs"
time="2023-10-11T09:41:24.159403855Z" level=info msg="setupLoopDev took 88.401µs"
time="2023-10-11T09:41:24.166136868Z" level=info msg="setupLoopDev took 62.046µs"
time="2023-10-11T09:41:24.171149337Z" level=info msg="setupLoopDev took 88.43µs"
time="2023-10-11T09:41:24.176550231Z" level=info msg="setupLoopDev took 81.036µs"
time="2023-10-11T09:41:24.181958554Z" level=info msg="setupLoopDev took 37.695µs"
time="2023-10-11T09:41:24.189107493Z" level=info msg="setupLoopDev took 107.347µs"
time="2023-10-11T09:41:24.189265355Z" level=info msg="setupLoopDev took 35.46µs"
time="2023-10-11T09:41:24.199063569Z" level=info msg="setupLoopDev took 95.02µs"
time="2023-10-11T09:41:24.201373260Z" level=info msg="setupLoopDev took 66.71µs"
time="2023-10-11T09:41:24.209287589Z" level=info msg="setupLoopDev took 65.708µs"
time="2023-10-11T09:41:24.211611203Z" level=info msg="setupLoopDev took 44.458µs"
time="2023-10-11T09:41:24.219393158Z" level=info msg="setupLoopDev took 81.391µs"
time="2023-10-11T09:41:24.224494579Z" level=info msg="setupLoopDev took 62.539µs"
time="2023-10-11T09:41:24.229603888Z" level=info msg="setupLoopDev took 61.68µs"
time="2023-10-11T09:41:24.236014011Z" level=info msg="setupLoopDev took 59.908µs"
time="2023-10-11T09:41:24.239516967Z" level=info msg="setupLoopDev took 45.7µs"
time="2023-10-11T09:41:24.242335483Z" level=info msg="setupLoopDev took 50.748µs"
time="2023-10-11T09:41:24.253009175Z" level=info msg="setupLoopDev took 76.108µs"

Ctr:

time ctr run --snapshotter blockfile --runtime io.containerd.run.runc.v2 -t --rm docker.io/library/busybox:latest hello true
INFO[0000] setupLoopDev took 59.892µs

real    0m0.105s
user    0m0.018s
sys     0m0.013s

@alex-matei alex-matei force-pushed the use-loop-configure branch 4 times, most recently from c4138f8 to 888ceba Compare October 11, 2023 10:04
Comment on lines 49 to 50
func ioctlConfigure(fd int, value *LoopConfig) error {
_, _, err := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), uintptr(LOOP_CONFIGURE), uintptr(unsafe.Pointer(value)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something that could eventually be upstreamed to golang.org/x/sys?

Copy link
Contributor Author

@alex-matei alex-matei Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, It might be useful to add it in there as well. I'll check what is the submission flow for it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know maintainers there have been quite receptive to such changes, but much of the code is generated, so it may indeed require a small amount of looking at existing examples to see how to create the patch. @tklauser (👋 sorry for the random ping ❤️) may be able to provide some guidance if needed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it looks pretty straightforward, I made the changes in code. I need to get the CLA signed by legal department and after I'll open a pull request.

Copy link
Member

@fuweid fuweid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

In my local, it's really good.

goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz

➜  mount git:(review-9211) sudo $(command -v go) test -test.root -run TestAttachDetachLoopDevice -count=1000 -failfast  ./...
PASS
ok      github.com/containerd/containerd/mount  19.471s

➜  mount git:(main) sudo $(command -v go) test -test.root -run TestAttachDetachLoopDevice -count=1000 -failfast  ./...
PASS
ok      github.com/containerd/containerd/mount  46.858s

@@ -32,8 +35,26 @@ const (
loopDevFormat = "/dev/loop%d"

ebusyString = "device or resource busy"

LOOP_CONFIGURE = 0x4c0a //nolint: revive
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This const doesn't need to be exported and the convention would be to use mixed caps.

Suggested change
LOOP_CONFIGURE = 0x4c0a //nolint: revive
loopConfigureIoctl = 0x4c0a //nolint: revive

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

LOOP_CONFIGURE is a new ioctl that is a lot faster than
the LOOP_SET_FD+LOOP_SET_STATUS64 calls

Signed-off-by: Alexandru Matei <alexandru.matei@uipath.com>
Copy link
Member

@fuweid fuweid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fuweid fuweid requested a review from samuelkarp October 17, 2023 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants