docker pull slow #1888
Description
I've noticed that docker pull is slow when downloading images from a private registry. But, I don't think it's the registry because downloading the images using wget is super fast. For example, the docker pull command below [1] takes over 6 minutes to complete. In comparison, grabbing the same images using wget completes in ~5 seconds [2]. I also noticed that docker is running the images through tar, so I decompressed the layers as well, which completes in ~30 seconds [3].
All in all, it appears that downloading and decompressing the tarballs this example should take less than a minute. Giving Docker some additional administrative overhead, it seems that the entire operation should complete in about a minute. Compared to nearly 7 minutes, that's a huge discrepancy.
Finally, while docker pull is running, top shows that docker is pegging the CPU at 100%.
Docker on both ends is Docker version 0.6.1, build 5105263 and the registry is docker-archive/docker-registry@f8723f8
Please let me know if I can do additional debugging on my end.
Thanks!
[1] docker pull example:
baremetal@dal06:~$ time docker pull registry.baremetal.io/baremetal/docker_registry
Pulling repository registry.baremetal.io/baremetal/docker_registry
fe737d3200fc: Download complete
f7f8b2bc787f: Download complete
e0f7c47bd69a: Download complete
e3f689b3e2ea: Download complete
8dbd9e392a96: Download complete
66f24c49bccb: Download complete
afe0c306cb73: Download complete
b8327e2d0a09: Download complete
64492b97a619: Download complete
2013/09/15 20:40:06 Internal server error: 404 trying to fetch remote history for baremetal/docker_registry
real 6m43.049s
user 0m0.044s
sys 0m0.044s[2] Rough download estimate using wget:
baremetal@dal06:~$ time (cat images | while read i; do echo $i; time wget https://registry.baremetal.io/v1/images/$i/layer; done;)
64492b97a6190f17d35106d4cde1c4e55261f0558ef15bb6bb6b82edc502cb26
--2013-09-15 20:30:53-- https://registry.baremetal.io/v1/images/64492b97a6190f17d35106d4cde1c4e55261f0558ef15bb6bb6b82edc502cb26/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 285279 (279K) [text/html]
Saving to: 'layer.9'
100%[========================================================================================================================================>] 285,279 --.-K/s in 0.02s
2013-09-15 20:30:54 (11.9 MB/s) - 'layer.9' saved [285279/285279]
real 0m0.408s
user 0m0.000s
sys 0m0.004s
66f24c49bccb54f282d95d2d1fb32170280c5431fca9c35f22c8b7e5ea412097
--2013-09-15 20:30:54-- https://registry.baremetal.io/v1/images/66f24c49bccb54f282d95d2d1fb32170280c5431fca9c35f22c8b7e5ea412097/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 178 [text/html]
Saving to: 'layer.10'
100%[========================================================================================================================================>] 178 --.-K/s in 0s
2013-09-15 20:30:54 (189 MB/s) - 'layer.10' saved [178/178]
real 0m0.026s
user 0m0.000s
sys 0m0.000s
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c
--2013-09-15 20:30:54-- https://registry.baremetal.io/v1/images/8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 71474905 (68M) [text/html]
Saving to: 'layer.11'
100%[========================================================================================================================================>] 71,474,905 111MB/s in 0.6s
2013-09-15 20:30:54 (111 MB/s) - 'layer.11' saved [71474905/71474905]
real 0m0.650s
user 0m0.156s
sys 0m0.128s
afe0c306cb73c6fd8c47d6c8d7aedc935641cfb01af37ec646d44debdbaa4adb
--2013-09-15 20:30:54-- https://registry.baremetal.io/v1/images/afe0c306cb73c6fd8c47d6c8d7aedc935641cfb01af37ec646d44debdbaa4adb/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45244332 (43M) [text/html]
Saving to: 'layer.12'
100%[========================================================================================================================================>] 45,244,332 40.0MB/s in 1.1s
2013-09-15 20:30:55 (40.0 MB/s) - 'layer.12' saved [45244332/45244332]
real 0m1.147s
user 0m0.108s
sys 0m0.056s
b8327e2d0a09b802cab5bde8624007d766d9d4b2a5f80f3cc6c9b1e1f2e759da
--2013-09-15 20:30:55-- https://registry.baremetal.io/v1/images/b8327e2d0a09b802cab5bde8624007d766d9d4b2a5f80f3cc6c9b1e1f2e759da/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 935 [text/html]
Saving to: 'layer.13'
100%[========================================================================================================================================>] 935 --.-K/s in 0s
2013-09-15 20:30:55 (903 MB/s) - 'layer.13' saved [935/935]
real 0m0.036s
user 0m0.004s
sys 0m0.000s
e0f7c47bd69aaa0850c35b6204df077cf9ef0fae3e08ce3c44276f4a1af7760c
--2013-09-15 20:30:55-- https://registry.baremetal.io/v1/images/e0f7c47bd69aaa0850c35b6204df077cf9ef0fae3e08ce3c44276f4a1af7760c/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 197135599 (188M) [text/html]
Saving to: 'layer.14'
100%[========================================================================================================================================>] 197,135,599 111MB/s in 1.7s
2013-09-15 20:30:57 (111 MB/s) - 'layer.14' saved [197135599/197135599]
real 0m1.769s
user 0m0.432s
sys 0m0.276s
e3f689b3e2eafc3a86631b1b515a066bc40b13886b8cab5a938f8eeac5069931
--2013-09-15 20:30:57-- https://registry.baremetal.io/v1/images/e3f689b3e2eafc3a86631b1b515a066bc40b13886b8cab5a938f8eeac5069931/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5666798 (5.4M) [text/html]
Saving to: 'layer.15'
100%[========================================================================================================================================>] 5,666,798 --.-K/s in 0.05s
2013-09-15 20:30:57 (103 MB/s) - 'layer.15' saved [5666798/5666798]
real 0m0.115s
user 0m0.012s
sys 0m0.012s
f7f8b2bc787f9232ef8c75a1fc2e3304c1b7f92906ee5a5c80c889f50c8e7cc3
--2013-09-15 20:30:57-- https://registry.baremetal.io/v1/images/f7f8b2bc787f9232ef8c75a1fc2e3304c1b7f92906ee5a5c80c889f50c8e7cc3/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 244 [text/html]
Saving to: 'layer.16'
100%[========================================================================================================================================>] 244 --.-K/s in 0s
2013-09-15 20:30:57 (233 MB/s) - 'layer.16' saved [244/244]
real 0m0.048s
user 0m0.000s
sys 0m0.000s
fe737d3200fcd0d5a08deb023b9ff29354d371ce21d79c44f93292fdf0901245
--2013-09-15 20:30:57-- https://registry.baremetal.io/v1/images/fe737d3200fcd0d5a08deb023b9ff29354d371ce21d79c44f93292fdf0901245/layer
Resolving registry.baremetal.io (registry.baremetal.io)... 10.41.142.194
Connecting to registry.baremetal.io (registry.baremetal.io)|10.41.142.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 176 [text/html]
Saving to: 'layer.17'
100%[========================================================================================================================================>] 176 --.-K/s in 0s
2013-09-15 20:30:57 (202 MB/s) - 'layer.17' saved [176/176]
real 0m0.015s
user 0m0.000s
sys 0m0.000s
real 0m4.217s
user 0m0.712s
sys 0m0.476s
baremetal@dal06:~$[3] Rough estimate of tar (layers are not applied in order):
baremetal@dal06:~/foo$ time (for i in $(ls ../layer* | sort -g); do echo $i; time sudo tar xzf $i; done;)
../layer
real 0m0.012s
user 0m0.004s
sys 0m0.004s
../layer.1
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.10
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.11
real 0m1.845s
user 0m0.984s
sys 0m0.356s
../layer.12
real 0m1.000s
user 0m0.956s
sys 0m0.188s
../layer.13
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.14
real 0m3.636s
user 0m2.360s
sys 0m0.672s
../layer.15
real 0m0.174s
user 0m0.168s
sys 0m0.076s
../layer.16
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.17
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.18
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.19
real 0m1.032s
user 0m1.000s
sys 0m0.348s
../layer.2
real 0m2.069s
user 0m1.004s
sys 0m0.328s
../layer.20
real 0m1.005s
user 0m0.960s
sys 0m0.184s
../layer.21
real 0m0.424s
user 0m0.000s
sys 0m0.000s
../layer.22
real 0m2.413s
user 0m2.324s
sys 0m0.688s
../layer.23
real 0m0.179s
user 0m0.172s
sys 0m0.072s
../layer.24
real 0m0.004s
user 0m0.004s
sys 0m0.000s
../layer.25
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.3
real 0m1.262s
user 0m0.984s
sys 0m0.180s
../layer.4
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.5
real 0m3.941s
user 0m2.396s
sys 0m0.632s
../layer.6
real 0m0.198s
user 0m0.176s
sys 0m0.068s
../layer.7
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.8
real 0m0.004s
user 0m0.000s
sys 0m0.000s
../layer.9
real 0m0.012s
user 0m0.004s
sys 0m0.004s
real 0m19.264s
user 0m13.496s
sys 0m3.800s
baremetal@dal06:~/foo$