I think we can even write generic functions:

ReadBufferPool(size int) *BytePool
WriteBufferPool(size int) *BytePool

which returns pools for size. So, archive/pool.go should be something like pkg/bytepool.go because I'd definitely want to use this somewhere :)

Tried this w/ docker push to a private repo w/ two images (391MB, 944MB). Slight performance improvement of 6-7% for both images.

391MB (40.8s -> 38.4s)
944MB (96.7s -> 90.2s)

Not seeing a massive difference here (1min38s vs 1min47s):
https://gist.github.com/ross-w/43d37883fb0747696451

@tianon should we remove 1.2 retrocompat in travis ?

I tested this with doing a docker load on a 1.3gb image:

2m58s -> 2m42s (average over several runs)

Tested, also tried upping the buffer size from 512 to 2048, but I haven't seen a significant difference with my test case.

I've pushed an update. I've been able to pull fedora:latest from the Hub in 12 seconds with this PR. Pushing and pulling should both be faster.

This PR is now using the external gzip to decompress archives. This is only for testing right now, but we can improve and expand this to compression as well.

Please feel free to test again and provide results.

Thanks @unclejack . That one shaved another 10s off (comparison between your last PR and the new one):
https://gist.github.com/ross-w/0ec92c98c81140c691c1

It's still only ~20s faster than the original though.

@ross-w Could you describe your environment in the gist and provide the sizes of the layers, please?

The pull/push performance depends a lot on your disk IO both on the private registry and on your docker host, on your CPU and on your network. If your server doesn't have support for AES-NI, docker push might be a bit slow during the TarSum and that's most likely what's slowing down the push in your case.

The layers are already buffered to disk before a push, so there's isn't a big overhead there which could be slowing down the upload over HTTP like there is on pull.

Depending on the sizes of your images, it might also take a while because there is a lot of data to push.

Once I'm able to take a look at your environment's description, I can figure out what other improvements can be made to improve performance even further.

@unclejack cat /proc/cpuinfo here:
https://gist.github.com/ross-w/4e31f6caf16b600061a4

"aes" flag is shown.

Network as mentioned is not a bottleneck. Our disks aren't the fastest, but as shown in the curl benchmarks given - we can curl the layers from the registry very quickly.

Push is also pretty slow, by the way.

This particular image does have a fair amount of data, as do most of our images (they're our production applications) - but as mentioned these layers can be transferred with curl extremely quickly. It's only once in docker it becomes slower.

@ross-w Downloading the layers with curl doesn't extract them on the file system, it's simply downloading them and that's what makes it so fast. We can change the code so that it buffers them to disk before extraction, but that won't make the extraction any faster.

The extraction of the layers is the part which has some overhead because a lot of syscalls have to be made to create files, directories and a lot of files need to be written. This can be improved by having faster disks. Improving this extraction code will be difficult.

If the extraction phase is the really slow part of a pull for a particular system, buffering to disk will only make the download really fast, but it won't make the extraction phase that follows afterwards much faster. If downloading the layer takes 10 seconds at 1gbps and extracting it to disk takes 40 seconds, buffering the layer to disk first won't make it that much faster.

You can do some benchmarks by using docker save, docker load, docker export and docker import. This should give you an idea of how long it'll take when layers get buffered to disk first.

Adding an option to buffer layers to disk on pull is the next step.

@unclejack would there be situations where layers are not directly needed and extracting can be performed lazily (on first access)?

For example, if a user does docker pull ubuntu, docker pulls all tags for the ubuntu repository. Chances are that not all tags are required directly. To speed up the user experience, docker could download all layers and write them to disk.

The first time a user performs a build using one of those images, docker will uncompress the layers required for that image.

This would have the additional advantage that unused layers/images would take up less space and are easier to clean up (only removing a single archive in stead of removing a whole directory-tree)

@unclejack Sorry yes, every time I tell our story I leave more and more details out. I am aware Docker does more than just download files when pulling.

However, I have modified my script to do just that. After running the script I have the full container's filesystem in the current directory, and its run is significantly faster than docker pull:
https://gist.github.com/ross-w/83a677e17ab3ed8b31ac
~15s vs ~1min30s

If it were really our disks, should I be able to extract that fast? Or am I missing out a major step?

By the way, thank you for your efforts so far towards solving this - it is much appreciated (even if I might seem grumpy in these updates)

@unclejack additional to the above comment, I've just done a test with /var/lib/docker on tmpfs and I get the same pull time - 1min33s.

It would appear our disks are not the bottleneck in our case.

@ross-w Based on what you've written, I've decided to add buffering to disk for the layers on pull. That will help speed up pull.

@unclejack thanks, let me know when you've got something we can test!

@ross-w Please take a look at and test #7704.

Here benchmark results compared to master:

benchmark             old ns/op     new ns/op     delta       
BenchmarkTarUntar     16610542      8518127       -48.72%     

benchmark             old MB/s     new MB/s     speedup     
BenchmarkTarUntar     0.02         0.05         2.50x       

benchmark             old allocs     new allocs     delta      
BenchmarkTarUntar     7487           7536           +0.65%     

benchmark             old bytes     new bytes     delta       
BenchmarkTarUntar     355268        425864        +19.87%     

Here we see +20% allocated memory. Maybe we should write more precise benchmark for separate Tar and Untar. Maybe we'll see ways to improve this.

I've updated this PR to make some of the discussed changes. I've also added detection for gzip to make sure we only attempt to shell out to gzip when it's present.

@crosbymichael @LK4D4 @tiborvass @vieux This is ready for review. Please review and let me know what you think.

LGTM

Also here is new awesome benchmark results:

benchmark             old ns/op     new ns/op     delta       
BenchmarkTarUntar     17145885      7601314       -55.67%     

benchmark             old MB/s     new MB/s     speedup     
BenchmarkTarUntar     0.02         0.05         2.50x       

benchmark             old allocs     new allocs     delta      
BenchmarkTarUntar     7537           7487           -0.66%     

benchmark             old bytes     new bytes     delta       
BenchmarkTarUntar     425687        355816        -16.41%     

@LK4D4 PTAL, I've moved the init function.

LGTM