-
Notifications
You must be signed in to change notification settings - Fork 18.8k
Update image specification for content-addressability changes #22264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
OHMAN. Lots of good work here, @aaronlehmann. |
``` | ||
|
||
There are one or more directories named with the ChainID for each layer in a | ||
full image. Each of these directories contains 3 files: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For docker-v1.10+
the directory layout isn't important anymore as all the paths are listed in manifest. We keep this structure because in tars from docker are compatible with both v1.0
and v1.1
specs. When we remove support for v1.0
we will probably also simplify this directory structure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tonistiigi as an aside, when do you think that will be? That is an important integration point for many.
Or rather, if we can evolve this into a common standard (read as OpenContainers) then would it need a --format=oci.v1
like integration?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know. There is no problem with supporting both atm but we don't want to promote the old one because we can't have ID stability with the old version. id/json
is the 1.0
image configuration with the parent chain and we don't use this method any more since v1.10
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
understood
On Wed, Apr 27, 2016 at 2:26 PM Tõnis Tiigi notifications@github.com
wrote:
In image/spec/v1.1.md
#22264 (comment):+.
+├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json
+├── 5f29f704785248ddb9d06b90a11b5ea36c534865e9035e4022bb2e71d4ecbb9a
+│ ├── VERSION
+│ ├── json
+│ └── layer.tar
+├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198
+│ ├── VERSION
+│ ├── json
+│ └── layer.tar
+├── manifest.json
+└── repositories
+```
+
+There are one or more directories named with the ChainID for each layer in a
+full image. Each of these directories contains 3 files:I don't know. There is no problem with supporting both atm but we don't
want to promote the old one because we can't have ID stability with the old
version. id/json is the 1.0 image configuration with the parent chain and
we don't use this method any more since v1.10.—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/22264/files/2801c4c22c04396ec912760a955752cd8955e841#r61310686
Overall LGTM. What are the next steps to getting it merged? |
1.0 | ||
``` | ||
|
||
The `repositories` file is another JSON file which describes names/tags: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should mention this file is also only used for backwards compatibility. Also at the end when discussing what "older implementations" will use.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a mention below.
22799c5
to
a07dd24
Compare
LGTM |
from moby/moby#22264 specifically commit 2801c4c22c04396ec912760a955752cd8955e841 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198 | ||
│ ├── VERSION | ||
│ ├── json | ||
│ └── layer.tar |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any comment on the consistency of the mtime of these objects? That was a common important ask from folks in the community for consistency of docker save
output (regardless that the checksum of the objects themselves being consistent)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we currently make any efforts to make mtimes consistent, or do anything else to support stable hashes of the save artifacts we generate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a decent conversation around this and I had written the code for
it. So it was a thing. The json and layer.tar were given an mtime of the
layer creation time, and repositories was Unix epoch of 0
It is nicely consistent. :-)
On Thu, May 12, 2016, 12:52 Aaron Lehmann notifications@github.com wrote:
In image/spec/v1.1.md
#22264 (comment):
- - all tar archives of each layer filesystem changesets
+
+For example, here's what the full archive oflibrary/busybox
is (displayed in
+tree
format):
+
+```
+.
+├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json
+├── 5f29f704785248ddb9d06b90a11b5ea36c534865e9035e4022bb2e71d4ecbb9a
+│ ├── VERSION
+│ ├── json
+│ └── layer.tar
+├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198
+│ ├── VERSION
+│ ├── json
+│ └── layer.tarI don't think we currently make any efforts to make mtimes consistent, or
do anything else to support stable hashes of the save artifacts we generate.—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/22264/files/a07dd240ac7148093987380edc10222e63ba19e2#r63055557
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vbatts @aaronlehmann Yes, we still do the modified time reset like @vbatts described.
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
</dt> | ||
<dd> | ||
A set of directories which should be created as data volumes in | ||
a container running this image. This JSON structure value is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this sentence get added opencontainers/image-spec@949ea01
It applies to the docker behaviour as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That isn't quite correct. Volume is replaced but there is a copy operation that copies files from image to the volume. So the data from the image rootfs is still present.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh no. This sounds like a "clever" use-case, but would allow for sneaky bugs. :-\
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tonistiigi just checking on this, and i'm not seeing that to be the case. Perhaps you can provide an example of what you're seeing with host files of a volume path being copied up?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cat <<EOT > Dockerfile
from busybox
run mkdir /foo && touch /foo/bar
volume /foo
EOT
docker build -t foo .
ls -l $(docker inspect -f '{{(index .Mounts 0).Source}}' $(docker run -d foo))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tonistiigi fair, but docker run -it -v /usr:/foo:ro foo ls /foo/
does not reveal /foo/bar
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but docker run -it -v /usr:/foo:ro foo ls /foo/
doesn't have much to do with Volumes
struct in the image configuration. Bind mounts just override everything and this behavior is independent of the image config.
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
|
||
``` | ||
. | ||
├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this file is not explicitly referenced below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is referenced in the example manifest.json
below.
How do people feel about this PR? Are there any other changes we should make before merging? |
I can do a review, once it's technically "approved", so happy to hear |
└── repositories | ||
``` | ||
|
||
There are one or more directories named with the ChainID for each layer in a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The directory name is not the ChainID
. It is just a deterministic random hex. I think ChainID
should just be an implementation detail, we probably don't need to define it at all here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tonistiigi: Fixed.
LGTM |
The image spec in image/spec/v1.md is quite a bit out of date. Not only is it missing the changes that went into 1.10 for content addressability, but it has inaccuracies that date back further, such as mentioning storing tarsum in the image configuration. This commit creates image/spec/v1.1.md which brings the specification up to date. It discusses content addressability, new fields in the image configuration, the repository/tag grammar, and the current mechanism for exporting an image. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
a07dd24
to
4fa0ecc
Compare
content LGTM |
LGTM |
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
from moby/moby#22264 specifically commit a0237314277148093987380edc10222e63ba19e2 This does remove the one-line-per-sentence formatting, but we can address that later. Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
A small fix from the v1.1 merge that happened inside of the docker repo after we pulled this in. See moby/moby#22264 Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
The image spec in image/spec/v1.md is quite a bit out of date. Not only
is it missing the changes that went into 1.10 for content
addressability, but it has inaccuracies that date back further, such as
mentioning storing tarsum in the image configuration.
This commit creates image/spec/v1.1.md which brings the specification up
to date. It discusses content addressability, new fields in the image
configuration, the repository/tag grammar, and the current mechanism for
exporting an image.
To review this, I suggest diffing against
v1.md
.cc @stevvooe @tonistiigi @dmcgowan @vbatts