Skip to content

Conversation

aaronlehmann
Copy link
Contributor

The image spec in image/spec/v1.md is quite a bit out of date. Not only
is it missing the changes that went into 1.10 for content
addressability, but it has inaccuracies that date back further, such as
mentioning storing tarsum in the image configuration.

This commit creates image/spec/v1.1.md which brings the specification up
to date. It discusses content addressability, new fields in the image
configuration, the repository/tag grammar, and the current mechanism for
exporting an image.

To review this, I suggest diffing against v1.md.

cc @stevvooe @tonistiigi @dmcgowan @vbatts

@vbatts
Copy link
Contributor

vbatts commented Apr 22, 2016

OHMAN. Lots of good work here, @aaronlehmann.
/me reads

```

There are one or more directories named with the ChainID for each layer in a
full image. Each of these directories contains 3 files:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For docker-v1.10+ the directory layout isn't important anymore as all the paths are listed in manifest. We keep this structure because in tars from docker are compatible with both v1.0 and v1.1 specs. When we remove support for v1.0 we will probably also simplify this directory structure.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonistiigi as an aside, when do you think that will be? That is an important integration point for many.
Or rather, if we can evolve this into a common standard (read as OpenContainers) then would it need a --format=oci.v1 like integration?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. There is no problem with supporting both atm but we don't want to promote the old one because we can't have ID stability with the old version. id/json is the 1.0 image configuration with the parent chain and we don't use this method any more since v1.10.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

understood

On Wed, Apr 27, 2016 at 2:26 PM Tõnis Tiigi notifications@github.com
wrote:

In image/spec/v1.1.md
#22264 (comment):

+.
+├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json
+├── 5f29f704785248ddb9d06b90a11b5ea36c534865e9035e4022bb2e71d4ecbb9a
+│ ├── VERSION
+│ ├── json
+│ └── layer.tar
+├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198
+│ ├── VERSION
+│ ├── json
+│ └── layer.tar
+├── manifest.json
+└── repositories
+```
+
+There are one or more directories named with the ChainID for each layer in a
+full image. Each of these directories contains 3 files:

I don't know. There is no problem with supporting both atm but we don't
want to promote the old one because we can't have ID stability with the old
version. id/json is the 1.0 image configuration with the parent chain and
we don't use this method any more since v1.10.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/22264/files/2801c4c22c04396ec912760a955752cd8955e841#r61310686

@philips
Copy link
Contributor

philips commented Apr 27, 2016

Overall LGTM. What are the next steps to getting it merged?

1.0
```

The `repositories` file is another JSON file which describes names/tags:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should mention this file is also only used for backwards compatibility. Also at the end when discussing what "older implementations" will use.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a mention below.

@aaronlehmann aaronlehmann force-pushed the updated-image-spec branch 2 times, most recently from 22799c5 to a07dd24 Compare April 27, 2016 18:09
@dmcgowan
Copy link
Member

LGTM

vbatts added a commit to vbatts/oci-image-spec that referenced this pull request Apr 27, 2016
from moby/moby#22264
specifically commit 2801c4c22c04396ec912760a955752cd8955e841

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198
│   ├── VERSION
│   ├── json
│   └── layer.tar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any comment on the consistency of the mtime of these objects? That was a common important ask from folks in the community for consistency of docker save output (regardless that the checksum of the objects themselves being consistent)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we currently make any efforts to make mtimes consistent, or do anything else to support stable hashes of the save artifacts we generate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a decent conversation around this and I had written the code for
it. So it was a thing. The json and layer.tar were given an mtime of the
layer creation time, and repositories was Unix epoch of 0
It is nicely consistent. :-)

On Thu, May 12, 2016, 12:52 Aaron Lehmann notifications@github.com wrote:

In image/spec/v1.1.md
#22264 (comment):

  • - all tar archives of each layer filesystem changesets
    +
    +For example, here's what the full archive of library/busybox is (displayed in
    +tree format):
    +
    +```
    +.
    +├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json
    +├── 5f29f704785248ddb9d06b90a11b5ea36c534865e9035e4022bb2e71d4ecbb9a
    +│ ├── VERSION
    +│ ├── json
    +│ └── layer.tar
    +├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198
    +│ ├── VERSION
    +│ ├── json
    +│ └── layer.tar

I don't think we currently make any efforts to make mtimes consistent, or
do anything else to support stable hashes of the save artifacts we generate.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/22264/files/a07dd240ac7148093987380edc10222e63ba19e2#r63055557

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vbatts @aaronlehmann Yes, we still do the modified time reset like @vbatts described.

vbatts added a commit to vbatts/oci-image-spec that referenced this pull request Apr 27, 2016
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
</dt>
<dd>
A set of directories which should be created as data volumes in
a container running this image. This JSON structure value is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this sentence get added opencontainers/image-spec@949ea01

It applies to the docker behaviour as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That isn't quite correct. Volume is replaced but there is a copy operation that copies files from image to the volume. So the data from the image rootfs is still present.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh no. This sounds like a "clever" use-case, but would allow for sneaky bugs. :-\

Copy link
Contributor

@vbatts vbatts Apr 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonistiigi just checking on this, and i'm not seeing that to be the case. Perhaps you can provide an example of what you're seeing with host files of a volume path being copied up?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cat <<EOT > Dockerfile
from busybox
run mkdir /foo && touch /foo/bar
volume /foo
EOT
docker build -t foo .
ls -l $(docker inspect -f '{{(index .Mounts 0).Source}}' $(docker run -d foo))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonistiigi fair, but docker run -it -v /usr:/foo:ro foo ls /foo/ does not reveal /foo/bar

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but docker run -it -v /usr:/foo:ro foo ls /foo/ doesn't have much to do with Volumes struct in the image configuration. Bind mounts just override everything and this behavior is independent of the image config.

vbatts added a commit to vbatts/oci-image-spec that referenced this pull request Apr 29, 2016
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>

```
.
├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file is not explicitly referenced below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is referenced in the example manifest.json below.

@aaronlehmann
Copy link
Contributor Author

How do people feel about this PR? Are there any other changes we should make before merging?

@thaJeztah
Copy link
Member

I can do a review, once it's technically "approved", so happy to hear

└── repositories
```

There are one or more directories named with the ChainID for each layer in a
Copy link
Member

@tonistiigi tonistiigi May 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The directory name is not the ChainID. It is just a deterministic random hex. I think ChainID should just be an implementation detail, we probably don't need to define it at all here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonistiigi: Fixed.

@stevvooe
Copy link
Contributor

LGTM

The image spec in image/spec/v1.md is quite a bit out of date. Not only
is it missing the changes that went into 1.10 for content
addressability, but it has inaccuracies that date back further, such as
mentioning storing tarsum in the image configuration.

This commit creates image/spec/v1.1.md which brings the specification up
to date. It discusses content addressability, new fields in the image
configuration, the repository/tag grammar, and the current mechanism for
exporting an image.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
@tonistiigi
Copy link
Member

content LGTM

@thaJeztah
Copy link
Member

LGTM

@thaJeztah thaJeztah merged commit 7faa2a4 into moby:master May 24, 2016
philips pushed a commit to philips/image-spec that referenced this pull request May 26, 2016
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264
philips pushed a commit to philips/image-spec that referenced this pull request May 26, 2016
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
philips pushed a commit to philips/image-spec that referenced this pull request May 27, 2016
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
dattgoswami9lk5g added a commit to dattgoswami9lk5g/bremlinr that referenced this pull request Jun 6, 2022
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
dattgoswami9lk5g added a commit to dattgoswami9lk5g/bremlinr that referenced this pull request Jun 6, 2022
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
7c00d pushed a commit to 7c00d/J1nHyeockKim that referenced this pull request Jun 6, 2022
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
7c00d pushed a commit to 7c00d/J1nHyeockKim that referenced this pull request Jun 6, 2022
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
7c00d added a commit to 7c00d/J1nHyeockKim that referenced this pull request Jun 6, 2022
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
7c00d added a commit to 7c00d/J1nHyeockKim that referenced this pull request Jun 6, 2022
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
laventuraw added a commit to laventuraw/Kihara-tony0 that referenced this pull request Jun 6, 2022
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
laventuraw added a commit to laventuraw/Kihara-tony0 that referenced this pull request Jun 6, 2022
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
tomalopbsr0tt added a commit to tomalopbsr0tt/fabiojosej that referenced this pull request Oct 6, 2022
from moby/moby#22264
specifically commit a0237314277148093987380edc10222e63ba19e2

This does remove the one-line-per-sentence formatting, but we can
address that later.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
tomalopbsr0tt added a commit to tomalopbsr0tt/fabiojosej that referenced this pull request Oct 6, 2022
A small fix from the v1.1 merge that happened inside of the docker repo
after we pulled this in.

See moby/moby#22264

Signed-off-by: Brandon Philips <brandon.philips@coreos.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants