Add target media type to blob upload #2014
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In order to support homogenuous repositories in a user friendly manner, blob uploads must fail fast when attempting to upload content which has a media type which differs from previously uploaded content. This change adds a header to the blob upload which specifies what the media type inside the corresponding manifest will be. Since content uploads start by uploading blobs failing fast requires a change to the blob upload endpoint. Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
|
Ping @stevvooe and @aaronlehmann, do you mind commenting on the design here, this change is targetting 1.13 |
|
@dmcgowan I don't think the returned registry mediatype should ever be trusted. When I spec'ed out this feature in the past, the right approach centered around storing the mediatype as part of the repo blob reference, rather than with the blob itself, preventing disagreements with users. Ultimately, hashes should never be accessed without a corresponding descriptor, which should have the media type, so any headers from the registry are really unnecessary. |
|
Trying to put this in context... are you also planning to store the media type to enable this "fail fast" functionality? Would that mean 2x the number of blobs in the system (one for data, one for the media type)? |
|
@aaronlehmann yes, but the media type would only get saved on manifest upload. This change is to enable the engine to send up the information needed to fail fast, there will be a follow up change to update to registry to support this.
The media type in this case would likely just be stored in a descriptor file in the repository, not effecting the storage of blobs. We have not completed the design yet but we see that as the simplest method to support it. The thought now is to make it off by default but if we came up with a design we are happy with which always stored the mediatypes with references in a lightweight, that would be acceptable. @stevvooe I agree with you about where to store the mediatype. This change is just intended to be informational so the registry can decided whether to disallow the blob upload, the registry will not be storing this mediatype with the repo. It does leave open a race where two different types (say image and plugin) are being uploaded for the first time simultaneously, only after the manifests are uploaded may the repository be recognized as a specific media type though. In that case either all the blobs get uploaded from both pushes or one of the pushes fails before completing. Either way we cannot strictly protect against a race since the backend is eventually consistent. |
stevvooe
reviewed
Nov 2, 2016
| @@ -202,6 +202,7 @@ type CreateOptions struct { | |||
| // Blob access check will be skipped if set. | |||
| Stat *Descriptor | |||
| } | |||
| TargetMediaType string | |||
There was a problem hiding this comment.
Why isn't this a URL parameter, like mount and digest?
There was a problem hiding this comment.
This value is only informational and will be used to allow policy enforcement rather than correctness (digest) or user optimization(mount).
|
@dmcgowan The media type cannot be stored with the main blobs. It must be stored by repository, if at all. We also have the issue that a single blob may take on the role of different media types. |
|
We are no longer considering storing this media type but I think we still may want this header to be able to hint to the registry what the client intent is. My concern is the challenge response may need this to return a scope needs to complete a plugin push/pull. |
Current coverage is 50.84% (diff: 15.38%)@@ master #2014 diff @@
==========================================
Files 125 125
Lines 11136 11144 +8
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
- Hits 6809 5666 -1143
- Misses 3443 4729 +1286
+ Partials 884 749 -135
|
|
No longer needed, original need accomplished with repository class |
thaJeztah
pushed a commit
to thaJeztah/distribution
that referenced
this pull request
Apr 22, 2021
* First pass of tabs-based organization * Improvements * Second pass at tabs org * Move tab highlighting to Liquid instead of JS * Adding forwarding links for in-product TOCs * Move to pre-rendered left-navs instead of post-load JS for TOC sync * Optimizations and nosync-ing the Reference section * Optimizations, fix Cloud YAML * Make a "Sample applications" node * Update index.md * Tabs CSS fixes and 12-factor reposition * Theme Start (distribution#1709) * Hooking up nav to real TOC data, formatting fixes * Fixing JS error * Layout updates, dark themes, tons o stuff (distribution#1971) * Add cookie saving for day/night mode * Newsite tabs (distribution#2004) * Layout updates, dark themes, tons o stuff * Update themes Theme updates + scaffolding * Update style.css * Update style-alt.css * Missing font fixes * Import Open Sans from Google * Font fix, archive removal in TOC, favicon, Feedback img fix * Oops, returning -webkit-font-smoothing: antialiased; * Add old favicon.ico * Make archives a non-tiered link * Reorder docs archive to newest-first, add local instructions * Commenting out day/night switch for now * Fix 'rate this page' * Rate this page fixes * Autocomplete and Docker Cloud fixes * Open tree to current page * Adding indentation for nav collapse in * Ensure left nav visibly displays the current topic * Update flex layout - adjust rescale - code block styles * add focus to search - force code block color (for now) - increase section max-width * increase content padding - add padding to toc for wrapping long strings. * grid adjustment - grid - content and wrapper adjustments for mobile * left/right sidebar adjustments - refine position on scroll for toc on landing - add default height to compensate for upcoming position absolute onScroll * side bar overflow - hidden on X-scroll * fix version button - override bstrap defaults * tabs + buttons * update landing svgs * fix sidebar height set to 100% on landing pre-affix * Update blurb about engine/editions on front page * add side menu to mobile collapse menu * update classnames * overall mobile tweaks * Right-nav highlighting and auto-scroll * Slightly slower right-nav highlighting, correct version * add toggle menus for small devices * Fixing JS error/Docker 1.13>17.03 * header updates * re-add fan to header * update transition time * Add first 20 words to Twitter card * fixed width of components - lockdown elements on rescale (wil need more TLC) * set max-width of content * Left and right nav resizing w/footer scroll and window resize * update links on landing page * Fix for overzealous resizing, JS redundancies * Fix for JS error on homepage * JS error fixes * toggle adjustments - wrap toggle button * add tab width * version button type * version button both headers * tabs - fix typo * landing page grid * components * Share images, JS fixes, Marketo removal * Anchor links fix * Fix for black space on mobile * Restore hamburger (partial) * Update run.md Minor grammar cleanup. * Update apparmor.md I'm a little confused about which one is better to be used here, a period (.) or a colon (:), as a command is given below. Or both are OK, and we only have to keep consistency in a single page. * Update apparmor.md Fixed the indentation for the codeblock (indented by 4 spaces). Thank you for your careful review. * Replacing service with secret * Update networking.md fix typo with triple "m" for command word * Update run.md Address PR feedback. * Update install instructions to latest version * Added "related topics" section * Add documentation for mem_swappiness * Update to new Docker version scheme (distribution#1926) * mem_swappiness for current version and v1 * merge other changes, fix typo * There is no OpenSuSE and there never was though we had SuSE and S.u.S.E. * Add release notes for 1.12.6-cs9 (distribution#2028) Signed-off-by: Brian Goff <cpuguy83@gmail.com> * need sudo to access key cache (distribution#1931) * need sudo to access key cache * List other keyservers to try for cs-engine install (distribution#2033) * List other keyservers to try for cs-engine install Sometimes ha.pool.sks-keyservers.net goes down, so let's provide some other keyservers to try in such cases. Signed-off-by: Brian Goff <cpuguy83@gmail.com> * Update work_issue.md (distribution#2030) Change "re-start" to "restart". Though not included in "Prefered usages" in the documentation guide, but I think "restart" is better and used more frequently. Besides, some other docs here, such as "Keep containers alive during daemon downtime" of "Admin Guide", also use "restart". * Update create_pr.md (distribution#2015) * Update work_issue.md (distribution#2013) Change "id" to "ID" except for those in code. * Update set_up_dev.md (distribution#2011) Add periods (.) in some steps. * Update set_up_dev.md (distribution#2010) Apply Oxford Comma as described in the documentation guide. * Update create_pr.md (distribution#2014) Delete an extra space. * Update trust_key_mng.md (distribution#1883) * Update trust_key_mng.md * Update trust_key_mng.md I don‘t know how the whitespace appears, and it seems that it appears because something happened related to its original format (right-aligned pipe characters) and my change. Still unknown. Now I've deleted some redundant whitespace. * Update I don‘t know how the whitespace appears, and it seems that it appears because something happened related to its original format (right-aligned pipe characters) and my change. Still unknown. Now I've deleted some redundant whitespace. * Update content_trust.md (distribution#1912) * Update content_trust.md * update deprecation policy Signed-off-by: Victor Vieux <victorvieux@gmail.com> * Update info about how to check whether Docker is running * Updated docs to reflect edge channel Signed-off-by: French Ben <frenchben@docker.com> * Updated wording for SP creation Signed-off-by: French Ben <frenchben@docker.com> * beta to edge, cloud features first draft added cloud images Signed-off-by: Victoria Bialas <victoria.bialas@docker.com> * Distinguish between cloud stack file and stack file * Added EE links Signed-off-by: French Ben <frenchben@docker.com> * Use variables Signed-off-by: French Ben <frenchben@docker.com> * Replace deprecated MAINTAINER with LABEL (distribution#1445) Replace MAINTAINER instruction with LABEL as MAINTAINER was deprecated in moby/moby#25466 * Updates for Docker CE and Docker EE * Updated DDC launch button Signed-off-by: French Ben <frenchben@docker.com> * added Docker Cloud topics for Mac and Windows Signed-off-by: Victoria Bialas <victoria.bialas@docker.com> * d4mac, d4win stable and beta release notes for 17.03.0 Signed-off-by: Victoria Bialas <victoria.bialas@docker.com>
thaJeztah
pushed a commit
to thaJeztah/distribution
that referenced
this pull request
Jan 19, 2022
* First pass of tabs-based organization * Improvements * Second pass at tabs org * Move tab highlighting to Liquid instead of JS * Adding forwarding links for in-product TOCs * Move to pre-rendered left-navs instead of post-load JS for TOC sync * Optimizations and nosync-ing the Reference section * Optimizations, fix Cloud YAML * Make a "Sample applications" node * Update index.md * Tabs CSS fixes and 12-factor reposition * Theme Start (distribution#1709) * Hooking up nav to real TOC data, formatting fixes * Fixing JS error * Layout updates, dark themes, tons o stuff (distribution#1971) * Add cookie saving for day/night mode * Newsite tabs (distribution#2004) * Layout updates, dark themes, tons o stuff * Update themes Theme updates + scaffolding * Update style.css * Update style-alt.css * Missing font fixes * Import Open Sans from Google * Font fix, archive removal in TOC, favicon, Feedback img fix * Oops, returning -webkit-font-smoothing: antialiased; * Add old favicon.ico * Make archives a non-tiered link * Reorder docs archive to newest-first, add local instructions * Commenting out day/night switch for now * Fix 'rate this page' * Rate this page fixes * Autocomplete and Docker Cloud fixes * Open tree to current page * Adding indentation for nav collapse in * Ensure left nav visibly displays the current topic * Update flex layout - adjust rescale - code block styles * add focus to search - force code block color (for now) - increase section max-width * increase content padding - add padding to toc for wrapping long strings. * grid adjustment - grid - content and wrapper adjustments for mobile * left/right sidebar adjustments - refine position on scroll for toc on landing - add default height to compensate for upcoming position absolute onScroll * side bar overflow - hidden on X-scroll * fix version button - override bstrap defaults * tabs + buttons * update landing svgs * fix sidebar height set to 100% on landing pre-affix * Update blurb about engine/editions on front page * add side menu to mobile collapse menu * update classnames * overall mobile tweaks * Right-nav highlighting and auto-scroll * Slightly slower right-nav highlighting, correct version * add toggle menus for small devices * Fixing JS error/Docker 1.13>17.03 * header updates * re-add fan to header * update transition time * Add first 20 words to Twitter card * fixed width of components - lockdown elements on rescale (wil need more TLC) * set max-width of content * Left and right nav resizing w/footer scroll and window resize * update links on landing page * Fix for overzealous resizing, JS redundancies * Fix for JS error on homepage * JS error fixes * toggle adjustments - wrap toggle button * add tab width * version button type * version button both headers * tabs - fix typo * landing page grid * components * Share images, JS fixes, Marketo removal * Anchor links fix * Fix for black space on mobile * Restore hamburger (partial) * Update run.md Minor grammar cleanup. * Update apparmor.md I'm a little confused about which one is better to be used here, a period (.) or a colon (:), as a command is given below. Or both are OK, and we only have to keep consistency in a single page. * Update apparmor.md Fixed the indentation for the codeblock (indented by 4 spaces). Thank you for your careful review. * Replacing service with secret * Update networking.md fix typo with triple "m" for command word * Update run.md Address PR feedback. * Update install instructions to latest version * Added "related topics" section * Add documentation for mem_swappiness * Update to new Docker version scheme (distribution#1926) * mem_swappiness for current version and v1 * merge other changes, fix typo * There is no OpenSuSE and there never was though we had SuSE and S.u.S.E. * Add release notes for 1.12.6-cs9 (distribution#2028) Signed-off-by: Brian Goff <cpuguy83@gmail.com> * need sudo to access key cache (distribution#1931) * need sudo to access key cache * List other keyservers to try for cs-engine install (distribution#2033) * List other keyservers to try for cs-engine install Sometimes ha.pool.sks-keyservers.net goes down, so let's provide some other keyservers to try in such cases. Signed-off-by: Brian Goff <cpuguy83@gmail.com> * Update work_issue.md (distribution#2030) Change "re-start" to "restart". Though not included in "Prefered usages" in the documentation guide, but I think "restart" is better and used more frequently. Besides, some other docs here, such as "Keep containers alive during daemon downtime" of "Admin Guide", also use "restart". * Update create_pr.md (distribution#2015) * Update work_issue.md (distribution#2013) Change "id" to "ID" except for those in code. * Update set_up_dev.md (distribution#2011) Add periods (.) in some steps. * Update set_up_dev.md (distribution#2010) Apply Oxford Comma as described in the documentation guide. * Update create_pr.md (distribution#2014) Delete an extra space. * Update trust_key_mng.md (distribution#1883) * Update trust_key_mng.md * Update trust_key_mng.md I don‘t know how the whitespace appears, and it seems that it appears because something happened related to its original format (right-aligned pipe characters) and my change. Still unknown. Now I've deleted some redundant whitespace. * Update I don‘t know how the whitespace appears, and it seems that it appears because something happened related to its original format (right-aligned pipe characters) and my change. Still unknown. Now I've deleted some redundant whitespace. * Update content_trust.md (distribution#1912) * Update content_trust.md * update deprecation policy Signed-off-by: Victor Vieux <victorvieux@gmail.com> * Update info about how to check whether Docker is running * Updated docs to reflect edge channel Signed-off-by: French Ben <frenchben@docker.com> * Updated wording for SP creation Signed-off-by: French Ben <frenchben@docker.com> * beta to edge, cloud features first draft added cloud images Signed-off-by: Victoria Bialas <victoria.bialas@docker.com> * Distinguish between cloud stack file and stack file * Added EE links Signed-off-by: French Ben <frenchben@docker.com> * Use variables Signed-off-by: French Ben <frenchben@docker.com> * Replace deprecated MAINTAINER with LABEL (distribution#1445) Replace MAINTAINER instruction with LABEL as MAINTAINER was deprecated in moby/moby#25466 * Updates for Docker CE and Docker EE * Updated DDC launch button Signed-off-by: French Ben <frenchben@docker.com> * added Docker Cloud topics for Mac and Windows Signed-off-by: Victoria Bialas <victoria.bialas@docker.com> * d4mac, d4win stable and beta release notes for 17.03.0 Signed-off-by: Victoria Bialas <victoria.bialas@docker.com>
dylanrhysscott
pushed a commit
to digitalocean/docker-distribution
that referenced
this pull request
Jan 5, 2023
* First pass of tabs-based organization * Improvements * Second pass at tabs org * Move tab highlighting to Liquid instead of JS * Adding forwarding links for in-product TOCs * Move to pre-rendered left-navs instead of post-load JS for TOC sync * Optimizations and nosync-ing the Reference section * Optimizations, fix Cloud YAML * Make a "Sample applications" node * Update index.md * Tabs CSS fixes and 12-factor reposition * Theme Start (distribution#1709) * Hooking up nav to real TOC data, formatting fixes * Fixing JS error * Layout updates, dark themes, tons o stuff (distribution#1971) * Add cookie saving for day/night mode * Newsite tabs (distribution#2004) * Layout updates, dark themes, tons o stuff * Update themes Theme updates + scaffolding * Update style.css * Update style-alt.css * Missing font fixes * Import Open Sans from Google * Font fix, archive removal in TOC, favicon, Feedback img fix * Oops, returning -webkit-font-smoothing: antialiased; * Add old favicon.ico * Make archives a non-tiered link * Reorder docs archive to newest-first, add local instructions * Commenting out day/night switch for now * Fix 'rate this page' * Rate this page fixes * Autocomplete and Docker Cloud fixes * Open tree to current page * Adding indentation for nav collapse in * Ensure left nav visibly displays the current topic * Update flex layout - adjust rescale - code block styles * add focus to search - force code block color (for now) - increase section max-width * increase content padding - add padding to toc for wrapping long strings. * grid adjustment - grid - content and wrapper adjustments for mobile * left/right sidebar adjustments - refine position on scroll for toc on landing - add default height to compensate for upcoming position absolute onScroll * side bar overflow - hidden on X-scroll * fix version button - override bstrap defaults * tabs + buttons * update landing svgs * fix sidebar height set to 100% on landing pre-affix * Update blurb about engine/editions on front page * add side menu to mobile collapse menu * update classnames * overall mobile tweaks * Right-nav highlighting and auto-scroll * Slightly slower right-nav highlighting, correct version * add toggle menus for small devices * Fixing JS error/Docker 1.13>17.03 * header updates * re-add fan to header * update transition time * Add first 20 words to Twitter card * fixed width of components - lockdown elements on rescale (wil need more TLC) * set max-width of content * Left and right nav resizing w/footer scroll and window resize * update links on landing page * Fix for overzealous resizing, JS redundancies * Fix for JS error on homepage * JS error fixes * toggle adjustments - wrap toggle button * add tab width * version button type * version button both headers * tabs - fix typo * landing page grid * components * Share images, JS fixes, Marketo removal * Anchor links fix * Fix for black space on mobile * Restore hamburger (partial) * Update run.md Minor grammar cleanup. * Update apparmor.md I'm a little confused about which one is better to be used here, a period (.) or a colon (:), as a command is given below. Or both are OK, and we only have to keep consistency in a single page. * Update apparmor.md Fixed the indentation for the codeblock (indented by 4 spaces). Thank you for your careful review. * Replacing service with secret * Update networking.md fix typo with triple "m" for command word * Update run.md Address PR feedback. * Update install instructions to latest version * Added "related topics" section * Add documentation for mem_swappiness * Update to new Docker version scheme (distribution#1926) * mem_swappiness for current version and v1 * merge other changes, fix typo * There is no OpenSuSE and there never was though we had SuSE and S.u.S.E. * Add release notes for 1.12.6-cs9 (distribution#2028) Signed-off-by: Brian Goff <cpuguy83@gmail.com> * need sudo to access key cache (distribution#1931) * need sudo to access key cache * List other keyservers to try for cs-engine install (distribution#2033) * List other keyservers to try for cs-engine install Sometimes ha.pool.sks-keyservers.net goes down, so let's provide some other keyservers to try in such cases. Signed-off-by: Brian Goff <cpuguy83@gmail.com> * Update work_issue.md (distribution#2030) Change "re-start" to "restart". Though not included in "Prefered usages" in the documentation guide, but I think "restart" is better and used more frequently. Besides, some other docs here, such as "Keep containers alive during daemon downtime" of "Admin Guide", also use "restart". * Update create_pr.md (distribution#2015) * Update work_issue.md (distribution#2013) Change "id" to "ID" except for those in code. * Update set_up_dev.md (distribution#2011) Add periods (.) in some steps. * Update set_up_dev.md (distribution#2010) Apply Oxford Comma as described in the documentation guide. * Update create_pr.md (distribution#2014) Delete an extra space. * Update trust_key_mng.md (distribution#1883) * Update trust_key_mng.md * Update trust_key_mng.md I don‘t know how the whitespace appears, and it seems that it appears because something happened related to its original format (right-aligned pipe characters) and my change. Still unknown. Now I've deleted some redundant whitespace. * Update I don‘t know how the whitespace appears, and it seems that it appears because something happened related to its original format (right-aligned pipe characters) and my change. Still unknown. Now I've deleted some redundant whitespace. * Update content_trust.md (distribution#1912) * Update content_trust.md * update deprecation policy Signed-off-by: Victor Vieux <victorvieux@gmail.com> * Update info about how to check whether Docker is running * Updated docs to reflect edge channel Signed-off-by: French Ben <frenchben@docker.com> * Updated wording for SP creation Signed-off-by: French Ben <frenchben@docker.com> * beta to edge, cloud features first draft added cloud images Signed-off-by: Victoria Bialas <victoria.bialas@docker.com> * Distinguish between cloud stack file and stack file * Added EE links Signed-off-by: French Ben <frenchben@docker.com> * Use variables Signed-off-by: French Ben <frenchben@docker.com> * Replace deprecated MAINTAINER with LABEL (distribution#1445) Replace MAINTAINER instruction with LABEL as MAINTAINER was deprecated in moby/moby#25466 * Updates for Docker CE and Docker EE * Updated DDC launch button Signed-off-by: French Ben <frenchben@docker.com> * added Docker Cloud topics for Mac and Windows Signed-off-by: Victoria Bialas <victoria.bialas@docker.com> * d4mac, d4win stable and beta release notes for 17.03.0 Signed-off-by: Victoria Bialas <victoria.bialas@docker.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to support homogeneous repositories in a user friendly manner, blob uploads must fail fast when attempting to upload content which has a media type which differs from previously uploaded content. This change adds a header to the blob upload which specifies what the media type inside the corresponding manifest will be. Since content uploads start by uploading blobs failing fast requires a change to the blob upload endpoint.
This change does not alter the behavior of blob uploads but is required to add support in the client for registries with this feature enabled.