Skip to content

Allow upload to a different origin on retryable errors #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 16, 2020

Conversation

evelynl94
Copy link
Collaborator

There is no retry in uploader so when client pushes to proxy and proxy uploads to origin, a failed call to origin could cause this chained push to fail, and registry's storage driver would return a non-informative 500 error:

{"log":"time=\"2020-09-15T18:29:24.098336695Z\" level=error msg=\"unknown error completing upload: upload: PUT https://10.86.32.14:9003/namespace/TODO/blobs/sha256:7e5a79dca6eaf28244b2787fc8b3ef388847363ba5cd705ac8a09b4fbf57b82c/uploads/a78df4be-9823-4859-855d-e6f4fa8d73d8 504: \u003chtml\u003e\\r\\n\u003chead\u003e\u003ctitle\u003e504 Gateway Time-out\u003c/title\u003e\u003c/head\u003e\\r\\n\u003cbody bgcolor=\\\"white\\\"\u003e\\r\\n\u003ccenter\u003e\u003ch1\u003e504 Gateway Time-out\u003c/h1\u003e\u003c/center\u003e\\r\\n\u003chr\u003e\u003ccenter\u003enginx/1.10.3\u003c/center\u003e\\r\\n\u003c/body\u003e\\r\\n\u003c/html\u003e\\r\\n\" go.version=go1.14.4 http.request.contenttype=application/octet-stream http.request.host=\"kraken-origin01-phx4:5367\" http.request.id=5ad657fe-c164-4ef5-aa70-8a63b294a610 http.request.method=PUT http.request.remoteaddr=@ http.request.uri=\"/v2/uber-usi/sdv-router/blobs/uploads/b0ffae9c-bf2f-4655-8254-a413b77db17d?_state=6s1EKv5wIWpiiOoPxg_dWXsw4pk1Z0L3SLurXEvORDV7Ik5hbWUiOiJ1YmVyLXVzaS9zZHYtcm91dGVyIiwiVVVJRCI6ImIwZmZhZTljLWJmMmYtNDY1NS04MjU0LWE0MTNiNzdkYjE3ZCIsIk9mZnNldCI6NjcwOTg2MTE1OCwiU3RhcnRlZEF0IjoiMjAyMC0wOS0xNVQxODoyNDo1M1oifQ%3D%3D\u0026digest=sha256%3A7e5a79dca6eaf28244b2787fc8b3ef388847363ba5cd705ac8a09b4fbf57b82c\" http.request.useragent=Go-http-client/1.1 vars.name=uber-usi/sdv-router vars.uuid=b0ffae9c-bf2f-4655-8254-a413b77db17d\n","stream":"stderr","time":"2020-09-15T18:29:24.098665108Z"}

Since the current origin is probably under high load due to hashing a big blob, retry on the same origin could make the load problem worse. This allow the cluster client to retry on a different origin if upload fails due to load error.

@codecov
Copy link

codecov bot commented Sep 16, 2020

Codecov Report

Merging #275 into master will increase coverage by 0.08%.
The diff coverage is 62.50%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #275      +/-   ##
==========================================
+ Coverage   66.34%   66.43%   +0.08%     
==========================================
  Files         186      186              
  Lines        9224     9226       +2     
==========================================
+ Hits         6120     6129       +9     
+ Misses       2340     2338       -2     
+ Partials      764      759       -5     
Impacted Files Coverage Δ
utils/httputil/httputil.go 58.20% <57.14%> (-0.26%) ⬇️
lib/store/ca_store.go 62.79% <100.00%> (+7.23%) ⬆️
lib/torrent/scheduler/dispatch/dispatcher.go 49.00% <0.00%> (-0.34%) ⬇️
lib/torrent/scheduler/events.go 72.59% <0.00%> (+1.44%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c58aef8...af04a74. Read the comment docs.

@evelynl94 evelynl94 merged commit db81d2c into master Sep 16, 2020
@evelynl94 evelynl94 deleted the evelynl/upload branch September 16, 2020 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants