fix: Add AQUA_DOWNLOAD_USER_AGENT variable and set User-Agent to GitHub Release downloads #4019
Conversation
- Add userAgentTransport to wrap HTTP requests with Mozilla User-Agent - This helps avoid being blocked by cloud storage services like Azure/S3 - Applies to both direct GitHub API downloads and redirect URL downloads
|
Thank you for your investigation and contribution! |
|
Sorry, I don't have reference. I just had many trial and errors. Since this is their internal specification, it might be difficult to give evidences. |
|
|
Nice! It seems to be good to set user-agent in the other places as well. Shall I do it? |
|
go-github usually creates http.Request by Client.NewRequest, and it sets the user agent header. https://github.com/google/go-github/blob/718c304661aa42ba7a01da5b54b2ce85f2f48234/github/repos_releases.go#L347 But when go-github downloads an asset from a redirect URL, it creates http.Request by http.NewRequest. I'll contribute to go-github. |
Looks good. Thank you. |
Makes sense. Can we use this fix until you fixed original issue of go-github as a workaround? |
|
Sure! |
pkg/download/github_release.go
Outdated
|
|
||
| func (t *userAgentTransport) RoundTrip(req *http.Request) (*http.Response, error) { | ||
| // Add a browser-like User-Agent to avoid being blocked by cloud storage services | ||
| req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; aqua/2.0; +https://aquaproj.github.io/)") |
There was a problem hiding this comment.
@suzuki-shunsuke As per additional investigation of us, it seems we don't need to align browser like user agent, what user-agent is preferred for you?
There was a problem hiding this comment.
- I tried to use
go-github/v73.0.0for user-agent, but it's flaky. - I tried to use
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36that is recent chrome user-agent. It's stable.
Since using recent browser to download assets always is succeeded, it might be better to use actual browser user-agent.
There was a problem hiding this comment.
I changed to use go-github/{version} for user-agent by default. But prepared AQUA_DOWNLOAD_USER_AGENT variable as well. Could you tell me your opinion 🙇
There was a problem hiding this comment.
Hmm. I don't have any strong opinion.
There was a problem hiding this comment.
Sorry. I didn't see your above two comments when I posted a previous comment.
I tried to use go-github/v73.0.0 for user-agent, but it's flaky.
I tried to use Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 that is recent chrome user-agent. It's stable.
I see. Then I think Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 is better than go-github/{version}.
There was a problem hiding this comment.
I changed it to chrome browser one. It's stable than go-gihub/{version} but sometime it's failed.
There might be other essential for success 🤔
|
ugh... when I test the change again, it becomes unstable again... |
|
I'll revert this pr. |
What
Set browser like user-agent for download.
fix: #4005
Why
I also faced this issue recently. And I was deeply look into it.
It seems the storage service or CDN verifying user-agent to reject bot access.
Since the current implementation is using default user-agent.
It might be identified as a bot.
If I changed the user-agent to browser like agent, the success rate was dramatically increased.
Check List
Require signed commits, so all commits must be signed