Make vendor content addressable #11632
Conversation
The following makes the vendor directory content addressable. It uses the Gopkg.lock file for the sha and then downloads the content from s3 if it's available. SO in order to update the vendor directory for CI tests, you need to update your Gopkg.toml, which will cause the Gopkg.lock file to update. Then run sha256sum on the Gopkg.lock and upload that to the s3 bucket, ensuring to make it public.
|
|
|
!!build!! |
1 similar comment
|
!!build!! |
|
|
1 similar comment
|
|
|
|
||
| ensure_dep() { | ||
| if [ ! -d vendor ]; then | ||
| SHA=$(sha256sum Gopkg.lock | cut -d ' ' -f 1) |
There was a problem hiding this comment.
Why not use s3cmd to check whether the $sha.tar exists and if it doesn't, do a dep ensure followed by an tar+upload? Should be fine even in case of a race occurring (last write wins but contents will always be the same) .
Also, unless I'm mistaken S3 bucket contents have a limited lifespan (unless the admin explicitly overrides this) so unless we auto-generate/upload the files future builds are likely to eventually break needing manual intervention.
There was a problem hiding this comment.
Actually, I was able to dig up my old proposal for doing something like this as part of the CI setup script. Perhaps you might be able to repurpose some bits from there to make the above steps more resilient.
There was a problem hiding this comment.
Why not use s3cmd to check whether the $sha.tar exists
So that means I'd need to introduce s3cmd to make and I don't want to do that. I'd then have to introduce secrets etc...
do a dep ensure followed by an tar+upload?
It fails even attempting to do that, so we can't do that. You have to do it locally and then just download the contents.
Also, unless I'm mistaken S3 bucket contents have a limited lifespan
It's actually the opposite. You have to explicitly set a lifespan when you create the bucket.
There was a problem hiding this comment.
Ah, my bad; was probably thinking of another bucket which already had a policy assigned to it.
In any case, given that this change mostly benefits CI/build bots (what I mean it's fine to do a make dep from time to time when developing locally) you could make this part of the build scripts where s3cmd will be already configured for you...
On the other hand, as we probably wont be bumping deps in 2.7 frequently after 2.8 is out perhaps having someone on the team manually regenerate/upload the tar file when a dep change is made is probably fine...
|
|
|
I have cherry-picked the commit from this PR into #11660 so we can bypass the dep ensure blockage on CI. |
…t-least-384-bytes-long #11660 ## Description of change This PR lands a number of changes to unblock our CI. More specifically: - Go will not allow TLS handshakes with TLS1.2+ if key is 64 bytes long. This PR removes the PatchValue calls from the various test suites that forced a 512bit key length so that all our tests use the preferred (384 byte long) default key size. - In addition, this PR cherry-picks a commit from #11632 to fetch a snapshot of the vendor folder from S3 and includes additional commits to bypass 'dep ensure' (which is called multiple times by the various Makefile targets) when using the cached deps. - Finally, the PR backports the fixes from #11654 and #11594.
Description of change
The following makes the vendor directory content addressable. It uses
the Gopkg.lock file for the sha and then downloads the content from s3
if it's available.
SO in order to update the vendor directory for CI tests, you need to
update your Gopkg.toml, which will cause the Gopkg.lock file to update.
Then run sha256sum on the Gopkg.lock and upload that to the s3 bucket,
ensuring to make it public.
QA steps