Upgrade sccache to v0.8.2 for CPU targets #121323
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/121323
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 1f7b209 with merge base b7524b0 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Because some implementations, like OpenDAL does not work with AWS IMDSv2, but this script will bridge the gap and enables more recent `sccache` releases(that switched from simple-s3 to OpenDAL) to work in current CI system When launched it prints something like: ``` export AWS_ACCESS_KEY_ID=XXXXX export AWS_SECRET_ACCESS_KEY=YYYY export AWS_SESSION_TOKEN=ZZZZ ``` which can be `eval`ed and passed then sccache can use those failures. Validated in #121323 Pull Request resolved: #121426 Approved by: https://github.com/Skylion007
|
IIRC, the upstream branch didn't seem to fully work with nvcc like our ancient fork when I tried this a year ago. I attempted to submit a PR to Mozilla with the additional changes but my Rust knowledge was too limit to make it work. May be upstream is good now, let's see |
I needed to make a tweak to our wrapper of wrappers and it seems to be working, but indeed, let's see... |
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
9976af4 to
cc2f0f7
Compare
|
Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as |
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
cc2f0f7 to
f484375
Compare
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
aaeb486 to
1a5b994
Compare
b77ea51 to
62f55b4
Compare
62f55b4 to
5561c1c
Compare
5561c1c to
61aa30c
Compare
63c60b5 to
1f7b209
Compare
|
@pytorch merge -f "Let's test in trunk!!!" |
|
@pytorchbot merge -f "Let's test in trunk!!!" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Fix a regression caused by #121323 Works for #114850 Pull Request resolved: #139604 Approved by: https://github.com/atalman, https://github.com/malfet
This essentially reverts pytorch#95997 but switches to builds from source to official mozilla's sccache repo for CPU builds, except PCH one, see pytorch#139188 - Define `SCCACHE_REGION` for the jobs that needs it. - Enable aarch64 builds to use sccache, which allows one to do incremental rebuilds under 10 min, see https://github.com/pytorch/pytorch/actions/runs/11565944328/job/32197278296 Fixes pytorch#121559 Pull Request resolved: pytorch#121323 Approved by: https://github.com/atalman
Instead of an ancient prebuilt binary This is a followup from #121323 For some reason, newer `sccache` does not work when `gcc` is invoked with `-E` option, so one have to special-case `-E` case in `/opt/ccache/bin/gcc` wrapper, which had to be special cased to work with `nvcc` by checking whether `-E` is passed not only as first or second, but as 3rd argument as well(to be followed up by a generic #142813 ), i.e. to generate following wrapper: ```shell #!/bin/sh if [ "$1" = "-E" ] || [ "$2" = "-E" ] || [ "$3" = "-E" ]; then exec /usr/bin/gcc "$@" elif [ $(env -u LD_PRELOAD ps -p $PPID -o comm=) != sccache ]; then exec sccache /usr/bin/gcc "$@" else exec /usr/bin/gcc "$@" fi ``` Without it `sccache nvcc hello.cu` failed with no-descriptive ``` sccache: error: failed to execute compile sccache: caused by: Compiler not supported: "" ``` Pull Request resolved: #140614 Approved by: https://github.com/wdvr Co-authored-by: Wouter Devriendt <wouterdevriendt@meta.com>
This essentially reverts #95997 but switches to builds from source to official mozilla's sccache repo for CPU builds, except PCH one, see #139188
SCCACHE_REGIONfor the jobs that needs it.Fixes #121559