Use working pre-built sccache binary #95997

huydhn · 2023-03-03T21:15:24Z

From #95938 where a new Docker image build fails to start sccache. This issue starts to happen today (Mar 3rd). The server fails to start with a cryptic sccache: error: Invalid argument (os error 22)

=================== sccache compilation log ===================
ERROR 2023-03-03T20:31:14Z: sccache::server: failed to start server: Invalid argument (os error 22)

sccache: error: Invalid argument (os error 22)

=========== If your build fails, please take a look at the log above for possible reasons ===========

+ sccache --show-stats
sccache: error: Connection to server timed out

I don't have a good explanation for this yet. The version of sccache we build from https://github.com/pytorch/sccache is ancient. If I start to build the exact same version on Ubuntu Docker image now, the issue will manifest. But the older binary built only few days ago https://hud.pytorch.org/pytorch/pytorch/commit/e50ff3fcdb3890ce3bbab99e60b1c27ff49be2af works without any issue. So I fix sccache binary to that version instead of rebuilding it every time in the image as a temporary mitigation while trying to root cause this further.

pytorch-bot · 2023-03-03T21:15:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/95997

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b7ec0c1:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

huydhn · 2023-03-03T22:28:37Z

~~Some above steps fails because I forgot to grant read access to the binary from S3. Once pull workflow finishes, retrying will work~~ All looks green now

huydhn · 2023-03-04T03:47:21Z

I probably need someone with Rust knowledge to debug the issue further. So here is my finding so far. The server failed to start because it couldn't bind to the port (invalid argument, errno 22). Our sccache fork uses this old crate https://crates.io/crates/tokio-tcp/0.1.4 for the job:

use tokio_tcp::TcpListener;
...
let addr = SocketAddrV4::new(Ipv4Addr::new(127, 0, 0, 1), port);
let listener = TcpListener::bind(&SocketAddr::V4(addr))?; <--- FAILING HERE
...

The code has not changed, so I still couldn't tell why it starts failing now. Confirm that there is no process running hoarding sccache port. So the only thing left from https://man7.org/linux/man-pages/man2/bind.2.html is:

EINVAL addrlen is wrong, or addr is not a valid address for this socket's domain.

But AFIAK, the code looks reasonable, binding to "127.0.0.1:4226". I already create a task T136185557 to update sccache version, but lack the Rust knowledge to finish it. That would be the long term fix for this issue.

huydhn · 2023-03-04T21:21:03Z

@ZainRizvi Let's get this merge to unblock your PR and others like #95896 to update triton pinned commit. Anything touching Docker image would be blocked otherwise.

@ngimel Those failed builds due to sccache in #95896 would be fixed once this PR is merged.

ngimel · 2023-03-04T22:15:56Z

@huydhn great, thanks! #95896 also needs CMake v 3.20 (we currently have v 3.17), can we update that?

huydhn · 2023-03-06T03:58:39Z

@huydhn great, thanks! #95896 also needs CMake v 3.20 (we currently have v 3.17), can we update that?

FYI, the PR to upgrade cmake version is here pytorch/builder#1331

ZainRizvi

Thanks for debugging this!

Any idea who owns our sccache fork? It would be good to raise this issue to them (if it even has an owner)

huydhn · 2023-03-06T18:20:42Z

Any idea who owns our sccache fork? It would be good to raise this issue to them (if it even has an owner)

Ed was last seen updating it https://github.com/pytorch/sccache/commits/master, but I think this is likely that we are the owner. Let's me bring this up for discussion this week.

huydhn · 2023-03-06T18:23:09Z

@pytorchbot merge -f 'Docker build and all pull jobs have passed. The cache looks fine. We can skip trunk and save some trees'

pytorchmergebot · 2023-03-06T18:24:52Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

From pytorch/pytorch#95938 where a new Docker image build fails to start sccache. This issue starts to happen today (Mar 3rd). The server fails to start with a cryptic `sccache: error: Invalid argument (os error 22)` ``` =================== sccache compilation log =================== ERROR 2023-03-03T20:31:14Z: sccache::server: failed to start server: Invalid argument (os error 22) sccache: error: Invalid argument (os error 22) =========== If your build fails, please take a look at the log above for possible reasons =========== + sccache --show-stats sccache: error: Connection to server timed out ``` I don't have a good explanation for this yet. The version of sccache we build from https://github.com/pytorch/sccache is ancient. If I start to build the exact same version on Ubuntu Docker image now, the issue will manifest. But the older binary built only few days ago https://hud.pytorch.org/pytorch/pytorch/commit/e50ff3fcdb3890ce3bbab99e60b1c27ff49be2af works without any issue. So I fix sccache binary to that version instead of rebuilding it every time in the image as a temporary mitigation while trying to root cause this further. Pull Request resolved: pytorch/pytorch#95997 Approved by: https://github.com/ZainRizvi

From pytorch#95938 where a new Docker image build fails to start sccache. This issue starts to happen today (Mar 3rd). The server fails to start with a cryptic `sccache: error: Invalid argument (os error 22)` ``` =================== sccache compilation log =================== ERROR 2023-03-03T20:31:14Z: sccache::server: failed to start server: Invalid argument (os error 22) sccache: error: Invalid argument (os error 22) =========== If your build fails, please take a look at the log above for possible reasons =========== + sccache --show-stats sccache: error: Connection to server timed out ``` I don't have a good explanation for this yet. The version of sccache we build from https://github.com/pytorch/sccache is ancient. If I start to build the exact same version on Ubuntu Docker image now, the issue will manifest. But the older binary built only few days ago https://hud.pytorch.org/pytorch/pytorch/commit/e50ff3fcdb3890ce3bbab99e60b1c27ff49be2af works without any issue. So I fix sccache binary to that version instead of rebuilding it every time in the image as a temporary mitigation while trying to root cause this further. Pull Request resolved: pytorch#95997 Approved by: https://github.com/ZainRizvi

malfet · 2024-03-06T18:23:48Z

We should have a followup task for this one, to resort back to the source builds

This essentially reverts #95997 but switches to builds from source to official mozilla's sccache repo for CPU builds, except PCH one, see #139188 - Define `SCCACHE_REGION` for the jobs that needs it. - Enable aarch64 builds to use sccache, which allows one to do incremental rebuilds under 10 min, see https://github.com/pytorch/pytorch/actions/runs/11565944328/job/32197278296 Fixes #121559 Pull Request resolved: #121323 Approved by: https://github.com/atalman

This essentially reverts pytorch#95997 but switches to builds from source to official mozilla's sccache repo for CPU builds, except PCH one, see pytorch#139188 - Define `SCCACHE_REGION` for the jobs that needs it. - Enable aarch64 builds to use sccache, which allows one to do incremental rebuilds under 10 min, see https://github.com/pytorch/pytorch/actions/runs/11565944328/job/32197278296 Fixes pytorch#121559 Pull Request resolved: pytorch#121323 Approved by: https://github.com/atalman

Use working pre-built sccache binary

6211638

huydhn requested review from ZainRizvi and a team March 3, 2023 21:15

pytorch-bot bot added the topic: not user facing topic category label Mar 3, 2023

huydhn added the test-config/default label Mar 3, 2023

huydhn marked this pull request as ready for review March 3, 2023 22:26

huydhn requested a review from jeffdaily as a code owner March 3, 2023 22:26

huydhn requested a review from malfet March 3, 2023 23:02

Add comment

b7ec0c1

ZainRizvi approved these changes Mar 6, 2023

View reviewed changes

pytorchmergebot added the Merged label Mar 6, 2023

pytorchmergebot closed this in 000d9ec Mar 6, 2023

malfet mentioned this pull request Mar 6, 2024

Upgrade sccache to v0.8.2 for CPU targets #121323

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use working pre-built sccache binary #95997

Use working pre-built sccache binary #95997

Uh oh!

huydhn commented Mar 3, 2023

Uh oh!

pytorch-bot bot commented Mar 3, 2023 •

edited

Loading

Uh oh!

huydhn commented Mar 3, 2023 •

edited

Loading

Uh oh!

huydhn commented Mar 4, 2023 •

edited

Loading

Uh oh!

huydhn commented Mar 4, 2023

Uh oh!

ngimel commented Mar 4, 2023

Uh oh!

huydhn commented Mar 6, 2023

Uh oh!

ZainRizvi left a comment

Uh oh!

huydhn commented Mar 6, 2023

Uh oh!

huydhn commented Mar 6, 2023

Uh oh!

pytorchmergebot commented Mar 6, 2023

Uh oh!

malfet commented Mar 6, 2024

Uh oh!

Uh oh!

Use working pre-built sccache binary #95997

Use working pre-built sccache binary #95997

Uh oh!

Conversation

huydhn commented Mar 3, 2023

Uh oh!

pytorch-bot bot commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/95997

✅ No Failures

Uh oh!

huydhn commented Mar 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huydhn commented Mar 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huydhn commented Mar 4, 2023

Uh oh!

ngimel commented Mar 4, 2023

Uh oh!

huydhn commented Mar 6, 2023

Uh oh!

ZainRizvi left a comment

Choose a reason for hiding this comment

Uh oh!

huydhn commented Mar 6, 2023

Uh oh!

huydhn commented Mar 6, 2023

Uh oh!

pytorchmergebot commented Mar 6, 2023

Merge started

Uh oh!

malfet commented Mar 6, 2024

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 3, 2023 •

edited

Loading

huydhn commented Mar 3, 2023 •

edited

Loading

huydhn commented Mar 4, 2023 •

edited

Loading