Make remoteCacheId content-based #6026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bjaglin
reviewed
Oct 26, 2020
bjaglin
reviewed
Oct 26, 2020
Comment on lines
54
to
57
| remoteCacheId := { | ||
| val compileExtraInc = (Compile / extraIncOptions).value | ||
| val compileInputs = (Compile / unmanagedSources / inputFileStamps).value | ||
| val compileCp = (Compile / externalDependencyClasspath / outputFileStamps).value |
There was a problem hiding this comment.
Could this be defined at the configuration level (and combined at the project level via a scope filter), so that other configurations such as IntegrationTest can more easily be supported?
sbt/main/src/main/scala/sbt/internal/RemoteCache.scala
Lines 206 to 207 in 5a960f2
Seq[Def.Setting[_]] ?
There was a problem hiding this comment.
Good idea. I'm hoping I can eventually implement pullRemoteCache at the configuration level.
Refactor remote caching to be scoped to configuration. In addition, this avoid the use of dependency resolver (since I'm not resolving anything) and directly invoke the Ivy resolver for the artifact, somewhat analogus to publishing process. This should speed up the `pullRemoteCache` since it avoids the POM download as well. For sbt-binrary-remote-cache this created a bit of complication since the (publishing) resolver doesn't act correctly as (downloading) resolver in terms of the credentials, so I had to create a new key `remoteCacheResolvers` to have asymmetric resolver.
eed3si9n
added a commit
to eed3si9n/sbt
that referenced
this pull request
Nov 22, 2020
Fixes sbt#6102 sbt#6026 changed the implementation of remote cache to NOT use dependency resolution (Coursier), and directly use Ivy resolver for efficiency. This was good, but when I made the change, I've changed the cache directory to be `crossTarget.value / "remote-cache"`. This was ok for local testing purpose, but not great for real usage since we don't want the cache to be wiped out either in the CI machines or on a local laptop. This adds a new Global key called `localCacheDirectory`. Similar to Coursier cache, this is meant to be shared across all builds running on a machine. Also similar to Coursier cache this will try to follow the operating system specifc caching directory. ### localCacheDirectory location - Environment variable: `SBT_LOCAL_CACHE` - System property: `sbt.global.localcache` - Windows: %LOCALAPPDATA%\sbt\v1 - macOS: $HOME/Library/Caches/sbt/v1 - Linux: $HOME/.cache/sbt/v1
eed3si9n
added a commit
to eed3si9n/sbt
that referenced
this pull request
Nov 22, 2020
Fixes sbt#6102 sbt#6026 changed the implementation of remote cache to NOT use dependency resolution (Coursier), and directly use Ivy resolver for efficiency. This was good, but when I made the change, I've changed the cache directory to be `crossTarget.value / "remote-cache"`. This was ok for local testing purpose, but not great for real usage since we don't want the cache to be wiped out either in the CI machines or on a local laptop. This adds a new Global key called `localCacheDirectory`. Similar to Coursier cache, this is meant to be shared across all builds running on a machine. Also similar to Coursier cache this will try to follow the operating system specifc caching directory. ### localCacheDirectory location - Environment variable: `SBT_LOCAL_CACHE` - System property: `sbt.global.localcache` - Windows: %LOCALAPPDATA%\sbt\v1 - macOS: $HOME/Library/Caches/sbt/v1 - Linux: $HOME/.cache/sbt/v1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #5842
Using Git commit for remote caching id was one of the first things people think about, but it doesn't scale because that means you're downloading all module JARs all the time anytime anyone adds a commit. The basis of remote caching is that people are adding commits to many subprojects, so that's bad.
As an alternative, this uses content hash of input files and external dependency JARs.
remoteCacheId,pushRemoteCache, andpullRemoteCacheare now scoped to configuration (CompileandTest). Since we're not resolving any version conflicts, dependency resolver is no longer used and the artifacts are directly downloaded from the Ivy resolver. This should speed up thepullRemoteCachesince it avoids the POM download as well.note
For sbt-binrary-remote-cache this created a bit of complication since the (publishing) resolver doesn't act correctly as (downloading) resolver in terms of the credentials, so I had to create a new key
remoteCacheResolversto have asymmetric resolver.