There are a couple usability issues around the SourceBuild metadata in the Version.Details.xml that should be addressed.

1. The SourceBuild metadata is only allowed on one dependency per repo (this is related to [ArPow] intermediate nupkg restore tooling should detect and fail if multiple <SourceBuild .../> elements exist for one upstream #2050). When a developer adds multiple, the UX is not intuitive on what is wrong. This is illustrated in Source build creates duplicate PackageReferences #3372.
2. In the past there have been numerous issues at release time when stable versions flow in. When the SourceBuild metadata is placed on a stable versioned dependency this breaks the source-build intermediate loading because the intermediate is non-shipping and usually doesn't get the stable versioning. When this happens the SourceBuild metadata is typically moved to a nonstable versioned dependency.

There are likely several options here. One possible solution is to explicitly add the SourceBuild.Intermediates as dependencies and only mark those as source-build.