-
Notifications
You must be signed in to change notification settings - Fork 566
Description
Describe the bug
The Dependency-Pinning Scoring is taking into account all ecosystems, even if there is no evidence for them in a repository.
For instance, if a repo only contains a single shell script with a single line "pip install -r requirements.txt", the pinned-dependency score for that repo will by 8/10.
To get to a score of 0/10, means that all ecosystems that are known by scorecard exist in the repo, at the same time, and all checks for unpinned dependencies for those ecosystems report for unpinned dependencies. That seems like an edge case, at best.
Reproduction steps
Steps to reproduce the behavior:
- Have a repo with only one ecosystem that is checked and scored by the unpinned dependency check (at the time of opening this issue the supported checks are: docker files, docker downloads, pip or scripts download, PR 🐛 Add npm installs to Pinned-Dependencies score #2960 by @gabibguti adds npm support)
- run Scorecard with --checks=Pinned-Dependencies and the repo name
- The result is 8/10.
Expected behavior
It is expected that the scoring is normalized only for the ecosystems that are actually found in the repository. so that a repository is only measured on the technology stacks that the repo actually implements.
Examples:
- A repository that uses python. pip install commands are unpinned: expected result is 0/10
- A repository that uses python. pip install commands are pinned: expected result is 10/10
- A repository that uses python and docker. pip install commands are pinned (i.e. use --require-hashes) while docker base images are not pinned: expected result is 5/10
Additional context
The root cause seems to be that raw dependency checks do not differentiate between not finding unpinned dependencies for an ecosystem (score 10), and not finding any evidence of using dependencies of the ecosystem the check is for.