Skip to content

Coverage failure on 3.10.0-rc.2 #478

@FollowTheProcess

Description

@FollowTheProcess

Describe the bug

See #477 for context. But short summary is that the python 3.10 CI runs I introduced in #475 actually weren't doing anything due to the difference between the version identifier 3.10.0-rc.2 and the actual interpreter path on the GHA runner. So nox just ignored it citing a missing interpreter (correct behavior) which made it appear like it was passing.

This was an easy fix to the Github Actions workflow but now I have a weird issue with coverage.

When running both locally and on my fork here: https://github.com/FollowTheProcess/nox/runs/3613749834?check_suite_focus=true the coverage is failing as it's less than 100%, which is super weird.

This obviously doesn't happen on any other version.

How to reproduce

Run nox -s test-3.10 cover locally in a 3.10 environment or on Github Actions as shown in my fork above.

Expected behavior

Coverage should be identical between python versions

Now it looks to me like this could be an upstream thing, there a few issues on coverage that reference 3.10 results being different (e.g. nedbat/coveragepy#1106 and nedbat/coveragepy#1187)

I also tried running it using the most up to date master branch from coverage but the results were the same

My question is what we should do with the 3.10.0-rc.2 runs. I see 3 potential ways forward:

  • Temporarily drop 3.10 runs from Github Actions until a later date (perhaps when 3.10 is officially released)
  • Add a condition in the cover nox session that if python is 3.10 just return (similar to the ON_WINDOWS_CI check, this way we're still testing against 3.10 just not testing coverage)
  • Add a similar condition that lowers the coverage requirement for 3.10 to 99% (the value I'm getting when running on 3.10, this way we're at least measuring coverage)

I'm happy with any approach, just wanted to get some input on how we want to handle this 🙂

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciIssues relating to nox's CI pipeline

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions