Skip to content

feat(test): create coverage reports when --coverage specified in deno test #28260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Apr 18, 2025
Merged

feat(test): create coverage reports when --coverage specified in deno test #28260

merged 12 commits into from
Apr 18, 2025

Conversation

kt3k
Copy link
Member

@kt3k kt3k commented Feb 23, 2025 •
edited
Loading

part of #21325
closes #28218
closes #25219

This PR updates the behavior of deno test --coverage option. Now if --coverage option is specified, deno test command automatically shows summary report in the terminal, and generates the lcov report in $coverage_dir/lcov.info and html report in $coverage_dir/html/

This change also adds --coverage-raw-data-only flag, which prevents the above reports generated, instead only generates the raw json coverage data (which is the same as current behavior)

An example of execution looks like:

$ deno test --coverage
running 41 tests from ./yaml/parse_test.ts
parse() handles single document yaml string ... ok (2ms)
parseAll() handles yaml string with multiple documents ... ok (0ms)
parse() throws with `!!js/*` yaml types with default schemas ... ok (0ms)

... snip ...

stringify() handles undefined object entry with skipInvalid and flowLevel option ... ok (0ms)
stringify() handles undefined object entry with big key and skipInvalid and flowLevel option ... ok (0ms)
stringify() handles duplicate binary references ... ok (0ms)

ok | 83 passed | 0 failed (442ms)

-----------------------------------------------------
File                            | Branch % | Line % |
-----------------------------------------------------
 assert/almost_equals.ts        |    100.0 |    4.5 |
 assert/array_includes.ts       |    100.0 |   10.3 |
 assert/assert.ts               |      0.0 |   50.0 |

... snip ...

 yaml/_utils.ts                 |    100.0 |  100.0 |
 yaml/parse.ts                  |    100.0 |  100.0 |
 yaml/stringify.ts              |    100.0 |  100.0 |
-----------------------------------------------------
 All files                      |     86.9 |   54.1 |
-----------------------------------------------------
Lcov coverage report has been generated at file:///Users/kt3k/denoland/std/coverage/lcov.info
HTML coverage report has been generated at file:///Users/kt3k/denoland/std/coverage/html/index.html

Note: jest --coverage outputs 'summary' and 'lcov' reports by default. This outputs 'html' reports in addition to them.

Planning to land for Deno 2.3

@kt3k kt3k added the ci-draft Run the CI on draft PRs. label Feb 23, 2025
@kt3k kt3k marked this pull request as ready for review February 25, 2025 00:47
@bartlomieju bartlomieju added this to the 2.3.0 milestone Feb 25, 2025
bartlomieju
bartlomieju reviewed Feb 25, 2025
View reviewed changes
Copy link
Member

@bartlomieju bartlomieju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good idea 👍

cli/tools/coverage/mod.rs
) -> Result<(), AnyError> {
if coverage_flags.files.include.is_empty() {
if files_include.is_empty() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this function should remove previously generated coverage before dumping new reports to the directory. Otherwise the results might be skewed from previous runs. WDYT?

Copy link
Member Author

@kt3k kt3k Feb 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. Now deno test --coverage clears the reports & raw data from the previous runs. It still keeps the contents of coverage_dir when --coverage-raw-data-only specified (and --clean not specified) to support the advanced scenarios.

cli/tools/coverage/reporter.rs Outdated
_coverage_root: &Path,
file_reports: &[(CoverageReport, String)],
) {
file_reports.iter().for_each(|(report, file_text)| {
self.report(report, file_text).unwrap();
});
if let Some((report, _)) = file_reports.first() {
if let Some(ref output) = report.output {
let path = output.canonicalize().unwrap().to_string_lossy().to_string();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unwrap is sus here 😬 maybe consider failing gracefully with an error message? Also you don't need to do the conversion to string first - Url::from_file_path will accept a PathBuf

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. Now it prints error message when .canonicalize failed

@kt3k kt3k removed the ci-draft Run the CI on draft PRs. label Mar 5, 2025
@kt3k kt3k requested a review from bartlomieju April 16, 2025 09:39
@kt3k
Copy link
Member Author

kt3k commented Apr 18, 2025

@bartlomieju PTAL when you have time

bartlomieju
bartlomieju approved these changes Apr 18, 2025
View reviewed changes
Copy link
Member

@bartlomieju bartlomieju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Could you please write a short description how one would migrate from using --coverage in Deno pre-2.3 vs using it in Deno 2.3? Based on that I'll write the blog post and docs. Thanks!

@kt3k kt3k merged commit 83f15ec into denoland:main Apr 18, 2025
18 checks passed
@kt3k
Copy link
Member Author

kt3k commented Apr 18, 2025 •
edited
Loading

@bartlomieju I drafted the migration guide below. What do you think?

Migration Guide

Now deno test --coverage automatically generates lcov and html reports by default.

If your current test setup looks something like the below:

deno test --coverage
deno coverage --lcov --output=lcov.info
deno coverage --html

What you need now is only:

deno test --coverage

Note

If you prefer to opt-out from the automatic generation of reports, you can specify --> coverage-raw-data-only to keep the existing behavior. This might be useful if you use advanced options of deno coverage command.

deno test --coverage --coverage-raw-data-only

@kt3k kt3k deleted the feat-test-report-coverage branch April 18, 2025 09:56
@bartlomieju
Copy link
Member

Great, thanks! I'll use it

@csvn
Copy link
Contributor

csvn commented Apr 30, 2025

It looks like this feature was included in the v2.3.0 release, but I cannot see it included in the release notes v2.3.0 release notes. Is there a bug in the generation of commits/PR's?

I was surprised, since I noticed these changes when our CI ran, but couldn't remember any mentions 😄

@kt3k
Copy link
Member Author

kt3k commented May 1, 2025

hmm it looks like the release notes for 2.3.0 only captures the items between 2.3.0-rc.3 and 2.3.0.

@kt3k
Copy link
Member Author

kt3k commented May 1, 2025

Trying to fix it in #29122

@kt3k
Copy link
Member Author

kt3k commented May 1, 2025

@csvn Thanks for catching this. We fixed the GitHub's release page https://github.com/denoland/deno/releases/tag/v2.3.0

@csvn
Copy link
Contributor

csvn commented May 2, 2025

@kt3k No problem, glad you found the issue and fixed it 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bartlomieju bartlomieju bartlomieju approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.3.0
Development

Successfully merging this pull request may close these issues.

Merge deno coverage to deno test --coverage with 3 default outputs DX: deno test --coverage doesn't show coverage
3 participants
@kt3k @bartlomieju @csvn