Skip to content

admin/coverage: only measure core crate coverage #2377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 12, 2025
Merged

Conversation

ctz
Copy link
Member

@ctz ctz commented Mar 12, 2025

At the minute the "headline" coverage number includes the core crate, the bogo test runner, and the provider-example example code:

image

This PR removes two of those components.

Copy link

codecov bot commented Mar 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.16%. Comparing base (94d4cb7) to head (8a95eb5).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2377      +/-   ##
==========================================
+ Coverage   94.91%   96.16%   +1.24%     
==========================================
  Files         103       94       -9     
  Lines       24676    22815    -1861     
==========================================
- Hits        23421    21939    -1482     
+ Misses       1255      876     -379     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link

rustls-benchmarking bot commented Mar 12, 2025

Benchmark results

Instruction counts

Significant differences

There are no significant instruction count differences

Other differences

Click to expand
Scenario Baseline Candidate Diff Threshold
handshake_no_resume_aws_lc_rs_1.2_rsa_aes_server 10436001 10509500 73499 (0.70%) 0.93%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_client 8287637 8319329 31692 (0.38%) 1.50%
handshake_no_resume_aws_lc_rs_1.3_rsa_chacha_server 10736448 10759041 22593 (0.21%) 1.88%
handshake_no_resume_aws_lc_rs_1.3_rsa_aes_server 10765331 10743383 -21948 (-0.20%) 1.22%
handshake_no_resume_ring_1.3_ecdsap256_chacha_client 3303038 3305266 2228 (0.07%) 0.40%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_aes_client 8316152 8319700 3548 (0.04%) 1.36%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_aes_client 3097485 3096917 -568 (-0.02%) 0.70%
handshake_no_resume_ring_1.3_ecdsap256_aes_client 3303911 3304006 95 (0.00%) 0.24%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_client 3104305 3104348 43 (0.00%) 0.57%
handshake_no_resume_aws_lc_rs_1.2_rsa_aes_client 1719418 1719399 -19 (-0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_aes_server 2061512 2061531 19 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_chacha_client 27815323 27815080 -243 (-0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_rsa_aes_client 1927404 1927419 15 (0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_rsa_chacha_client 1934150 1934165 15 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap256_chacha_server 1297105 1297095 -10 (-0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_chacha_server 28741022 28740848 -174 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_chacha_client 28218666 28218822 156 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_aes_client 27874613 27874460 -153 (-0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_aes_client 27876838 27876970 132 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_chacha_server 28743951 28744087 136 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_aes_server 28819381 28819255 -126 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_aes_server 30259168 30259038 -130 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_aes_client 28241664 28241778 114 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_aes_client 28244517 28244411 -106 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_chacha_server 30219426 30219529 103 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_chacha_client 28214988 28214893 -95 (-0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_chacha_client 27817568 27817642 74 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_chacha_client 28211869 28211798 -71 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_aes_server 30262174 30262246 72 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_chacha_client 27822163 27822228 65 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap256_aes_server 1295930 1295927 -3 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_aes_server 30262031 30262101 70 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_chacha_server 7229513 7229521 8 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_chacha_server 30216540 30216511 -29 (-0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_aes_client 27881438 27881415 -23 (-0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_chacha_client 34743507 34743530 23 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_chacha_server 28743803 28743822 19 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_aes_server 28822276 28822260 -16 (-0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_aes_server 28822437 28822446 9 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_chacha_server 30219283 30219274 -9 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_aes_server 46302456 46302466 10 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_chacha_server 80553336 80553346 10 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_server 80650245 80650254 9 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_client 92697227 92697218 -9 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_chacha_client 92668570 92668562 -8 (-0.00%) 0.20%
transfer_no_resume_ring_1.2_rsa_aes_client 58021707 58021702 -5 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_client 92694982 92694989 7 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.2_rsa_aes_client 58163475 58163478 3 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_aes_client 58236035 58236038 3 (0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_aes_server 46292201 46292203 2 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_aes_server 46465931 46465929 -2 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_server 80642629 80642632 3 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_aes_client 58120623 58120621 -2 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_chacha_server 80548545 80548547 2 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_chacha_server 80645588 80645586 -2 (-0.00%) 0.20%
transfer_no_resume_ring_1.2_rsa_aes_server 46198576 46198577 1 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_aes_server 46298937 46298938 1 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.2_rsa_aes_server 46407682 46407681 -1 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_aes_server 46461267 46461268 1 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_aes_client 58127835 58127834 -1 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_aes_client 58131604 58131603 -1 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_chacha_server 80541812 80541811 -1 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_chacha_client 92657591 92657592 1 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_chacha_client 92666719 92666720 1 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_chacha_client 92701315 92701314 -1 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_aes_client 58231629 58231629 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_aes_client 30950337 30950337 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_aes_server 32118600 32118600 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_chacha_client 30866152 30866152 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_aes_server 11128487 11128487 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_chacha_client 31184720 31184720 0 (0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_aes_server 1144808 1144808 0 (0.00%) 1.57%
handshake_no_resume_ring_1.2_rsa_aes_client 2245956 2245956 0 (0.00%) 0.20%
handshake_session_id_ring_1.2_rsa_aes_client 4327852 4327852 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_aes_server 32118529 32118529 0 (0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_server 2064554 2064554 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_chacha_client 2339228 2339228 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_chacha_client 31177373 31177373 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_chacha_client 30862056 30862056 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_chacha_server 31996830 31996830 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_aes_client 2333578 2333578 0 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.2_rsa_aes_server 5016022 5016022 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_chacha_client 30859107 30859107 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_chacha_server 31993720 31993720 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_aes_server 7227558 7227558 0 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.2_rsa_aes_server 3868725 3868725 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_aes_server 32564266 32564266 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_chacha_server 32459155 32459155 0 (0.00%) 0.20%
handshake_tickets_ring_1.2_rsa_aes_client 4590973 4590973 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_chacha_client 31180772 31180772 0 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_aes_client 28248510 28248510 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_chacha_server 32462232 32462232 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_aes_client 30953286 30953286 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_aes_server 32115490 32115490 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_chacha_server 32462236 32462236 0 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_aes_client 58233862 58233862 0 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_aes_server 46458944 46458944 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_aes_client 31255931 31255931 0 (0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_server 1151272 1151272 0 (0.00%) 1.54%
handshake_tickets_ring_1.3_ecdsap256_aes_server 32564307 32564307 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_aes_client 31251983 31251983 0 (0.00%) 0.20%
handshake_tickets_ring_1.2_rsa_aes_server 4765952 4765952 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_aes_client 34741792 34741792 0 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.2_rsa_aes_client 3896601 3896601 0 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.2_rsa_aes_client 4239222 4239222 0 (0.00%) 0.20%
handshake_no_resume_ring_1.2_rsa_aes_server 11000087 11000087 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_aes_server 32561230 32561230 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_chacha_server 11134244 11134244 0 (0.00%) 0.20%
handshake_session_id_ring_1.2_rsa_aes_server 4301258 4301258 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_aes_client 30957382 30957382 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_aes_client 31248563 31248563 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_chacha_server 31996759 31996759 0 (0.00%) 0.20%

Wall-time

Significant differences

There are no significant wall-time differences

Other differences

Click to expand
Scenario Baseline Candidate Diff Threshold
transfer_no_resume_aws_lc_rs_1.3_rsa_aes 5.24 ms 5.27 ms 0.04 ms (0.69%) 5.39%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_aes 4.53 ms 4.56 ms 0.03 ms (0.67%) 6.57%
transfer_no_resume_ring_1.3_rsa_aes 6.00 ms 6.03 ms 0.04 ms (0.63%) 4.37%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_aes 5.21 ms 5.25 ms 0.03 ms (0.63%) 6.30%
transfer_no_resume_ring_1.3_ecdsap256_aes 5.50 ms 5.53 ms 0.03 ms (0.61%) 5.01%
handshake_no_resume_aws_lc_rs_1.3_rsa_aes 1.15 ms 1.16 ms 0.01 ms (0.60%) 2.48%
handshake_no_resume_aws_lc_rs_1.2_rsa_aes 1.13 ms 1.13 ms 0.01 ms (0.56%) 3.71%
transfer_no_resume_aws_lc_rs_1.2_rsa_aes 5.17 ms 5.19 ms 0.02 ms (0.44%) 5.28%
transfer_no_resume_ring_1.2_rsa_aes 5.91 ms 5.93 ms 0.02 ms (0.41%) 4.24%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_chacha 4.31 ms 4.29 ms -0.02 ms (-0.37%) 1.90%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_aes 4.54 ms 4.53 ms -0.02 ms (-0.36%) 1.60%
handshake_no_resume_ring_1.3_ecdsap256_chacha 476.04 µs 477.75 µs 1.71 µs (0.36%) 4.00%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_aes 5.03 ms 5.01 ms -0.02 ms (-0.36%) 1.28%
transfer_no_resume_ring_1.3_ecdsap384_aes 8.62 ms 8.65 ms 0.03 ms (0.35%) 3.14%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_chacha 4.98 ms 4.96 ms -0.02 ms (-0.33%) 1.42%
handshake_no_resume_ring_1.3_rsa_chacha 978.17 µs 981.03 µs 2.86 µs (0.29%) 1.34%
transfer_no_resume_ring_1.3_rsa_chacha 13.50 ms 13.53 ms 0.04 ms (0.28%) 2.06%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_chacha 13.00 ms 13.03 ms 0.04 ms (0.28%) 2.29%
handshake_tickets_aws_lc_rs_1.2_rsa_aes 1.86 ms 1.87 ms 0.01 ms (0.28%) 2.27%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_aes 459.37 µs 460.62 µs 1.25 µs (0.27%) 3.60%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_aes 4.34 ms 4.33 ms -0.01 ms (-0.27%) 1.90%
handshake_session_id_aws_lc_rs_1.3_rsa_chacha 5.02 ms 5.00 ms -0.01 ms (-0.27%) 1.00%
transfer_no_resume_ring_1.3_ecdsap256_chacha 13.00 ms 13.04 ms 0.03 ms (0.27%) 2.17%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_chacha 13.68 ms 13.71 ms 0.04 ms (0.27%) 2.00%
handshake_tickets_ring_1.3_ecdsap256_chacha 5.59 ms 5.58 ms -0.01 ms (-0.26%) 1.00%
handshake_session_id_aws_lc_rs_1.3_rsa_aes 5.05 ms 5.04 ms -0.01 ms (-0.25%) 1.11%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_chacha 457.47 µs 458.62 µs 1.15 µs (0.25%) 4.05%
handshake_no_resume_ring_1.3_rsa_aes 979.26 µs 981.60 µs 2.34 µs (0.24%) 1.24%
handshake_no_resume_aws_lc_rs_1.3_rsa_chacha 1.14 ms 1.15 ms 0.00 ms (0.24%) 1.25%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_chacha 5.18 ms 5.17 ms -0.01 ms (-0.23%) 1.42%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_chacha 4.51 ms 4.49 ms -0.01 ms (-0.23%) 1.80%
transfer_no_resume_aws_lc_rs_1.3_rsa_chacha 13.71 ms 13.74 ms 0.03 ms (0.22%) 1.80%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_aes 1.14 ms 1.15 ms 0.00 ms (0.22%) 1.38%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_aes 5.23 ms 5.21 ms -0.01 ms (-0.21%) 1.29%
transfer_no_resume_ring_1.3_ecdsap384_chacha 16.13 ms 16.16 ms 0.03 ms (0.19%) 1.75%
handshake_tickets_ring_1.3_ecdsap256_aes 5.80 ms 5.79 ms -0.01 ms (-0.18%) 1.00%
handshake_tickets_ring_1.3_rsa_chacha 6.08 ms 6.07 ms -0.01 ms (-0.17%) 1.00%
handshake_no_resume_ring_1.3_ecdsap256_aes 482.64 µs 483.43 µs 0.80 µs (0.17%) 3.45%
handshake_session_id_ring_1.3_ecdsap256_chacha 5.52 ms 5.51 ms -0.01 ms (-0.15%) 1.00%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_chacha 1.13 ms 1.14 ms 0.00 ms (0.13%) 1.36%
handshake_tickets_aws_lc_rs_1.3_rsa_aes 5.25 ms 5.24 ms -0.01 ms (-0.12%) 1.09%
handshake_tickets_ring_1.3_rsa_aes 6.30 ms 6.29 ms -0.01 ms (-0.11%) 1.00%
handshake_tickets_ring_1.3_ecdsap384_chacha 8.70 ms 8.70 ms -0.01 ms (-0.11%) 1.00%
handshake_session_id_ring_1.3_ecdsap384_chacha 8.63 ms 8.62 ms -0.01 ms (-0.11%) 1.00%
handshake_tickets_aws_lc_rs_1.3_rsa_chacha 5.22 ms 5.22 ms -0.01 ms (-0.10%) 1.20%
handshake_session_id_ring_1.3_rsa_chacha 6.01 ms 6.00 ms -0.01 ms (-0.10%) 1.00%
handshake_session_id_ring_1.3_ecdsap256_aes 5.88 ms 5.88 ms -0.01 ms (-0.10%) 1.00%
handshake_session_id_ring_1.2_rsa_aes 1.58 ms 1.58 ms -0.00 ms (-0.09%) 1.23%
handshake_no_resume_ring_1.3_ecdsap384_aes 3.60 ms 3.60 ms 0.00 ms (0.08%) 1.00%
handshake_no_resume_ring_1.2_rsa_aes 967.85 µs 968.47 µs 0.62 µs (0.06%) 1.56%
handshake_tickets_ring_1.3_ecdsap384_aes 8.91 ms 8.91 ms -0.01 ms (-0.06%) 1.00%
handshake_session_id_ring_1.3_ecdsap384_aes 8.99 ms 8.98 ms -0.01 ms (-0.06%) 1.00%
handshake_session_id_ring_1.3_rsa_aes 6.37 ms 6.37 ms -0.00 ms (-0.05%) 1.00%
handshake_tickets_ring_1.2_rsa_aes 1.66 ms 1.66 ms -0.00 ms (-0.05%) 1.54%
handshake_session_id_aws_lc_rs_1.2_rsa_aes 1.70 ms 1.70 ms 0.00 ms (0.03%) 2.36%
handshake_no_resume_ring_1.3_ecdsap384_chacha 3.60 ms 3.60 ms 0.00 ms (0.01%) 1.00%

Additional information

Historical results

Checkout details:

Copy link
Member

@cpu cpu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea 👍

@ctz ctz enabled auto-merge March 12, 2025 17:51
@ctz ctz added this pull request to the merge queue Mar 12, 2025
Merged via the queue into main with commit 7a64c74 Mar 12, 2025
62 checks passed
@ctz ctz deleted the jbp-coverage-adjustments branch March 12, 2025 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants