Skip to content

Upgrade to criterion 0.6 #2464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 20, 2025
Merged

Upgrade to criterion 0.6 #2464

merged 3 commits into from
May 20, 2025

Conversation

djc
Copy link
Member

@djc djc commented May 20, 2025

No description provided.

Copy link

rustls-benchmarking bot commented May 20, 2025

Benchmark results

Instruction counts

Significant differences

⚠️ There are significant instruction count differences

Click to expand
Scenario Baseline Candidate Diff Threshold
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_aes_server 2916873 2910160 -6713 (-0.23%) 0.20%

Other differences

Click to expand
Scenario Baseline Candidate Diff Threshold
handshake_no_resume_aws_lc_rs_1.2_rsa_aes_server 10493102 10442544 -50558 (-0.48%) 1.01%
handshake_no_resume_aws_lc_rs_1.3_rsa_chacha_server 11624574 11590342 -34232 (-0.29%) 1.61%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_client 9971647 9955097 -16550 (-0.17%) 0.68%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_aes_client 9973313 9985121 11808 (0.12%) 0.68%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_aes_client 78487191 78545797 58606 (0.07%) 0.20%
handshake_no_resume_ring_1.3_ecdsap256_chacha_client 3305475 3303519 -1956 (-0.06%) 0.27%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_aes_server 55286249 55315850 29601 (0.05%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_chacha_client 78685598 78702873 17275 (0.02%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_aes_client 4746581 4747601 1020 (0.02%) 0.27%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_aes_server 2000629 2001046 417 (0.02%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_client 4750104 4749162 -942 (-0.02%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_aes_client 78518998 78503463 -15535 (-0.02%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_chacha_client 78477038 78462823 -14215 (-0.02%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_rsa_chacha_client 3586687 3586091 -596 (-0.02%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_chacha_server 56561680 56570110 8430 (0.01%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_aes_server 55298976 55291295 -7681 (-0.01%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_chacha_server 55232891 55225634 -7257 (-0.01%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_aes_client 78698616 78707704 9088 (0.01%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_server 2003472 2003288 -184 (-0.01%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_server 2913416 2913156 -260 (-0.01%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_chacha_client 78685651 78692639 6988 (0.01%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_aes_server 56594139 56598683 4544 (0.01%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_aes_client 78502805 78508756 5951 (0.01%) 0.20%
handshake_session_id_aws_lc_rs_1.2_rsa_aes_client 3876537 3876267 -270 (-0.01%) 0.20%
handshake_tickets_aws_lc_rs_1.2_rsa_aes_client 4218640 4218370 -270 (-0.01%) 0.20%
handshake_session_id_ring_1.2_rsa_aes_client 4306434 4306164 -270 (-0.01%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_rsa_aes_server 11587436 11588156 720 (0.01%) 1.14%
handshake_tickets_ring_1.2_rsa_aes_client 4569978 4569708 -270 (-0.01%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_chacha_server 56560105 56563397 3292 (0.01%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_aes_server 55284373 55287065 2692 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_chacha_client 78428161 78431208 3047 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap256_aes_client 3304254 3304142 -112 (-0.00%) 0.22%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_chacha_server 55205090 55206660 1570 (0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.3_rsa_aes_client 3579939 3580035 96 (0.00%) 1.31%
handshake_session_id_aws_lc_rs_1.3_rsa_chacha_client 78441402 78439351 -2051 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_aes_client 78696251 78697650 1399 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_chacha_server 56569158 56570158 1000 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_aes_server 56594476 56595434 958 (0.00%) 0.20%
handshake_no_resume_aws_lc_rs_1.2_rsa_aes_client 1719767 1719743 -24 (-0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.3_rsa_chacha_server 55203497 55202735 -762 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_aes_client 78711891 78711156 -735 (-0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_chacha_client 78720748 78720107 -641 (-0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap256_chacha_server 1296656 1296666 10 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.3_rsa_aes_server 56592330 56591972 -358 (-0.00%) 0.20%
handshake_no_resume_ring_1.2_rsa_aes_client 2246303 2246298 -5 (-0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_aes_client 2334062 2334066 4 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_chacha_client 2339711 2339715 4 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_aes_client 58226794 58226779 -15 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_aes_server 46284279 46284288 9 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_aes_server 46291015 46291024 9 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_aes_client 34742294 34742300 6 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_chacha_client 34744033 34744039 6 (0.00%) 0.20%
transfer_no_resume_ring_1.2_rsa_aes_client 58035384 58035394 10 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_server 80634519 80634530 11 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_server 80645577 80645566 -11 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_aes_server 46450200 46450206 6 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_aes_server 46461255 46461249 -6 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_chacha_client 92653697 92653687 -10 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_chacha_server 80637673 80637665 -8 (-0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_chacha_client 92717530 92717521 -9 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_aes_client 58117366 58117371 5 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_aes_server 46452705 46452708 3 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_chacha_server 80545416 80545421 5 (0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_aes_client 58148973 58148976 3 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_aes_client 58227324 58227327 3 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_rsa_aes_client 58251608 58251611 3 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_chacha_client 92691424 92691428 4 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.2_rsa_aes_server 46397779 46397781 2 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.2_rsa_aes_client 58157302 58157304 2 (0.00%) 0.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_chacha_client 92691969 92691972 3 (0.00%) 0.20%
transfer_no_resume_ring_1.2_rsa_aes_server 46189315 46189314 -1 (-0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_chacha_client 92662826 92662828 2 (0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_chacha_server 80533887 80533888 1 (0.00%) 0.20%
transfer_no_resume_ring_1.3_rsa_chacha_client 92685297 92685298 1 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_aes_client 31231793 31231793 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_aes_client 30929086 30929086 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_aes_server 32112593 32112593 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_aes_server 32112394 32112394 0 (0.00%) 0.20%
handshake_no_resume_ring_1.2_rsa_aes_server 11000039 11000039 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_aes_client 30926341 30926341 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_chacha_server 32451180 32451180 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_aes_server 32553048 32553048 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_aes_client 31224515 31224515 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_aes_server 32109433 32109433 0 (0.00%) 0.20%
handshake_tickets_ring_1.2_rsa_aes_server 4757720 4757720 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_aes_server 11128086 11128086 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_chacha_server 32451138 32451138 0 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_aes_server 46294544 46294544 0 (0.00%) 0.20%
handshake_session_id_aws_lc_rs_1.2_rsa_aes_server 3867855 3867855 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_chacha_server 7229084 7229084 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_aes_server 32553135 32553135 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap256_aes_server 1295484 1295484 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_chacha_client 30841974 30841974 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_chacha_client 31156472 31156472 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_chacha_server 31990943 31990943 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_chacha_server 32448134 32448134 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap256_aes_client 31227743 31227743 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_ecdsap384_aes_server 7227117 7227117 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_aes_client 30933264 30933264 0 (0.00%) 0.20%
handshake_session_id_ring_1.2_rsa_aes_server 4298144 4298144 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_aes_server 32550089 32550089 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_rsa_chacha_client 31160522 31160522 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_rsa_chacha_server 31987783 31987783 0 (0.00%) 0.20%
handshake_no_resume_ring_1.3_rsa_chacha_server 11133847 11133847 0 (0.00%) 0.20%
handshake_tickets_ring_1.3_ecdsap384_chacha_client 31153265 31153265 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_chacha_client 30837796 30837796 0 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap256_chacha_server 80540623 80540623 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap384_chacha_client 30835051 30835051 0 (0.00%) 0.20%
handshake_session_id_ring_1.3_ecdsap256_chacha_server 31990744 31990744 0 (0.00%) 0.20%
transfer_no_resume_ring_1.3_ecdsap384_aes_client 58124573 58124573 0 (0.00%) 0.20%
handshake_tickets_aws_lc_rs_1.2_rsa_aes_server 5012134 5012134 0 (0.00%) 0.20%

Wall-time

Significant differences

⚠️ There are significant wall-time differences

Click to expand
Scenario Baseline Candidate Diff Threshold
handshake_no_resume_aws_lc_rs_1.3_rsa_aes 1.34 ms 1.30 ms ✅ -0.04 ms (-3.20%) 2.91%
handshake_no_resume_aws_lc_rs_1.3_rsa_chacha 1.33 ms 1.30 ms ✅ -0.04 ms (-2.94%) 2.82%
handshake_session_id_aws_lc_rs_1.3_rsa_chacha 11.30 ms 11.11 ms ✅ -0.19 ms (-1.70%) 1.31%
handshake_tickets_aws_lc_rs_1.3_rsa_chacha 11.49 ms 11.30 ms ✅ -0.19 ms (-1.61%) 1.40%
handshake_session_id_aws_lc_rs_1.3_rsa_aes 11.33 ms 11.16 ms ✅ -0.17 ms (-1.52%) 1.19%

Other differences

Click to expand
Scenario Baseline Candidate Diff Threshold
handshake_no_resume_aws_lc_rs_1.2_rsa_aes 1.10 ms 1.07 ms -0.03 ms (-3.04%) 3.61%
handshake_tickets_aws_lc_rs_1.2_rsa_aes 1.82 ms 1.79 ms -0.03 ms (-1.81%) 4.45%
handshake_session_id_aws_lc_rs_1.2_rsa_aes 1.64 ms 1.61 ms -0.03 ms (-1.75%) 3.33%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_chacha 10.62 ms 10.46 ms -0.16 ms (-1.51%) 1.54%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_chacha 11.29 ms 11.13 ms -0.16 ms (-1.43%) 1.69%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_chacha 10.81 ms 10.66 ms -0.15 ms (-1.40%) 1.75%
handshake_tickets_aws_lc_rs_1.3_rsa_aes 11.49 ms 11.33 ms -0.16 ms (-1.36%) 1.39%
handshake_session_id_aws_lc_rs_1.3_ecdsap256_aes 10.64 ms 10.50 ms -0.14 ms (-1.31%) 1.57%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_chacha 11.48 ms 11.33 ms -0.15 ms (-1.30%) 1.44%
handshake_session_id_aws_lc_rs_1.3_ecdsap384_aes 11.32 ms 11.18 ms -0.14 ms (-1.26%) 1.28%
handshake_tickets_aws_lc_rs_1.3_ecdsap256_aes 10.82 ms 10.69 ms -0.13 ms (-1.17%) 1.44%
handshake_tickets_aws_lc_rs_1.3_ecdsap384_aes 11.50 ms 11.38 ms -0.12 ms (-1.07%) 1.53%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_chacha 669.07 µs 664.93 µs -4.14 µs (-0.62%) 4.01%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_aes 4.73 ms 4.76 ms 0.03 ms (0.57%) 5.65%
transfer_no_resume_ring_1.3_ecdsap256_aes 5.43 ms 5.46 ms 0.03 ms (0.55%) 4.79%
handshake_no_resume_aws_lc_rs_1.3_ecdsap256_aes 670.51 µs 667.15 µs -3.36 µs (-0.50%) 4.37%
transfer_no_resume_aws_lc_rs_1.2_rsa_aes 5.14 ms 5.11 ms -0.02 ms (-0.46%) 6.20%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_aes 5.43 ms 5.45 ms 0.02 ms (0.39%) 5.02%
handshake_tickets_ring_1.3_ecdsap256_aes 5.64 ms 5.62 ms -0.02 ms (-0.37%) 1.00%
handshake_tickets_ring_1.3_rsa_aes 6.13 ms 6.11 ms -0.02 ms (-0.37%) 1.06%
handshake_session_id_ring_1.3_rsa_aes 6.06 ms 6.04 ms -0.02 ms (-0.28%) 1.35%
transfer_no_resume_ring_1.3_rsa_aes 5.93 ms 5.94 ms 0.02 ms (0.27%) 4.58%
transfer_no_resume_ring_1.3_ecdsap384_aes 8.53 ms 8.55 ms 0.02 ms (0.27%) 2.85%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_aes 1.35 ms 1.35 ms -0.00 ms (-0.27%) 1.46%
handshake_no_resume_ring_1.3_ecdsap256_chacha 479.06 µs 477.78 µs -1.28 µs (-0.27%) 3.33%
handshake_session_id_ring_1.3_ecdsap256_aes 5.56 ms 5.55 ms -0.01 ms (-0.24%) 1.45%
transfer_no_resume_ring_1.2_rsa_aes 5.86 ms 5.87 ms 0.01 ms (0.24%) 5.30%
handshake_tickets_ring_1.3_ecdsap256_chacha 5.59 ms 5.58 ms -0.01 ms (-0.24%) 1.00%
handshake_tickets_ring_1.3_rsa_chacha 6.08 ms 6.06 ms -0.01 ms (-0.22%) 1.00%
transfer_no_resume_aws_lc_rs_1.3_rsa_aes 5.40 ms 5.39 ms -0.01 ms (-0.22%) 5.32%
handshake_no_resume_ring_1.2_rsa_aes 964.82 µs 966.84 µs 2.02 µs (0.21%) 1.27%
handshake_tickets_ring_1.3_ecdsap384_aes 8.73 ms 8.71 ms -0.02 ms (-0.21%) 1.00%
transfer_no_resume_aws_lc_rs_1.3_rsa_chacha 13.89 ms 13.86 ms -0.03 ms (-0.20%) 2.12%
handshake_tickets_ring_1.2_rsa_aes 1.60 ms 1.60 ms -0.00 ms (-0.19%) 1.40%
transfer_no_resume_ring_1.3_rsa_chacha 13.48 ms 13.51 ms 0.02 ms (0.18%) 2.07%
handshake_no_resume_ring_1.3_rsa_aes 968.46 µs 969.67 µs 1.20 µs (0.12%) 1.48%
transfer_no_resume_ring_1.3_ecdsap256_chacha 13.00 ms 13.01 ms 0.02 ms (0.12%) 1.96%
handshake_no_resume_aws_lc_rs_1.3_ecdsap384_chacha 1.34 ms 1.34 ms -0.00 ms (-0.12%) 1.39%
handshake_session_id_ring_1.3_rsa_chacha 6.01 ms 6.01 ms -0.01 ms (-0.12%) 1.00%
handshake_session_id_ring_1.3_ecdsap384_aes 8.65 ms 8.64 ms -0.01 ms (-0.11%) 1.00%
transfer_no_resume_aws_lc_rs_1.3_ecdsap384_chacha 13.89 ms 13.91 ms 0.02 ms (0.11%) 1.53%
transfer_no_resume_ring_1.3_ecdsap384_chacha 16.09 ms 16.11 ms 0.02 ms (0.11%) 1.46%
handshake_tickets_ring_1.3_ecdsap384_chacha 8.69 ms 8.68 ms -0.01 ms (-0.10%) 1.00%
handshake_session_id_ring_1.3_ecdsap256_chacha 5.52 ms 5.51 ms -0.01 ms (-0.10%) 1.00%
handshake_no_resume_ring_1.3_ecdsap384_chacha 3.58 ms 3.58 ms -0.00 ms (-0.10%) 1.00%
transfer_no_resume_aws_lc_rs_1.3_ecdsap256_chacha 13.21 ms 13.23 ms 0.01 ms (0.09%) 1.75%
handshake_session_id_ring_1.2_rsa_aes 1.52 ms 1.52 ms -0.00 ms (-0.07%) 2.46%
handshake_no_resume_ring_1.3_ecdsap256_aes 481.16 µs 480.83 µs -0.33 µs (-0.07%) 3.29%
handshake_no_resume_ring_1.3_ecdsap384_aes 3.58 ms 3.58 ms -0.00 ms (-0.06%) 1.00%
handshake_no_resume_ring_1.3_rsa_chacha 969.83 µs 969.29 µs -0.55 µs (-0.06%) 1.38%
handshake_session_id_ring_1.3_ecdsap384_chacha 8.62 ms 8.61 ms -0.00 ms (-0.04%) 1.00%

Additional information

Historical results

Checkout details:

Copy link

codecov bot commented May 20, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.26%. Comparing base (b30fc9e) to head (ce60776).
Report is 18 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2464      +/-   ##
==========================================
- Coverage   95.26%   95.26%   -0.01%     
==========================================
  Files          95       95              
  Lines       21256    21251       -5     
==========================================
- Hits        20250    20245       -5     
  Misses       1006     1006              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@djc djc added this pull request to the merge queue May 20, 2025
Merged via the queue into main with commit e9335c8 May 20, 2025
34 checks passed
@djc djc deleted the criterion-0.6 branch May 20, 2025 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants