Change Request: Repeatable performance test #19680
Description
ESLint version
HEAD
What problem do you want to solve?
Right now we have npm run test:performance as a performance tests, however, it gives me inconsistent results when run over time. In most cases, the times increase the more times I run the test even without any changes to the code. This makes it difficult to determine if changes are actually affecting overall performance.
For instance, here are two runs, executed back-to-back on my machine. The median for single-file linting is almost an entire one second longer in the second run then the first, and the median for the multi-file linting is more than two seconds longer.
Run 1
$ npm run test:performance
> eslint@9.25.1 test:performance
> node Makefile.js perf
Loading:
Load performance Run #1: 393.2312ms
Load performance Run #2: 362.5725ms
Load performance Run #3: 375.9464ms
Load performance Run #4: 418.9092ms
Load performance Run #5: 289.1527ms
Load Performance median: 375.9464ms
Single File:
CPU Speed is 2304 with multiplier 13000000
Performance Run #1: 5141.3512ms
Performance Run #2: 4238.4965ms
Performance Run #3: 3736.1255ms
Performance Run #4: 3756.1516ms
Performance Run #5: 3902.634ms
Performance budget ok: 3902.634ms (limit: 5642.361111111111ms)
Multi Files (450 files):
CPU Speed is 2304 with multiplier 39000000
Performance Run #1: 10183.0626ms
Performance Run #2: 9867.0768ms
Performance Run #3: 10272.8082ms
Performance Run #4: 10528.0819ms
Performance Run #5: 12737.2459ms
Performance budget ok: 10272.8082ms (limit: 16927.083333333332ms)
Run 2
$ npm run test:performance
> eslint@9.25.1 test:performance
> node Makefile.js perf
Loading:
Load performance Run #1: 292.4174ms
Load performance Run #2: 309.7286ms
Load performance Run #3: 317.5898ms
Load performance Run #4: 486.638ms
Load performance Run #5: 320.4985ms
Load Performance median: 317.5898ms
Single File:
CPU Speed is 2304 with multiplier 13000000
Performance Run #1: 3710.871ms
Performance Run #2: 3886.6845ms
Performance Run #3: 4845.4551ms
Performance Run #4: 5080.1957ms
Performance Run #5: 4950.8853ms
Performance budget ok: 4845.4551ms (limit: 5642.361111111111ms)
Multi Files (450 files):
CPU Speed is 2304 with multiplier 39000000
Performance Run #1: 12910.4252ms
Performance Run #2: 12239.1613ms
Performance Run #3: 12555.3277ms
Performance Run #4: 13088.5363ms
Performance Run #5: 12608.6794ms
Performance budget ok: 12608.6794ms (limit: 16927.083333333332ms)
What do you think is the correct solution?
I'm not sure. Maybe the perf test should run in a Docker container?
Participation
- I am willing to submit a pull request for this change.
Additional comments
Looking for a volunteer to dig into this.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status