Improve web benchmarks measurements #127900
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yjbanov
approved these changes
May 31, 2023
| // Provide COOP/COEP headers so that the browser loads the page as | ||
| // crossOriginIsolated. This will make sure that we get high-resolution | ||
| // timers for our benchmark measurements. | ||
| if (mimeType == 'text/html' || mimeType == 'text/javascript') { |
There was a problem hiding this comment.
Should we also add this to flutter run?
There was a problem hiding this comment.
We'll have to address this question eventually, but I think we should really just do this for our benchmarks for now. Enabling crossOriginIsolated in flutter run could potentially break clients... we might want to do so conditionally or something. I think we should have a broader conversation about it, but it's probably out of scope of this PR. I'll have to figure this out for skwasm either way.
There was a problem hiding this comment.
I filed #127902 as a follow on
|
To follow up, Skia Perf is picking up the change: https://flutter-flutter-perf.skia.org/t/?begin=1685500141&end=1685500143&subset=all |
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Jun 3, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Aug 16, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Aug 17, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Aug 17, 2023
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/packages
that referenced
this pull request
Aug 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By default, the browser fuzzes the timer APIs such that they have a granularity of approximately 100 microseconds (this is due to Spectre mitigation techniques). However, many of the thing we are trying to measure actually have a much finer granularity than 100 microseconds. As a result, many of our benchmarks are extremely noisy and don't provide accurate data.
By serving the initial script files with the
Cross-Origin-Opener-Policy: same-originandCross-Origin-Embedder-Policy: require-corpHTTP headers, the browser runs the benchmarks in acrossOriginIsolatedcontext, which restores the fine granularity of APIs such asperformance.now()to microsecond precision.Also, we were considering anything an outlier that was more than one standard deviation away from the mean. In a normal distribution, that means we are only capturing 68% of the data and the rest are considered outliers. This is not ideal. Doing two standard deviations away captures 95% of the data, and the outliers are in the remaining 5%, which seems much more reasonable.