Capture screenshots with Canvas and capture stream. #1994
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR: Working implementation of screenshot capturing using the canvas, the relatively new APIs "captureStream" and "ImageCapture", and Inter-plugin-communication.
Implementation was partially inspired by #1909 and the commit cc77737
A bit of the background information: I use a High-DPI screen (2880x1920 on a 10.1" monitor, ~342ppi) and must set Window's display scaling to 200% to use properly.
Because of this, whenever launch poi it would be upscaled by 200%. Everything looks right and the UI is as usable as not scaled.
However, screenshots looks bad and the resolution it uses is weird.
For example, when I use poi at 100% scaling, the resolution poi thinks I'm using is 1200x720. The actual rendered resolution is 2400x1440 due to the 200% scaling by Windows.
The screenshot taken is also scaled by 200%, i.e. 2400x1440. That's the problem.
The 2400x1440 screenshot is upscaled and doesn't look very good, especially if you downscale it back to lower resolutions for sharing purposes.
Actually, viewing this upscaled screenshot in any non-high DPI monitor it would look bad.
What's worse is that when I use poi to play at reduced resolutions, e.g. 67%, the resolution becomes 800x480 (actual rendered 1600x960).
The image looks apparently scaled.
Even if I play at 50%, i.e. 600x360, and screenshot taken at 1200x720, it looks like the image had been upscaled and then downscaled (color distortion and reduced sharpness).
Also, 50% is pretty hard to get in the current poi version because there's only 33%/67%/100% to choose from, and than you can no long type in the desired resolution anymore.
I've tried overriding high-DPI settings on Windows and turning on compatibility mode, but it seems that Electron doesn't care.
I've also looked into the poi source code for taking screenshots, and found out that poi uses WebContents.capturePage to take screenshots.
It seems to be a known issue of Electron that got ignored:
electron/electron#13154
So I looked into taking a screenshot using Canvas (and later realized that this was previously attempted in the commit cc77737).
The implementation in that commit does not seem to work because KanColle's rendering context is a WebGL context, which clears out the buffer after rendering. This made it impossible to directly export the image data.
I had the same problem in my initial attempt to use the canvas. After some research, there seems to be an experimental API on canvas: canvas.captureStream and ImageCapture. These two APIs provided a simple way to capture screenshots, or even videos, Although experimental, I don't think it would be an issue because Poi uses Electron which ensures we have the right browser running. At the worst case, in case the experimental API changes, we'd simple revert this change.