Not bad, but looks like only a partial fix for #4667. Anything which calls setFrame will also cause a jitter, like pinch-to-zoom , scale changes & enter/exit full screen.

Oh, I was searching for that issue but I didn't find that. I'll try to extend the fix for that issue.

Added fix for pinch-to-zoom. I've tried to set async when entering/exiting full screen, but that didn't help. I think the animation of fullscreen should be another problem.

Added fix for pinch-to-zoom. I've tried to set async when entering/exiting full screen, but that didn't help.

Very nice! And yeah, looks like when setFrame is called with animate: true, it fires the window*LiveResize listeners, so the commands to change window scale are covered by this fix as well. I guess this now covers almost all of the ways window sizes are changed in IINA.

I think the animation of fullscreen should be another problem.

I think you might be right. Can say it's out of scope.

When I was testing to resize the window, I also notice that, with this PR, when the OSC is not displayed, we got the smoothest resizing. However, if the OSC is displayed while resizing (especially the top and bottom one), the framerate drops.

I profiled IINA while resizing the window, and the most time-consuming part on main thread is on autolayout. For every frame when the window is live resizing, the system triggers a layoutIfNeeded, which will perform autolayout using the new window size. The autolayout for "floating" OSC is much simpler than the other two which may contribute to a slightly better performance.

If my hypothesis is correct, then I don't see a simple solution to this. This should be a separate issue as well. I put it here because it's easier to observe using this PR.

I profiled IINA while resizing the window, and the most time-consuming part on main thread is on autolayout. For every frame when the window is live resizing, the system triggers a layoutIfNeeded, which will perform autolayout using the new window size. The autolayout for "floating" OSC is much simpler than the other two which may contribute to a slightly better performance.

Mmm yes. I did some similar profiling, and also did a bunch of work rendering thumbnails, and my conclusion is: anything starting with NS is verrry slow. For example, scaling a CGImage using CoreGraphics APIs is 10-100x faster than scaling an NSImage using view-level APIs.

But I also found this blog post which suggests changing the NSView property layerContentsRedrawPolicy so that we don't redraw every single view with every resize.

From that post I created this thing, which I executed on window.contentView at the end of windowDidLoad():

extension NSView {
  /// Recursive func which configures all views in the given subtree for smoother animation.
  ///
  /// By configuring each view to use a layer with the correct redraw policy, AppKit will use Core Animation to draw
  /// them, which uses a dedicated background thread instead of the main thread and does not make excessive draws.
  func configureSubtreeForCoreAnimation() {
    if self is NSButton || self is NSSlider || self is NSProgressIndicator {
      // these still need to be redrawn on every resize or they get very buggy
      return
    }
    self.wantsLayer = true
    self.layerContentsRedrawPolicy = .onSetNeedsDisplay
    for subview in self.subviews {
      subview.configureSubtreeForCoreAnimation()
    }
  }
}

With this, resizing does appear smoother to me, but I feel like it needs more exploration.

layerContentsRedrawPolicy so that we don't redraw every single view with every resize.

Oh wow thanks for pointing this out, I always assumed that layer backed views did this by default because I observed during live resizes that the video layer content was being scaled (instead of rerendered at the new frame size), but maybe what it's actually doing for CAOpenGLLayer at least is triggering a re-draw but with a target opengl viewport size as the initial frame size. I don't know much about rest of iina, but at least for the video layer it would seem like an easy performance win to set this to "onSetNeedsDisplay" since redrawing at the original frame size is completely redundant. Only thing to check is that when the video layer is in async mode we still periodically get called during a live-resize even when set to NSViewLayerContentsRedrawOnSetNeedsDisplay

I observed during live resizes that the video layer content was being scaled (instead of rerendered at the new frame size

@krackers hope you see this comment since the PR is now closed -

I have noticed this as well. In particular, if the video is left paused ~~ for at least 6 seconds (i.e., so that the display link has stopped)~~ then when resizing the window, the VideoView will grow or shrink according to its layout constraints, and the layer will simply stretch whatever image is already there, as though it's a fixed texture which is mapped to a polygon (which it basically is?). This is totally fine if the last render occurred while the window was relatively large (more specifically, the size of VideoView, adjusted for contentsScale, was larger than the video's native resolution). But if it was very small when that render occurred, then when the window is resized the displayed picture will appear blocky or pixelated. EDIT 2: I was reciting from memory but can't seem to reproduce this now! I believe the behavior I'm describing is what would happen except that "forced" renders are being used to correct it. But I need to dig in to it again. Can you isolate a particular case which can lead to this symptom in IINA?

I think what is happening in this case is:

1. The window resize triggers a call to CAOpenGLLayer's display method
2. This queries canDraw, but mpv reports there are no new frames to draw and forceRender is false, so canDraw returns false.

So even if a window resize results in extra unneeded calls to display, for our VideoView at least they won't trigger any extra work to be done because they're essentially no-ops.

It looks to be the case (as relating to window resizing at least) that the performance problems are coming the display implementations of AppKit's various views (NSView subclasses) and not CAOpenGLLayer. Almost certainly the best way to improve performance here is simply to reduce the number of views in the window. [I mentioned in a previous issue that based on the performance metrics I gathered, if we were to replace the play slider with a version which used more efficient drawing APIs, we'd reduce CPU usage by 10% for each window in which the play slider is visible]. But since that's not very feasible, we might at least try to reduce the number of times these views are redrawn.

But coming back to the video scaling issue you identified...while I don't think it's a serious performance concern, it is certainly visually unpleasant and it would be nice to improve it, so I am open to any ideas.

EDIT 1: I realized I didn't talk about the case where the video is playing during resize. But the result is essentially the same as the paused case - the video will rerender when there is a frame to render, not necessarily when the window has resized. This could result in some redraws of the window for which the rendered video's scale is from a previous window size and thus is mismatched. Could this be the cause of the problem?

For window resize wobbling, the root cause of wobble was that the drawing was not coordinated with the resizing. Resizing happens on the main thread, and before this PR the drawing happened on the separate thread mostly independently. Because of this, you could have cases where the the system resizes the frame but the last draw was for the older frame size. (Now CAOpenGLLayer resizes things so it's not as noticeable in most cases, but for low-fps video where draw is less often, this mismatch is responsible for jittering.)

Traditional (non layer backed) openglviews have this issue too. It's not very well documented so I had to piece together this, but if you draw to opengl on a non-main thread you run into similar issues getting proper live resize to work: you essentially need to ensure that in a live resize the main thread only unblocks after a frame with the proper size is rendered, or you get glitches (this can be done either by just drawing in the main thread [with proper locking], or making the main thread wait until the 2nd thread finishes rendering).

You could try to fix sync CAOpenGLlayer use by forcing a redraw whenever the window size changes in a live resize, but this also technically still exhibits the same issue because it's not being synchronized with the frame resizing of the main thread (although any resulting wobble is likely negligible enough to be unnoticeable). It is also more likely to exhibit performance issues because now you're essentially redrawing at resize fps (which may be the screen fps?).

This PR fixes it by making drawing async. It seems when CAOpenGLLayer is marked async, during a live-resize the system never updates the glViewport until the end. Not sure why you say it has to do with displayLink, I've never tested IINA in particular but for mpv at least it's purely a property of being async and is reproducible: the gl viewport is unchanged during live resize. This fixes wobbles, at the cost of introducing blurriness.

I wonder if theoretically instead of using GL_VIEWPORT you directly accessed the window frame size to force it to render at the window size regardless of current gl viewport, you could keep the async mode but force it to render crisply.

This PR fixes it by making drawing async. It seems when CAOpenGLLayer is marked async, during a live-resize the system never updates the glViewport until the end. Not sure why you say it has to do with displayLink, I've never tested IINA in particular but for mpv at least it's purely a property of being async and is reproducible: the gl viewport is unchanged during live resize. This fixes wobbles, at the cost of introducing blurriness

@krackers I did some more investigation on the blurriness issue and it looks like it might be another problem entirely. Forget the display link - I had that mixed up in my mind with a separate issue which is no longer relevant. Let's move this conversation to #5022 which I just opened which is more relevant and also not closed.