Looks like a good idea to me, but it needs some changes to the implementation. I would keep the changes you made to SubChooseViewController.xib, but replace the changes to MainWindowController.swift with this:

  override func keyDown(with event: NSEvent) {
    if isShowingPersistentOSD {
      let keyCode = KeyCodeHelper.mpvKeyCode(from: event)
      let normalizedKeyCode = KeyCodeHelper.normalizeMpv(keyCode)
      if normalizedKeyCode == "ESC", osdStackView.performKeyEquivalent(with: event) {
        Logger.log("ESC key was handled by OSD", level: .verbose)
        return
      }
    }

    super.keyDown(with: event)
  }

Rationale:

1. You need to add code in keyDown instead of handleKeyBinding, because that will only get called if there is a key binding for ESC in the Settings > Key Bindings table. (And that binding would have to be for a different action because there is no support for any kind of "dismiss OSD" action there.)
2. Check for isShowingPersistentOSD first because it's the fastest check and will require less CPU when it is false (which is most of the time).
3. No need to check hideOSDTimer now. The function osdStackView.performKeyEquivalent will return true only if it recognized the event and accepted it. Other OSDs such as "screenshot taken" do not recognize ESC and will return false, so those will fall through and call super.keyDown which will allow the key to be processed normally.

Thank you for the feedback! These are good points—I've updated the PR with exactly the code you suggested.

Looks good to me, but I think keyUp is a better place for key shortcuts. keyDown can be called repeatedly if the key is long-pressed, while keyUp is only fired once and represents the action better. Could you change it?

I don't think so. Changing it to keyUp the window will be close when the user releases the ESC key, however this is not how the system handle ESC when it's been used to close aux window. Try opening Music.app, and open the settings of it. The settings window can be closed via ESC, and it is immediately closed when you press ESC key, instead of waiting until you release the key. Besides, using keyDown shouldn't cause any performance issue in this context.

I'd agree with staying on keyDown for this.
In general it's the way macOS handles key equivalents, but here specifically, switching to keyUp actually makes the functionality glitchier: since the key down event is let through, it causes the window to exit fullscreen (or perform any other action that's bound to the key) in addition to then closing the OSD.

Sounds reasonable, you are right. @Cykelero thanks for your contribution!