This commit as-is doesn't seem to cause any noticeable problems with my AMD card & SDR only setup, but it has a bad interaction when tested together with #660 (so feel free to disregard in case you are aware) using the amdvce encoder (software seems ok):

When connecting to the host with no significant movement on-screen, the first frame received will look OK, but moving the mouse will trigger a full black frame to display and then the desktop image again, which repeats with each successive mouse movement for perhaps 3-5 seconds, and then the video will appear to stabilize with no further black frames being inserted. There is no log activity when this happens.

Edit: I also attempted to enable debugging via the D3D11_CREATE_DEVICE_DEBUG flag, but it causes Sunshine to exit with nothing logged when the first black frame displays (but I'm not an expert with DX debugging, so I'm not sure if I need to look elsewhere for the debug logs or run in a debugger).

Ok, thanks for testing. I think I'm going to take a different approach with #660 and try using 2 separate ID3D11Device objects (one for capture and one for encode) then pass textures between them using D3D11_RESOURCE_MISC_SHARED_NTHANDLE and ID3D11Device1::OpenSharedResource1().

Edit: I also attempted to enable debugging via the D3D11_CREATE_DEVICE_DEBUG flag, but it causes Sunshine to exit with nothing logged when the first black frame displays (but I'm not an expert with DX debugging, so I'm not sure if I need to look elsewhere for the debug logs or run in a debugger).

Yeah, I think certain fatal errors (like synchronization issues) will cause the debug runtime to raise an exception that will kill the process without a debugger attached. If you attach to sunshine.exe with WinDbg, it will show you debug messages from the debug runtime.

While I was redoing #660, I figured out what the problem was and it was indeed a bug in this PR, though one that's only likely to show up with PARALLEL_ENCODING because DEFAULT only uses a single image. I forgot to call complete_img() in the newly added codepath, so we'd pass back an image with no texture inside if this was the first capture using that image.

I pushed an additional fix for an issue in #654 that I discovered when testing my parallel encoding work. We don't actually want to fall back to DuplicateOutput() if DuplicateOutput1() is available.

There are temporary reasons that desktop duplication fails (such as being in the middle of a modeset operation or the secure desktop being displayed when not running as service). If we happen to fail our 2 tries of DuplicateOutput1() and succeed in the call to DuplicateOutput(), then we'll be stuck with suboptimal capture performance until something else triggers the capture to reinitialize.