This is definitely out of my depth and I'm not confident all cases are considered or the necessary state in the PipelineExecutor is preserved to support pausing this process.

The original problem was that ExecutePushInternal would only return when the Execute(chunk, chunk, idx_t) call would not return HAVE_MORE_OUTPUT

TryFlushCachingOperators is not prepared to deal with ExecutePushInternal returning HAVE_MORE_OUTPUT, that's the main thing I see that still needs to be fixed

TryFlushCachingOperators is not prepared to deal with ExecutePushInternal returning HAVE_MORE_OUTPUT, that's the main thing I see that still needs to be fixed

I fixed this by detecting that HAVE_MORE_OUTPUT gets returned from ExecutePushInternal, then returning false without setting remaining_sink_chunk to indicate that the flushing did not complete yet, but the sink was succesful.

Outside of TryFlushCachingOperator we check if remaining_sink_chunk was set:

				if (remaining_sink_chunk) {
					return PipelineExecuteResult::INTERRUPTED;
				} else {
					return PipelineExecuteResult::NOT_FINISHED;
				}

By not setting remaining_sink_chunk we don't try to re-sink the chunk, enter TryFlushCachingOperators as normal, and because in_process_operators is non-empty (otherwise HAVE_MORE_OUTPUT would not be returned) we will get back into ExecutePushInternal where we continue pushing the produced chunks into the Sink.

Thanks!