Checklist

• I've searched the issue tracker for similar bugs.

Describe the bug
The application I have here boils down to a simple client-server pair, where each party is sending a large amount of data to the other, while simultaneously reading the data from the other party.

The underlying sockets have been set to non-blocking. There is an I/O thread that spins and keeps trying to read from/write to a rustls::Stream, with proper buffers on the application side (so it is always accepting newly read data). The issue is that after a bit, the app deadlocks for both parties.

After a bit of investigation I think the issue is that, when the rustls::Stream tries to read, it first tries to 'complete prior IO'. This involves writing any completed TLS records to the underlying socket. Unfortunately, if the underlying socket is full and won't accept new data (i.e. WouldBlock), 'write TLS' just fails, so 'complete prior IO' just fails.
Then the stream never attempts to read from the underlying socket at all but instead directly returns WouldBlock. As a result, if the TCP stream is full on both directions, neither party is reading from the TCP stream (because neither party can successfully 'write TLS'), and the TCP stream never gets emptied, so no more progress is made.

Applicable Version(s)
Tested on 0.22.4 and 0.23.29