If this can't send the full message one of the two breaks below will be executed which may result in this function returning with all Send() calls using MSG_MORE. For example: let there be 3 messages, 10 bytes each. Send() of the first one will use MSG_MORE, let it return 10 (full message sent). Send() of the second one will use MSG_MORE, let it return any number in [0..9]. Then this function will return, without ever calling Send() without MSG_MORE and the first message may remain buffered/unsent somewhere in the OS.

I think this is unwanted effect, defeating TCP_NODELAY and the check if (it + 1 != node.vSendMsg.end()).

I wonder if it is possible to tell the OS that we are done, without sending further data? Maybe Send(nullptr, 0, MSG_NOSIGNAL | MSG_DONTWAIT); // without MSG_MORE would do that?

If not all of our pending outgoing messages can fit in the socket's transmit buffer, then we want all of our calls to send to specify MSG_MORE. The kernel will immediately (without delay) transmit as many full segments as it can assemble and that will fit in the congestion window. It will not immediately transmit a partial segment but instead will call back to us (via returning a writable status from select, poll, epoll, etc.) to fetch more data to transmit. If we have no more data to transmit, then yes, calling send with a zero length and without MSG_MORE will tell the kernel that we have nothing left to send, and it will transmit a partial segment as soon as such segment will fit in the congestion window.

In your example, if the second of the three 10-byte messages can only partially be copied to the socket transmit buffer, then it must be the case that the transmit buffer is now full. The transmit buffer is always larger than MSS, the maximum segment size, meaning the kernel can now assemble a full segment, which it will transmit as soon as the window allows. It will then indicate the socket is writable again, and we will be called again to write more of the second message.

So, in short, your concern that we will unnecessarily delay transmitting data is unwarranted. The kernel never delays transmitting a full segment if it fits in the congestion window. The kernel does delay transmitting a partial segment that fits in the congestion window if the last call to send on that socket specified MSG_MORE, but that's exactly what we want, as we want the opportunity to write more data into the transmit buffer so the kernel will potentially transmit fewer and larger segments.

If send(2) writes less bytes than requested, does it follow that the transmit buffer is full? I am not sure about that. Is it documented somewhere? What about a signal interrupting the send midway (or even before it started)? Is that not a case where send(2) writes less bytes than requested, but the transmit buffer is not full?

What about a signal interrupting the send midway (or even before it started)?

I'm pretty sure that's only possible if the socket is in blocking mode. But regardless, if send returns a short write due to invoking a signal handler, then the socket is still writable, and the user code will call send again without delay (i.e., on the next iteration of the I/O loop).

That would be after accepting new connections, sending, receiving and parsing messages from all other (possibly 100s) connections. I guess this would be rare and even when it happens would probably have no adverse effects on the communication, but I would feel more comfortable if this PR does not introduce that. If a dummy Send() without MSG_MORE before the two breaks is a way to avoid that, then I think it should be added.

It is documented

Even better, thanks!

Thanks, I understand now.

FWIW, I switched my (public, well-connected) node to (a rebased version of) this PR, together with @vasild's proposed patch above + logging whenever these empty non-MSG_MORE message would be sent.

On this node with ~250 connections, in 16 hours time, having uploaded 7.2 GB and downloaded 1.2 GB, 221 times a MSG_MORE was interrupted (always after sending some bytes, usually at least a few kB; never with having sent nothing).

221 times a MSG_MORE was interrupted

I think what you may have meant was that a send call indicated that fewer bytes were copied than were requested, not that the call was interrupted (by a signal handler).

The real test would be querying the SIOCOUTQ ioctl on the socket the next time the network I/O loop gets around to servicing it. That's the number of untransmitted bytes sitting in the kernel's transmit buffer. If that number is smaller than the connection's MSS, then the kernel exhausted its transmit buffer and started to delay sending the remaining bytes due to MSG_MORE.

never with having sent nothing

N.B.: It would be impossible for zero bytes to have been copied to a socket in non-blocking mode if that socket had indicated that it was writable and no other thread had written to it since such indication was made. Again, I'm pretty sure that even signal handlers won't interrupt a non-blocking write since the kernel only enters an interruptible wait state during a blocking write.

(Edit: Actually, there may be an edge case where a socket indicates writable despite having no free space in its transmit buffer: specifically the case where the next write call would immediately return an error without copying any bytes. That would be a fatal error on the socket, though, such as "connection reset.")

I think what you may have meant was that a send call indicated that fewer bytes were copied than were requested, not that the call was interrupted (by a signal handler).

Yes, exactly - I tried to be succinct, but that may have been confusing. Specifically, it refers to fewer bytes being accepted by send() than handed to it, while MSG_MORE was set.

N.B.: It would be impossible for zero bytes to have been copied to a socket in non-blocking mode if that socket had indicated that it was writable and no other thread had written to it since such indication was made.

Indeed, so that sounds very much expected that 0 never happened (except when the socket errorred and the connected closed, but in that case, there is obviously no point in adding a non-MSG_MORE padding either).