Fast sync gets stuck if there's not many peers because requestRoutine in pool requests blocks out of order #1283
Description
Fast sync gets stuck (or proceeds very slowly) because requestRoutine in pool requests blocks in a different order because requesters get made asynchronously by makeNextRequester (https://github.com/tendermint/tendermint/blob/main/blockchain/v0/pool.go#L375C24-L375C41). This causes fast sync to get stuck if there's not many peers, because maxPendingRequestsPerPeer is 20.
Sample debugging logs from manually added debugging prints:
debug0: making next requester for 256513
debug0: making next requester for 256514
debug0: making next requester for 256515
debug0: making next requester for 256516
debug0: making next requester for 256517
debug0: making next requester for 256518
debug0: making next requester for 256519
debug0: making next requester for 256520
debug0: making next requester for 256521
debug0: making next requester for 256522
debug0: making next requester for 256523
debug0: making next requester for 256524
debug0: making next requester for 256525
debug0: making next requester for 256526
debug0: making next requester for 256527
debug0: making next requester for 256528
debug0: making next requester for 256529
debug0: bpr making request for 256529
debug0: bpr making request for 256521
debug0: bpr making request for 256514
debug0: bpr making request for 256516
debug0: bpr making request for 256517
debug0: bpr making request for 256515
debug0: bpr making request for 256518
debug0: bpr making request for 256527
debug0: bpr making request for 256519
debug0: bpr making request for 256520
debug0: bpr making request for 256522
debug0: bpr making request for 256513
debug0: bpr making request for 256523
debug0: bpr making request for 256524
debug0: bpr making request for 256525
debug0: bpr making request for 256526
Increasing maxPendingRequestsPerPeer to 600 fixes the problem (though obviously not a proper fix). edit: still having issues with increasing it to that param, unless I also decrease maxTotalRequesters significantly