So, here's a breakdown:

• partial snapshot recovery is started (bottom log line)
• there's a Local shard 1 not found message
  • which simply means that local replica is unavailable, because partial snapshot is being recovered on the node
  • (the wording is a bit confusing in this case)
• however, by the time this message is logged in execute_cluster_read_operation, all "locks" required to execute read-request should have been acquired already
  • so, the only thing that is being executed during these 2 seconds should be network requests to remote nodes
  • (and preparing response to the user, but this is pure CPU work that should be practically instant)
• judging by the timestamp of the message, it correlates exactly with the total response time
  • e.g., 07:55:58.294 - 07:55:56.006 = 2.288s
• we can also see that there are Recovering shard... and Recovered collection... logs right between Local shard 1 not found and the top log line
  • and all spikes are like these: start recovery, local shard not found, recovering/recovered, search finished (with timestamps correlating exactly)
• the network on rw-seg is fast and remote replicas should not be blocked by anything
  • (partial snapshot is recovered on the same node, that is executing the client request)
• but if there's something blocking the async runtime, that might explain why requests are stalling...
  • and lo and behold, there is a blocking call right in the code path that we know is executed in parallel with our request 🙄