Seash upload timeouts

[choksi81]

Seash sometimes times out on uploads. We've done a number of packet traces on the host running seash, and here is what we've found out:

You can categorize three types of errors:

1. signedcommunicate failed on session_recvmessage with error 'recv() timed out!' usually means the file was fully uploaded, but the node failed to return "Success" in time.
2. signedcommunicate failed on session_recvmessage with error 'send() timed out!' means the file was not fully transferred.
3. signedcommunicate failed on session_sendmessage with error 'Socket closed' is adressed in #1009 -- I didn't see this one on uploads so far, only when browsing for nodes.

The recv() issue was tackled in #971 and thought to be solved by speeding up the crypto parts of the communication between node manager and seash (parts of which remain to be improved, see #990). The packet traces show a typical string of events leading to this error:

• The file is fully uploaded, the node acknowledges each TCP segment it sees.
• It fails to produce a "Success" message however.
• Therefore, seash times out after 17 seconds. This is remarkable in itself as the default timeout should be 10 seconds.
• The node acknowledges seash's closing of its half of the connection.
• 3-100 seconds later (empirical values), the node tries to send the "Success" message, but is (correctly) greeted with a RST by seash's TCP.

I've also seen "show files" fail with a recv() timeout, but this is rare. Surprisingly, the timeout is 10 seconds there.

The send() issue manifests like this:

• Parts of the file are uploaded, but TCP flow control kicks in, telling seash's TCP to stop sending data. This means that the node doesn't empty its receive buffer. In my experiments with a 92kB file, only 15kB+ were transferred. Additionally, seash is blocked in session_sendmessage, which uses timeout_sockets.
• Seash's TCP probes the node's for changes in the receive window size. The node's TCP answers that the recv window is still full. This can go on for minutes, so seash times out.
• Finally, buffer space becomes available at the node, and it receives whatever is left in seash's TCP's send buffer. (While seash timed out on application layer, it can't get rid of "old" data it already buffered for sending.)
• Thus, 26kB are transmitted overall in my experiment.

Both recv() and send() issues happen even if I increase seash's timeout (see #892). I tried 90 seconds, but would have had to use 300 or so for the slowest nodes.

I checked with CoMoN -- all of the nodes that produced errors had a high load average, and those with the highest numbers had the most persistent ones. This might hint at how to reproduce the problem on a node where we can locally trace node manager packets.

[choksi81]

Attachments:
https://github.com/SeattleTestbed/attic/blob/master/TICKET_ATTACHMENTS/seash_set_maxthreads.diff