rx: For AFS_RXERRQ_ENV, retry sendmsg on error

rx: For AFS_RXERRQ_ENV, retry sendmsg on error

When AFS_RXERRQ_ENV is defined, we currently end up doing something
like this for our sendmsg abstractions:

    if (sendmsg(...) < 0) {
        while (rxi_HandleSocketError(sock))
            ;
        return error;
    }
    return success;

This means that when sendmsg() returns an error, we process the socket
error queue before returning an error.

The problem with this is that when we receive an ICMP error on our
socket, it creates a pending socket error that is returned for any
operation on the socket. So, if we receive an ICMP error after trying
to contact any peer, sendmsg() could return an error when trying to
send for any other peer. Even though there is no issue preventing us
from sending the packet, we'll fail to actually send the packet
because sendmsg() returned an error. This effectively causes an extra
outgoing packet drop, possibly delaying the related RPC.

To avoid this, change Rx to retry the sendmsg call when it returns an
error, since the error may be due to an unrelated ICMP error.

To avoid needing to implement this retry loop in multiple places, move
around our sendmsg code for AFS_RXERRQ_ENV, so that the higher-level
function rxi_NetSend performs the retry and checks for socket errors
(instead of the lower-level rxi_Sendmsg or osi_NetSend). Also change
our functions to process socket errors to be more consistent between
kernel and userspace: now we always have rxi_HandleSocketErrors, which
runs a loop around the platform-specific osi_HandleSocketError.

With this commit, osi_HandleSocketError is now required to be
implemented when AFS_RXERRQ_ENV is defined. We hadn't been
implementing this for UKERNEL, so just turn off AFS_RXERRQ_ENV for
UKERNEL.

Thanks to mbarbosa@sinenomine.net for discovering and providing
information about the relevant issue.

Change-Id: Iccceddcd2d28992ed7a00dc308816a0cb1a0195f
Reviewed-on: https://gerrit.openafs.org/14424
Reviewed-by: Cheyenne Wills <cwills@sinenomine.net>
Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
Tested-by: BuildBot <buildbot@rampaginggeek.com>