viced: consistently enforce host thread quota for ICBS(3)
From time to time, the fileserver may issue potentially long-running
RXAFSCB_* RPCs back to a host (client). If these are holding h_Lock_r
(host->lock) while running, they may cause other service threads for the
same host (client) to block.
In order to prevent a given host from tying up too many service threads
in this way, the fileserver enforces a quota limiting how many threads
can be waiting for h_Lock_r on a particular host while waiting for one
of the following RPCs to complete:
- RXAFSCB_TellMeABoutYourself (TMAY)
- RXAFSCB_WhoAreYou
- RXAFSCB_ProbeUuid
- RXAFSCB_InitCallBackState (ICBS)
- RXAFSCB_InitCallBackState3 (ICBS3)
Note: Although some of these RPCs are relatively lightweight, they may
still experience network delays.
This quota is enforced by calling h_threadquota() in h_Lookup_r and
h_GetHost_r. The quota check is enabled for a given host by turning on
host->hostFlags HWHO_INPROGRESS for the duration of the RXAFSCB_* RPC.
The quota check is only needed, and should only be enabled, when the RPC
is issued while h_Lock_r is held.
However, there are a few paths to ICBS(3) where h_Lock_r is held but
HWHO_INPROGRESS is not set. A delay in those paths may allow a host to
consume an unlimited number of fileserver threads. One such path
observed in a field report was SRXAFS_FetchStatus -> CallPreamble ->
BreakDelayedCallBacks_r -> RXAFSCB_ICBS3.
Instead, enable host thread quotas for all remaining unregulated ICBS(3)
RPCs.
Change-Id: I70b96055ff80d8650bdbaec0302b7d18a8f22d56
Reviewed-on: https://gerrit.openafs.org/13873
Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
Tested-by: BuildBot <buildbot@rampaginggeek.com>