viced: Avoid issuing redundant TMAY requests

Currently, if a new Rx connection comes in from a host we already have
a host struct for, we make a TellMeAboutYourself (TMAY) call to the
given host, to verify the UUID (and caps, interface info, etc) is what
we expect it to be. That is, if it's still the "same" host that we
know about. This is necessary because we otherwise have no way of
telling if the Rx connection is from the same host, or from a new host
that just happens to have the same IP address (e.g. in the case that
hosts are moving around and changing IPs). We do this while the host
is locked, so we only issue these TMAY calls one at a time.

If a large number of Rx connections come in from the same host at
around the same time, this can result in a lot of TMAY requests being
issued against the host, even for hosts that never change IPs and
never do anything strange. In these situations, issuing so many TMAYs
is useless. If we have several calls waiting to lock the host to issue
a TMAY, some of the extra TMAY calls are provably useless. So instead
of calling TMAY repeatedly, remember what the last successful TMAY
result was, and reuse it for the "provably useless" calls.

Note that this 'cache' stores the actual raw results of
TellMeAboutYourself. We could save some memory by storing just how we
interpret that data later on in h_GetHost_r, but this way results in
way simpler h_GetHost_r logic. Since, we can use the same code paths
as for a "real" TMAY call.

Change-Id: I6df74e625e90499bd64c9eb34f20db440f6605a6
Reviewed-on: http://gerrit.openafs.org/9711
Tested-by: BuildBot <buildbot@rampaginggeek.com>
Reviewed-by: Derrick Brashear <shadow@your-file-system.com>

diff --git a/src/viced/host.c b/src/viced/host.c
index c24841c..49a6e49 100644 (file)
--- a/src/viced/host.c
+++ b/src/viced/host.c
@@ -907,6 +907,9 @@ h_TossStuff_r(struct host *host)
            free(host->hcps.prlist_val);
        host->hcps.prlist_val = NULL;
        host->hcps.prlist_len = 0;
+       free(host->tmay_caps.Capabilities_val);
+       host->tmay_caps.Capabilities_val = NULL;
+       host->tmay_caps.Capabilities_len = 0;
        DeleteAllCallBacks_r(host, 1);
        host->hostFlags &= ~RESETDONE;  /* just to be safe */
 
@@ -1602,7 +1605,210 @@ removeInterfaceAddr_r(struct host *host, afs_uint32 addr, afs_uint16 port)
     return 0;
 }
 
+/*
+ * The following few functions deal with caching TellMeAboutYourself calls
+ * that we issued to clients. Why do we do this? Well:
+ *
+ * Q: First, why do we need to issue a TMAY against a client on an incoming new
+ * Rx connection?
+ *
+ * A: We must verify that the incoming Rx connection is the same host that
+ * we have a 'host' structure for. On new calls for existing connections, we
+ * can remember which host corresponds to that connection, but for new
+ * connections, we have no way to find what host it is for, except by looking
+ * up the host by IP address. Since hosts can change IP addresses, we need to
+ * contact the IP address to see if it's the host we think it is.
+ *
+ * Q: Okay, then why cache the results?
+ *
+ * A: The TMAY calls to a single host are serialized, because they are issued
+ * with the host's host->lock held. If we get 4 Rx calls each on new
+ * connections to the same host at the same time, the 1st call will lock the
+ * host, and issue a TMAY. Once that's done, the second call will issue a
+ * TMAY, then the 3rd and then the 4th. However, the 3rd and 4th calls have
+ * been waiting to issue a TMAY since before the 2nd call even _started_ to
+ * issue a TMAY. So, we can just effectively give the results of the 2nd
+ * call's TMAY to the 3rd and 4th, too. Since it is nondeterministic which of
+ * those calls gets to issue a TMAY "first", we can just assume they all got
+ * the same result.
+ *
+ * Note that in the above example, we cannot reuse the results of the 1st TMAY
+ * for the 2nd, 3rd, and 4th calls (we only reuse the results of the 2nd).
+ * This is because by the time the 2nd call starts waiting for the host lock,
+ * it doesn't know how long the 1st TMAY has been running, so the 2nd call
+ * might indeed get a different TMAY result (though this probably would be
+ * extremely rare).
+ *
+ * Anyway, so, if we don't cache the results, we are issuing TMAYs that are
+ * pure overhead. In an environment with clients that create a lot of
+ * connections to a fileserver (keep in mind each new PAG on OpenAFS clients
+ * creates a new connection), this can mean a significant amount of overhead.
+ * So, we use this "TMAY cache" to avoid this overhead for the common case.
+ */
 
+/**
+ * Should we skip calling TellMeAboutYourself on this host, and instead rely
+ * on cached TMAY results?
+ *
+ * @param[in] host  The host we are dealing with
+ * @param[in] prewait_tmays  What the value of host->n_tmays was _before_ we
+ *                           locked 'host'
+ * @param[in] prewait_host   What the primary IP address for 'host' was before
+ *                           we locked 'host'
+ * @param[in] prewait_port   What the primary port was for 'host' before we
+ *                           locked 'host'
+ *
+ * @return Whether we should skip calling TMAY, and instead rely on cached
+ *         results in host
+ */
+static int
+ShouldSkipTMAY(struct host *host, int prewait_tmays, afs_uint32 prewait_host,
+               afs_uint16 prewait_port)
+{
+    int skiptmay = 0;
+    if (host->n_tmays > prewait_tmays + 1) {
+       /* while we were waiting for the host lock, the in-progress TMAY
+        * call finished, someone else started a new TMAY call, and that
+        * finished. so, calling TMAY again won't give us any information
+        * or additional guarantees. */
+       skiptmay = 1;
+    }
+    if (host->host != prewait_host || host->port != prewait_port) {
+       /* ...but don't skip it if the host has changed */
+       skiptmay = 0;
+    }
+    return skiptmay;
+}
+
+/**
+ * If appropriate, simulate a TellMeAboutYourself call on 'host' by extracting
+ * cached interfaceAddr and Capabilities information from 'host' itself.
+ *
+ * @param[in] host  The host we're dealing with
+ * @param[inout] askiptmay  On entering this function, this should contain the
+ *                          result of ShouldSkipTMAY. On return, it is 1 if we
+ *                          actually did use cached TMAY results, or 0 if we
+ *                          did not.
+ * @param[out] interf  The interfaceAddr result of the simulated TMAY call
+ * @param[out] caps    The Capabilities result of the simulated TMAY call
+ *
+ * @return status
+ * @retval 0 We used the cached TMAY results; do NOT make a real TMAY request
+ * @retval otherwise We did not use cached TMAY results; issue a real TMAY request
+ */
+static int
+SimulateTMAY(struct host *host, int *askiptmay, struct interfaceAddr *interf,
+             Capabilities *caps)
+{
+    size_t capsize;
+
+    if (!*askiptmay) {
+       /* we're not supposed to skip the actual TMAY call */
+       return -1;
+    }
+
+    *interf = host->tmay_interf;
+
+    free(caps->Capabilities_val);
+    caps->Capabilities_val = NULL;
+    caps->Capabilities_len = 0;
+
+    if (!host->tmay_caps.Capabilities_val) {
+       return 0;
+    }
+
+    capsize = sizeof(caps->Capabilities_val[0]) * host->tmay_caps.Capabilities_len;
+
+    caps->Capabilities_val = malloc(capsize);
+    if (!caps->Capabilities_val) {
+       /* we should/did _not_ skip the real TMAY call, since we couldn't
+        * alloc memory to use the cached results */
+       *askiptmay = 0;
+       return -1;
+    }
+    caps->Capabilities_len = host->tmay_caps.Capabilities_len;
+    memcpy(caps->Capabilities_val, host->tmay_caps.Capabilities_val, capsize);
+
+    return 0;
+}
+
+/**
+ * If appropriate, store the given results from a real TellmeAboutYourself
+ * call, and cache them in the given host structure.
+ *
+ * @param[in] host  The host we're dealing with
+ * @param[in] skiptmay  1 if we skipped making a real TMAY call, 0 otherwise
+ * @param[in] didtmay  1 if we issued a successful real TMAY call, 0 otherwise
+ * @param[in] interf  The interfaceAddr result from the real TMAY call
+ * @param[in] caps    The Capabilities result from the real TMAY call
+ */
+static void
+CacheTMAY(struct host *host, int skiptmay, int didtmay,
+          struct interfaceAddr *interf, Capabilities *caps)
+{
+    size_t capsize;
+
+    if (skiptmay) {
+       /* we simulated the TMAY call, so the state of the world hasn't
+        * changed; don't touch anything */
+       return;
+    }
+    if (!didtmay) {
+       /* we did not perform a successful TMAY, so we don't have valid
+        * results to cache. blow away the existing cache so we don't use
+        * stale results */
+       goto resetcache;
+    }
+    if (host->n_tmays == INT_MAX) {
+       /* make sure int rollover doesn't screw up our ordering */
+       goto resetcache;
+    }
+    if (host->lock.num_waiting == 0) {
+       /* nobody is waiting for this host, so no reason to cache anything */
+       goto resetcache;
+    }
+
+    /* okay, if we got here, everything looks good; let's cache the given
+     * 'interf' and 'caps' */
+
+    host->tmay_interf = *interf;
+
+    if (!caps->Capabilities_val) {
+       free(host->tmay_caps.Capabilities_val);
+       host->tmay_caps.Capabilities_val = NULL;
+       host->tmay_caps.Capabilities_len = 0;
+
+    } else {
+       if (caps->Capabilities_len != host->tmay_caps.Capabilities_len) {
+           free(host->tmay_caps.Capabilities_val);
+           host->tmay_caps.Capabilities_val = NULL;
+           host->tmay_caps.Capabilities_len = 0;
+       }
+
+       capsize = sizeof(caps->Capabilities_val[0]) * caps->Capabilities_len;
+
+       if (!host->tmay_caps.Capabilities_val) {
+           host->tmay_caps.Capabilities_val = malloc(capsize);
+           if (!host->tmay_caps.Capabilities_val) {
+               goto resetcache;
+           }
+       }
+
+       host->tmay_caps.Capabilities_len = caps->Capabilities_len;
+       memcpy(host->tmay_caps.Capabilities_val, caps->Capabilities_val, capsize);
+    }
+    host->n_tmays++;
+    return;
+
+ resetcache:
+    /* blow away the cached TMAY data; pretend we never saw anything */
+    free(host->tmay_caps.Capabilities_val);
+    host->tmay_caps.Capabilities_val = NULL;
+    host->tmay_caps.Capabilities_len = 0;
+    memset(&host->tmay_interf, 0, sizeof(host->tmay_interf));
+
+    host->n_tmays = 0;
+}
 
 static int
 h_threadquota(int waiting)
@@ -1653,12 +1859,24 @@ h_GetHost_r(struct rx_connection *tcon)
         * structure for this address. Verify that the identity
         * of the caller matches the identity in the host structure.
         */
+
+       int didtmay = 0; /* did we make a successful TMAY call against host->host? */
+       unsigned int prewait_tmays;
+       afs_uint32 prewait_host;
+       afs_uint16 prewait_port;
+       int skiptmay;
+
        if ((host->hostFlags & HWHO_INPROGRESS) &&
            h_threadquota(host->lock.num_waiting)) {
                h_Release_r(host);
            host = NULL;
            goto gethost_out;
        }
+
+       prewait_tmays = host->n_tmays;
+       prewait_host = host->host;
+       prewait_port = host->port;
+
        h_Lock_r(host);
        if (!(host->hostFlags & ALTADDR) ||
             (host->hostFlags & HOSTDELETED)) {
@@ -1689,15 +1907,29 @@ h_GetHost_r(struct rx_connection *tcon)
 
        cb_conn = host->callback_rxcon;
        rx_GetConnection(cb_conn);
+
+       skiptmay = ShouldSkipTMAY(host, prewait_tmays, prewait_host, prewait_port);
+
        H_UNLOCK;
         if (haddr == host->host && hport == host->port) {
             /* The existing callback connection matches the
              * incoming connection so just use it.
              */
-           code =
-               RXAFSCB_TellMeAboutYourself(cb_conn, &interf, &caps);
-           if (code == RXGEN_OPCODE)
-               code = RXAFSCB_WhoAreYou(cb_conn, &interf);
+
+           if (SimulateTMAY(host, &skiptmay, &interf, &caps) == 0) {
+               /* noop; we don't need to call TellMeAboutYourself; we can
+                * trust the results from the last TMAY call */
+               code = 0;
+
+           } else {
+               code =
+                   RXAFSCB_TellMeAboutYourself(cb_conn, &interf, &caps);
+               if (code == RXGEN_OPCODE)
+                   code = RXAFSCB_WhoAreYou(cb_conn, &interf);
+               if (code == 0) {
+                   didtmay = 1;
+               }
+           }
        } else {
             /* We do not have a match.  Create a new connection
              * for the new addr/port and use multi_Rx to probe
@@ -1718,6 +1950,9 @@ h_GetHost_r(struct rx_connection *tcon)
        rx_PutConnection(cb_conn);
        cb_conn=NULL;
        H_LOCK;
+
+       CacheTMAY(host, skiptmay, didtmay, &interf, &caps);
+
        if ((code == RXGEN_OPCODE) ||
            ((code == 0) && (afs_uuid_equal(&interf.uuid, &nulluuid)))) {
            identP = malloc(sizeof(struct Identity));

diff --git a/src/viced/host.h b/src/viced/host.h
index 06776a2..d4a3a3e 100644 (file)
--- a/src/viced/host.h
+++ b/src/viced/host.h
@@ -80,6 +80,12 @@ struct host {
      * the index fields isn't zeroed. XXX
      */
     afs_uint32 index;          /* Host table index, for vicecb.c */
+    unsigned int n_tmays;       /* how many successful TellMeAboutYourself calls
+                                 * have we made against this host? */
+    /* cache of the result of the last successful TMAY call to this host */
+    struct interfaceAddr tmay_interf;
+    Capabilities tmay_caps;
+
     struct Lock lock;          /* Write lock for synchronization of
                                 * VenusDown flag */
     pthread_cond_t cond;       /* used to wait on hcpsValid */