----++ Demand-Attach File-Server (DAFS)<br /><br /> The first production release of OpenAFS containing the Demand-Attach File-Server (DAFS) code was \[[ReleaseInformation#
-
-<div id=":175">RelOneFourFiveTwo][1.4.5-2]]. DAFS is a significant departure from the more <em>traditional</em> AFS file-server and this document details those changes.<br />
- <br />
- <br />
- <br /> ---++ Why Demand-Attach File-Server (DAFS) ?<br />
- <br /> On a traditional file-server, volumes are attached at start-up and detached only at shutdown. Any attached volume can be modified and changes are periodically flushed to disk or on shutdown. When a file-server isn't shutdown cleanly, the integrity of every attached volume has to be verified by the salvager, whether the volume had been modified or not. As file-servers grow larger (and the number of volumes increase), the length of time required to salvage and attach volumes increases, e.g. it takes around two hours for a file-server housing 512GB data to salvage and attach volumes !<br />
- <br /> On a Demand-Attach File-Server (DAFS), volumes are attached only when accessed by clients. On start-up, the file-server reads only the volume headers to determine what volumes reside on what partitions. When accessed by clients, the volumes are attached. After some period of inactivity, volumes are automatically detached. This dramatically improves start-up and shutdown times. A demand-attach file-server can be restarted in seconds compared to hours for the same traditional file-server.<br />
- <br /> The primary objective of the demand-attach file-server was to dramatically reduce the amount of time required to restart an AFS file-server.<br />
- <br /> Large portions of this document were taken / influenced by the presentation entitled [[ATarget%3d_blankHref%3dHttp://workshop/openafs/org/afsbpw06/talks/tkeiser-dafs/pdf5D5BDemandHttp://workshop/openafs/org/afsbpw06/talks/tkeiser-dafs/Pdf]] Attach / Fast-Restart Fileserver given by Tom Keiser at the [[ATarget%3d_blankHref%3dHttp://workshop/openafs/org/5D5BAFSHttp://workshop/openafs/org/WebHome]] and Kerberos Best Practices Workshop in [[<a href="http://workshop.openafs.org/afsbpw06/%5D%5B2006%5D" target="_blank">http://workshop.openafs.org/afsbpw06/][2006]</a>].<br />
- <br />
- <br /> ---++ An Overview of Demand-Attach File-Server<br />
- <br /> Demand-attach necessitated a significant re-design of certain aspects of the AFS code, including:<br /> * volume package has a number of severe limitations<br /> * single global lock, leading to poor scaling<br /> * lock is held across high latency operations, e.g. disk I/O<br /> * no notion of state for concurrently accessed objects<br /> * the vnode package suffers from the same limitations<br /> * breaking callbacks is time consuming<br /> * salvaging does not have to be done with the file-server offline<br />
- <br /> The changes implemented for demand-attach include:<br /> * [[Main/DemandAttachFileServer#Volume_Finite_State_Automata]]<br /> * volumes are attached on demand<br /> * volume <em>garbage collector</em> to detach unused volumes<br /> * notion of volume state means read-only volumes aren't salvaged<br /> * [[Main/DemandAttachFileServer#Vnode_Finite_State_Automata]]<br /> * global lock is only held when required and never held across high-latency operations<br /> * automatic salvaging of volumes<br /> * shutdown is done in parallel (maximum number of threads utilized)<br /> * callbacks are no longer broken on shutdown<br /> * instead, host / callback state is preserved across restarts<br />
- <br />
- <br /> ---++ The Gory Details of the Demand-Attach File-Server<br />
- <br /> ---+++ File-server Start-up / Shutdown Sequence<br />
- <br /> The table below compares the start-up sequence for a traditional file-server and a demand-attach file-server.<br />
- <br /> | <strong>Traditional</strong> | <strong>Demand-Attach</strong> |<br /> || %BULLET% host / callback state restored |<br /> || %BULLET% host / callback state consistency verified |<br /> | %BULLET% build vice partition list | %BULLET% build vice partition list |<br /> | %BULLET% volumes are attached | %BULLET% volume headers read |<br /> || %BULLET% volumes placed into <em>pre-attached</em> state |<br />
- <br /> The [[Main/FSStateDat]] is covered later. The <em>pre-attached</em> state indicates that the file-server has read the volume headers and is aware that the volume exists, but that it has not been attached (and hence is not on-line).<br />
- <br /> The shutdown sequence for both file-server types is:<br /> | <strong>Traditional</strong> | <strong>Demand-Attach</strong> |<br /> | %BULLET% break callbacks | %BULLET% quiesce host / callback state |<br /> | %BULLET% shutdown volumes | %BULLET% shutdown on-line volumes |<br /> || %BULLET% verify host / callback state consistency |<br /> || %BULLET% save host / callback state |<br />
- <br /> On a traditional file-server, volumes are off-lined (detached) serially. In demand-attach, as many threads as possible are used to detach volumes, which is possible due to the notion of a volume has an associated state.<br />
- <br />
- <br /> ---+++ Volume Finite-State Automata<br /> The volume finite-state automata is available in the source under doc/arch/dafs-fsa.dot.</div>
-<div id=":175">
- <br /> #VolumeLeastRecentlyUsed<br /> ---+++ Volume Least Recently Used (VLRU) Queues<br />
- <br /> The Volume Least Recently Used (VLRU) is a garbage collection facility which automatically off-lines volumes in the background. The purpose of this facility is to pro-actively off-line infrequently used volumes to improve shutdown and<br /> salvage times. The process of off-lining a volume from the "attached" state to the "pre-attached" state is called soft detachment.<br />
- <br /> VLRU works in a manner similar to a generational garbage collector. There are five queues on which volumes can reside.<br />
- <br /> | <strong>Queue</strong> | <strong>Meaning</strong> |<br /> | candidate | Volumes which have not been accessed recently and hence are candidates for soft detachment. |<br /> | held | Volumes which are administratively prevented from VLRU activity, i.e. will never be detached. |<br /> | intermediate (mid) | Volumes transitioning from new -> old (see [[DemandAttachFileServer#VLRUStateTransitions][state transitions] for details). |<br /> | new | Volumes which have been accessed. See [[DemandAttachFileServer#VLRUStateTransitions][state transitions] for details. |<br /> | old | Volumes which are continually accessed. See [[DemandAttachFileServer#VLRUStateTransitions][state transitions] for details. |<br />
- <br /> The state of the various VLRU queues is dumped with the file-server state and at shutdown.<br />
- <br /> #VLRUStateTransitions<br /> The VLRU queues new, mid (intermediate) and old are generational queues for active volumes. State transitions are controlled by inactivity timers and are<br />
- <br /> | <strong>Transition</strong> | <strong>Timeout (minutes)</strong> | <strong>Actual Timeout (in MS)</strong> | <strong>Reason (since last transition)</strong> |<br /> | candidate->new | - | - | new activity |<br /> | new->candidate | 1 * vlruthresh | 24 hrs | no activity |<br /> | new->mid | 2 * vlruthresh | 48 hrs | activity |<br /> | mid->old | 4 * vlruthresh | 96 hrs | activity |<br /> | old->mid | 2 * vlruthresh | 48 hrs | no activity |<br /> | mid->new | 1 * vlruthresh | 24 hrs | no activity |<br />
- <br /><code>vlruthresh</code> has been optimized for RO file-servers, where volumes are frequently accessed once a day and soft-detaching has little effect (RO volumes are not salvaged; one of the main reasons for soft detaching).<br />
- <br />
- <br /> ---+++ Vnode Finite-State Automata<br /> The <a href="http://www.dementia.org/twiki//view/dafs-vnode-fsa.jpg" target="_top">vnode finite-state automata</a> was documented by Steven Jenkins.<br />
- <br /><code>/usr/afs/bin/fssync-debug</code> provides low-level inspection and control of the file-server volume package. <strong>Indiscriminate use of <code><b>fsync-debug</b></code> can lead to extremely bad things occurring. Use with care. %ENDCOLOR%<br />
- <br />
- <br /> #SalvageServer<br /> ---+++ Demand Salvaging<br />
- <br /> Demand salvaging is implemented by the <code>salvageserver</code>. The actual code for salvaging a volume remains largely unchanged. However, the method for invoking salvaging with demand-attach has changed:<br /> * file-server automatically requests volumes be salvaged as required, i.e. they are marked as requiring salvaging when attached.<br /> * manual initiation of salvaging may be required when access is through the <code>volserver</code> (may be addressed at some later date).<br /> * <code>bos salvage</code> requires the <code>-forceDAFS</code> flag to initiate salvaging wit DAFS. However, %RED% *salvaging should not be initiated using this method</strong>. %ENDCOLOR%<br /> * infinite salvage, attach, salvage, ... loops are possible. There is therefore a hard-limit on the number of times a volume will be salvaged which is reset when the volume is removed or the file-server is restarted.<br /> * volumes are salvaged in parallel and is controlled by the <code>-Parallel</code> argument to the <code>salvageserver</code>. Defaults to 4.<br /> * the <code>salvageserver</code> and the <code>inode</code> file-server are incompatible:<br /> * because volumes are inter-mingled on a partition (rather than being separated), a lock for the entire partition on which the volume is located is held throughout. Both the <code>fileserver</code> and <code>volserver</code> will block if they require this lock, e.g. to restore / dump a volume located on the partition.<br /> * inodes for a particular volume can be located anywhere on a partition. Salvaging therefore results in <strong>every</strong> inode on a partition having to be read to determine whether it belongs to the volume. This is extremely I/O intensive and leads to horrendous salvaging performance.<br /> * <code>/usr/afs/bin/salvsync-debug</code> provides low-level inspection and control over the <code>salvageserver</code>. %RED% <strong>Indiscriminate use of <code><b>salvsync-debug</b></code> can lead to extremely bad things occurring. Use with care.</strong> %ENDCOLOR%<br />
- <br />
- <br />
- <br /> #FSStateDat<br /> ---+++ File-Server Host / Callback State<br /> Host / callback information is persistent across restarts with demand-attach. On shutdown, the file-server writes the data to <code>/usr/afs/local/fsstate.dat</code>. The contents of this file are read and verified at start-up and hence it is unnecessary to break callbacks on shutdown with demand-attach.<br />
- <br /> The contents of <code>fsstate.dat</code> can be inspected using <code>/usr/afs/bin/state_analyzer</code>.<br />
- <br />
- <br /> ---++ File-Server Arguments (relating to Demand-Attach)<br />
- <br /> These are available in the man-pages (section 8) for the fileserve; some details are provided here for convenience.<br />
- <br /> Arguments controlling the host / callback state:<br /> | <strong>Parameter</strong> | <strong>Options</strong> | <strong>Default</strong> | <strong>Modified Default</strong> | <strong>Meaning</strong> |<br /> | <code>fs-state-dont-save</code> | n/a | state saved | - | <code>fileserver</code> state will not be saved during shutdown |<br /> | <code>fs-state-dont-restore</code> | n/a | state restored | - | <code>fileserver</code> state will not be restored during startup |<br /> | <code>fs-state-verify</code> | <none %VBAR% save %VBAR^% restore %VBAR% both> | both | - | Controls the behavior of the state verification mechanism. Before saving or restoring the <code>fileserver</code> state information, the internal host and callback data structures are verified. A value of 'none' turns off all verification. A value of 'save' only performs the verification steps prior to saving state to disk. A value of 'restore' only performs the verification steps after restoring state from disk. A value of 'both' performs all verification steps both prior to saving and after restoring state. |<br />
- <br /> Arguments controlling the [[Main/WebHome#VolumeLeastRecentlyUsed]]<br /> | <strong>Parameter</strong> | <strong>Options</strong> | <strong>Default</strong> | <strong>Modified Default</strong> | <strong>Meaning</strong> |<br /> | <code>vattachpar</code> | +int | 1 | 128 | Controls the parallelism of the volume package start-up and shutdown routines. On start-up, vice partitions are scanned for volumes to pre-attach using a number of worker threads, the number of which is the minimum of <code>vattachpar</code> or the number of vice partitions. On shutdown, <code>vattachpar</code> worker threads are used to detach volumes. The shutdown code is mp-scaleable well beyond the number of vice partitions. Tom Keiser (from SNA) found 128 threads for a single vice partition had a statistically significant performance improvement over 64 threads. |<br /> | <code>vhashsize</code> | +int | 8 | 11 | This parameter controls the size of the volume hash table. The table will contain 2^(<code>vhashsize</code>) entries. Hash bucket utilization statistics are given in the <code>fileserver</code> state information as well as on shutdown. |<br /> | <code>vlrudisable</code> | n/a | enabled | - | Disables the Volume Least Recently Used (VLRU) cache. |<br /> | <code>vlruthresh</code> | +int | 120 minutes | 1440 (24 hrs) | Minutes of inactivity before a volume is eligible for soft detachment. |<br /> | <code>vlruinterval</code> | +int | 120 seconds | - | Number of seconds between VLRU candidate queue scans |<br /> | <code>vlrumax</code> | +int | 8 | 8 | Max number of volumes which will be soft detached in a single pass of the scanner |</div>
-<div id=":175">
-</div>
-<div id=":175">
- <p> -- [[Main/StevenJenkins]] - 10 Mar 2008 </p>
- <p> The Demand Attach fileserver is a new architecture for the fileserver, volserver, and salvager that brings volumes online as as needed, and takes volumes offline when they have been idle. This greatly speeds up the fileserver startup and shutdown times. </p>
- <p> The full presentation on the Demand Attach fileserver was given by Tom Keiser at the 2007 AFS Workshop. This entry is merely a summary of key information. </p>
- <p>
- </p>
- <ul>
- <li> The format of the volume and vnode structures did not change; however, the bosserver, fileserver, volserver, salvageserver, salvager, and salvage must be rebuilt (using the --enable-demand_attach_fs argument to configure). </li>
- <li> The fs instance must be removed, and a replacement dafs instance added. See the bos_create(8) man page for details </li>
- <li> There are new options to the fileserver. Note in particular that unless -vattachpar is larger than 1, many of the parallel I/O advantages of DAFS will not be enabled (Tom Keiser notes that 1 thread per partition is the logical extreme during startup; however, during shutdown, more threads can be used, so a value of 128 is not unreasonable). However, tuning this value for a given environment will require experimentation. </li>
- </ul>
- <p> The key engine for offlining volumes is the VLRU (Volume Least Recently Used) queue. This is a garbage collection facility which automatically offlines unused volumes in the background. The process of offlining a volume from the "attached" state to the"pre-attached" state is called "soft detachment". </p>
- <p> VLRU works in a manner similar to a generational garbage collector. There are five queues on which volumes can reside: </p>
- <p>
- </p>
- <ol>
- <li> new </li>
- <li> mid </li>
- <li> old </li>
- <li> held: queue for volumes which are administratively barred from VLRU activity </li>
- <li> candidate: queue for volumes which have not been accessed recently, and thus are canddidates for soft detachment </li>
- </ol>
- <p> The new, mid, and old queues are generational queues for active volumes, with the state transitions among them controlled by the following inactivity rules: </p>
- <p>
- </p>
+## <a name="Demand-Attach File-Server (DAFS)"></a> Demand-Attach File-Server (DAFS)
+OpenAFS 1.5 contains Demand-Attach File-Server (DAFS). DAFS is a significant departure from the more _traditional_ AFS file-server and this document details those changes.
+
+<div>
<ul>
- <li> candidate->new activity present </li>
- <li> new->candidate (1*vlruthresh) minutes since last transition; no activity </li>
- <li> new->mid (2*vlruthresh) minutes since last transition; activity present </li>
- <li> mid->old (4*vlruthresh) minutes since last transition; activity present </li>
- <li> old->mid (2*vlruthresh) minutes since last transition; no activity </li>
- <li> mid->new (1*vlruthresh) minutes since last transition; no activity </li>
+ <li><a href="#Demand-Attach File-Server (DAFS)"> Demand-Attach File-Server (DAFS)</a></li>
+ <li><a href="#Why Demand-Attach File-Server (D"> Why Demand-Attach File-Server (DAFS) ?</a></li>
+ <li><a href="#An Overview of Demand-Attach Fil"> An Overview of Demand-Attach File-Server</a></li>
+ <li><a href="#The Gory Details of the Demand-A"> The Gory Details of the Demand-Attach File-Server</a><ul>
+ <li><a href="#Bos Configuration"> Bos Configuration</a></li>
+ <li><a href="#File-server Start-up / Shutdown"> File-server Start-up / Shutdown Sequence</a></li>
+ <li><a href="#Volume Finite-State Automata"> Volume Finite-State Automata</a></li>
+ <li><a href="#Volume Least Recently Used (VLRU"> Volume Least Recently Used (VLRU) Queues</a></li>
+ <li><a href="#Vnode Finite-State Automata"> Vnode Finite-State Automata</a></li>
+ <li><a href="#Demand Salvaging"> Demand Salvaging</a></li>
+ <li><a href="#File-Server Host / Callback Stat"> File-Server Host / Callback State</a></li>
+ </ul>
+ </li>
+ <li><a href="#File-Server Arguments (relating"> File-Server Arguments (relating to Demand-Attach)</a></li>
+ <li><a href="#Tools for Debugging Demand-Attac"> Tools for Debugging Demand-Attach File-Server</a><ul>
+ <li><a href="#==fssync-debug=="> fssync-debug</a></li>
+ <li><a href="#==salvsync-debug=="> salvsync-debug</a></li>
+ <li><a href="#==state_analyzer=="> state_analyzer</a><ul>
+ <li><a href="#Header Information"> Header Information</a></li>
+ <li><a href="#Host Information"> Host Information</a></li>
+ <li><a href="#Callback Information"> Callback Information</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
</ul>
- <p> The idea is that as a volume is used, it gets pushed into a 'more used' generation (i.e., 'old' is 'more used' than 'mid', which is 'more used' than 'new'). this provides a weighting so that volumes which are used more will tend to be in the mid and old queues, not in the new or candidate queues, thus staying in online. </p>
- <p> Note that the VLRU engine can be controlled and tuned by the vlrudisable, vlruthresh, vlruinterval, and vlrumax parameters to the fileserver. </p>
- <p>
- </p>
</div>
+
+## <a name="Why Demand-Attach File-Server (D"></a> Why Demand-Attach File-Server (DAFS) ?
+
+On a traditional file-server, volumes are attached at start-up and detached only at shutdown. Any attached volume can be modified and changes are periodically flushed to disk or on shutdown. When a file-server isn't shutdown cleanly, the integrity of every attached volume has to be verified by the salvager, whether the volume had been modified or not. As file-servers grow larger (and the number of volumes increase), the length of time required to salvage and attach volumes increases, e.g. it takes around two hours for a file-server housing 512GB data to salvage and attach volumes !
+
+On a Demand-Attach File-Server (DAFS), volumes are attached only when accessed by clients. On start-up, the file-server reads only the volume headers to determine what volumes reside on what partitions. When accessed by clients, the volumes are attached. After some period of inactivity, volumes are automatically detached. This dramatically improves start-up and shutdown times. A demand-attach file-server can be restarted in seconds compared to hours for the same traditional file-server.
+
+The primary objective of the demand-attach file-server was to dramatically reduce the amount of time required to restart an AFS file-server.
+
+Large portions of this document were taken / influenced by the presentation entitled [Demand Attach / Fast-Restart Fileserver](http://workshop.openafs.org/afsbpw06/talks/tkeiser-dafs.pdf) given by Tom Keiser at the [AFS and Kerberos Best Practices Workshop](http://workshop.openafs.org/) in [2006](http://workshop.openafs.org/afsbpw06/).
+
+## <a name="An Overview of Demand-Attach Fil"></a> An Overview of Demand-Attach File-Server
+
+Demand-attach necessitated a significant re-design of certain aspects of the AFS code, including:
+
+- volume package has a number of severe limitations
+ - single global lock, leading to poor scaling
+ - lock is held across high latency operations, e.g. disk I/O
+ - no notion of state for concurrently accessed objects
+- the vnode package suffers from the same limitations
+- breaking callbacks is time consuming
+- salvaging does not have to be done with the file-server offline
+
+The changes implemented for demand-attach include:
+
+- [[volume finite-state automata|AFSLore/DemandAttach#Volume_Finite_State_Automata]]
+- volumes are attached on demand
+- volume _garbage collector_ to detach unused volumes
+- notion of volume state means read-only volumes aren't salvaged
+- [[vnode finite-state automata|AFSLore/DemandAttach#Vnode_Finite_State_Automata]]
+- global lock is only held when required and never held across high-latency operations
+- automatic salvaging of volumes
+- shutdown is done in parallel (maximum number of threads utilized)
+- callbacks are no longer broken on shutdown
+- instead, host / callback state is preserved across restarts
+
+## <a name="The Gory Details of the Demand-A"></a> The Gory Details of the Demand-Attach File-Server
+
+### <a name="Bos Configuration"></a> Bos Configuration
+A traditional file-server uses the `bnode` type `fs` and has a definition similar to
+
+ bnode fs fs 1
+ parm /usr/afs/bin/fileserver -p 123 -pctspare 20 -L -busyat 200 -rxpck 2000 -rxbind
+ parm /usr/afs/bin/volserver -p 127 -log -rxbind
+ parm /usr/afs/bin/salvager -parallel all32
+ end
+
+Since an additional component was required for the demand-attach file-server, a new `bnode` type ( `dafs`) is required. The definition should be similar to
+
+ bnode dafs dafs 1
+ parm /usr/afs/bin/fileserver -p 123 -pctspare 20 -L -busyat 50 -rxpck 2000 -rxbind -cb 4000000 -vattachpar 128 -vlruthresh 1440 -vlrumax 8 -vhashsize 11
+ parm /usr/afs/bin/volserver -p 64 -log -rxbind
+ parm /usr/afs/bin/salvageserver
+ parm /usr/afs/bin/salvager -parallel all32
+ end
+
+The instance for a demand-attach file-server is therefore `dafs` instead of `fs`.
+
+### <a name="File-server Start-up / Shutdown"></a><a name="File-server Start-up / Shutdown "></a> File-server Start-up / Shutdown Sequence
+
+The table below compares the start-up sequence for a traditional file-server and a demand-attach file-server.
+
+<table border="1" cellpadding="0" cellspacing="0">
+ <tr>
+ <th bgcolor="#99CCCC"><strong> Traditional </strong></th>
+ <th bgcolor="#99CCCC"><strong> Demand-Attach </strong></th>
+ </tr>
+ <tr>
+ <td> </td>
+ <td> %BULLET% host / callback state restored </td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td> %BULLET% host / callback state consistency verified </td>
+ </tr>
+ <tr>
+ <td> %BULLET% build vice partition list </td>
+ <td> %BULLET% build vice partition list </td>
+ </tr>
+ <tr>
+ <td> %BULLET% volumes are attached </td>
+ <td> %BULLET% volume headers read </td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td> %BULLET% volumes placed into <em>pre-attached</em> state </td>
+ </tr>
+</table>
+
+The [[host / callback state|AFSLore/DemandAttach#FSStateDat]] is covered later. The _pre-attached_ state indicates that the file-server has read the volume headers and is aware that the volume exists, but that it has not been attached (and hence is not on-line).
+
+The shutdown sequence for both file-server types is:
+
+<table border="1" cellpadding="0" cellspacing="0">
+ <tr>
+ <th bgcolor="#99CCCC"><strong> Traditional </strong></th>
+ <th bgcolor="#99CCCC"><strong> Demand-Attach </strong></th>
+ </tr>
+ <tr>
+ <td> %BULLET% break callbacks </td>
+ <td> %BULLET% quiesce host / callback state </td>
+ </tr>
+ <tr>
+ <td> %BULLET% shutdown volumes </td>
+ <td> %BULLET% shutdown on-line volumes </td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td> %BULLET% verify host / callback state consistency </td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td> %BULLET% save host / callback state </td>
+ </tr>
+</table>
+
+On a traditional file-server, volumes are off-lined (detached) serially. In demand-attach, as many threads as possible are used to detach volumes, which is possible due to the notion of a volume has an associated state.
+
+### <a name="Volume Finite-State Automata"></a> Volume Finite-State Automata
+
+The volume finite-state automata is available in the source tree under `doc/arch/dafs-fsa.dot`. See [[=fssync-debug=|AFSLore/DemandAttach#fssync_debug]] for information on debugging the volume package.
+
+<a name="VolumeLeastRecentlyUsed"></a>
+
+### <a name="Volume Least Recently Used (VLRU"></a> Volume Least Recently Used (VLRU) Queues
+
+The Volume Least Recently Used (VLRU) is a garbage collection facility which automatically off-lines volumes in the background. The purpose of this facility is to pro-actively off-line infrequently used volumes to improve shutdown and salvage times. The process of off-lining a volume from the "attached" state to the "pre-attached" state is called soft detachment.
+
+VLRU works in a manner similar to a generational garbage collector. There are five queues on which volumes can reside.
+
+<table border="1" cellpadding="0" cellspacing="0">
+ <tr>
+ <th bgcolor="#99CCCC"><strong> Queue </strong></th>
+ <th bgcolor="#99CCCC"><strong> Meaning </strong></th>
+ </tr>
+ <tr>
+ <td> candidate </td>
+ <td> Volumes which have not been accessed recently and hence are candidates for soft detachment. </td>
+ </tr>
+ <tr>
+ <td> held </td>
+ <td> Volumes which are administratively prevented from VLRU activity, i.e. will never be detached. </td>
+ </tr>
+ <tr>
+ <td> intermediate (mid) </td>
+ <td> Volumes transitioning from new -> old (see [[AFSLore.DemandAttach#VLRUStateTransitions][state transitions] for details). </td>
+ </tr>
+ <tr>
+ <td> new </td>
+ <td> Volumes which have been accessed. See [[AFSLore.DemandAttach#VLRUStateTransitions][state transitions] for details. </td>
+ </tr>
+ <tr>
+ <td> old </td>
+ <td> Volumes which are continually accessed. See [[AFSLore.DemandAttach.#VLRUStateTransitions][state transitions] for details. </td>
+ </tr>
+</table>
+
+The state of the various VLRU queues is dumped with the file-server state and at shutdown.
+
+<a name="VLRUStateTransitions"></a> The VLRU queues new, mid (intermediate) and old are generational queues for active volumes. State transitions are controlled by inactivity timers and are
+
+<table border="1" cellpadding="0" cellspacing="0">
+ <tr>
+ <th bgcolor="#99CCCC"><strong> Transition </strong></th>
+ <th bgcolor="#99CCCC"><strong> Timeout (minutes) </strong></th>
+ <th bgcolor="#99CCCC"><strong> Actual Timeout (in MS) </strong></th>
+ <th bgcolor="#99CCCC"><strong> Reason (since last transition) </strong></th>
+ </tr>
+ <tr>
+ <td> candidate->new </td>
+ <td> - </td>
+ <td> - </td>
+ <td> new activity </td>
+ </tr>
+ <tr>
+ <td> new->candidate </td>
+ <td> 1 * vlruthresh </td>
+ <td> 24 hrs </td>
+ <td> no activity </td>
+ </tr>
+ <tr>
+ <td> new->mid </td>
+ <td> 2 * vlruthresh </td>
+ <td> 48 hrs </td>
+ <td> activity </td>
+ </tr>
+ <tr>
+ <td> mid->old </td>
+ <td> 4 * vlruthresh </td>
+ <td> 96 hrs </td>
+ <td> activity </td>
+ </tr>
+ <tr>
+ <td> old->mid </td>
+ <td> 2 * vlruthresh </td>
+ <td> 48 hrs </td>
+ <td> no activity </td>
+ </tr>
+ <tr>
+ <td> mid->new </td>
+ <td> 1 * vlruthresh </td>
+ <td> 24 hrs </td>
+ <td> no activity </td>
+ </tr>
+</table>
+
+`vlruthresh` has been optimized for RO file-servers, where volumes are frequently accessed once a day and soft-detaching has little effect (RO volumes are not salvaged; one of the main reasons for soft detaching).
+
+### <a name="Vnode Finite-State Automata"></a> Vnode Finite-State Automata
+
+The vnode finite-state automata is available in the source tree under `doc/arch/dafs-vnode-fsa.dot`
+
+`/usr/afs/bin/fssync-debug` provides low-level inspection and control of the file-server volume package. \*Indiscriminate use of <code>**fsync-debug**</code> can lead to extremely bad things occurring. Use with care. %ENDCOLOR%
+
+<a name="SalvageServer"></a>
+
+### <a name="Demand Salvaging"></a> Demand Salvaging
+
+Demand salvaging is implemented by the `salvageserver`. The actual code for salvaging a volume remains largely unchanged. However, the method for invoking salvaging with demand-attach has changed:
+
+- file-server automatically requests volumes be salvaged as required, i.e. they are marked as requiring salvaging when attached.
+- manual initiation of salvaging may be required when access is through the `volserver` (may be addressed at some later date).
+- `bos salvage` requires the `-forceDAFS` flag to initiate salvaging wit DAFS. However, %RED% **salvaging should not be initiated using this method**.%ENDCOLOR%
+- infinite salvage, attach, salvage, ... loops are possible. There is therefore a hard-limit on the number of times a volume will be salvaged which is reset when the volume is removed or the file-server is restarted.
+- volumes are salvaged in parallel and is controlled by the `-Parallel` argument to the `salvageserver`. Defaults to 4.
+- the `salvageserver` and the `inode` file-server are incompatible:
+ - because volumes are inter-mingled on a partition (rather than being separated), a lock for the entire partition on which the volume is located is held throughout. Both the `fileserver` and `volserver` will block if they require this lock, e.g. to restore / dump a volume located on the partition.
+ - inodes for a particular volume can be located anywhere on a partition. Salvaging therefore results in **every** inode on a partition having to be read to determine whether it belongs to the volume. This is extremely I/O intensive and leads to horrendous salvaging performance.
+- `/usr/afs/bin/salvsync-debug` provides low-level inspection and control over the `salvageserver`. %RED% **Indiscriminate use of `salvsync-debug` can lead to extremely bad things occurring. Use with care.** %ENDCOLOR%
+- See [[=salvsync-debug=|AFSLore/DemandAttach#salvsync_debug]] for information on debugging problems with the salvageserver.
+
+<a name="FSStateDat"></a>
+
+### <a name="File-Server Host / Callback Stat"></a> File-Server Host / Callback State
+
+Host / callback information is persistent across restarts with demand-attach. On shutdown, the file-server writes the data to `/usr/afs/local/fsstate.dat`. The contents of this file are read and verified at start-up and hence it is unnecessary to break callbacks on shutdown with demand-attach.
+
+The contents of `fsstate.dat` can be inspected using `/usr/afs/bin/state_analyzer`.
+
+## <a name="File-Server Arguments (relating"></a><a name="File-Server Arguments (relating "></a> File-Server Arguments (relating to Demand-Attach)
+
+These are available in the man-pages (section 8) for the fileserver; some details are provided here for convenience:
+
+Arguments controlling the host / callback state:
+
+<table border="1" cellpadding="0" cellspacing="0">
+ <tr>
+ <th bgcolor="#99CCCC"><strong> Parameter </strong></th>
+ <th bgcolor="#99CCCC"><strong> Options </strong></th>
+ <th bgcolor="#99CCCC"><strong> Default </strong></th>
+ <th bgcolor="#99CCCC"><strong> Suggested Value </strong></th>
+ <th bgcolor="#99CCCC"><strong> Meaning </strong></th>
+ </tr>
+ <tr>
+ <td><code>fs-state-dont-save</code></td>
+ <td> n/a </td>
+ <td> state saved </td>
+ <td> - </td>
+ <td><code>fileserver</code> state will not be saved during shutdown </td>
+ </tr>
+ <tr>
+ <td><code>fs-state-dont-restore</code></td>
+ <td> n/a </td>
+ <td> state restored </td>
+ <td> - </td>
+ <td><code>fileserver</code> state will not be restored during startup </td>
+ </tr>
+ <tr>
+ <td><code>fs-state-verify</code></td>
+ <td><none %vbar%="%VBAR%" %vbar^%="%VBAR^%" both="both" restore="restore" save="save"> </none></td>
+ <td> both </td>
+ <td> - </td>
+ <td> Controls the behavior of the state verification mechanism. Before saving or restoring the <code>fileserver</code> state information, the internal host and callback data structures are verified. A value of 'none' turns off all verification. A value of 'save' only performs the verification steps prior to saving state to disk. A value of 'restore' only performs the verification steps after restoring state from disk. A value of 'both' performs all verification steps both prior to saving and after restoring state. </td>
+ </tr>
+</table>
+
+Arguments controlling the [[VLRU:|Main/WebHome#VolumeLeastRecentlyUsed]]
+
+<table border="1" cellpadding="0" cellspacing="0">
+ <tr>
+ <th bgcolor="#99CCCC"><strong> Parameter </strong></th>
+ <th bgcolor="#99CCCC"><strong> Options </strong></th>
+ <th bgcolor="#99CCCC"><strong> Default </strong></th>
+ <th bgcolor="#99CCCC"><strong> Suggested Value </strong></th>
+ <th bgcolor="#99CCCC"><strong> Meaning </strong></th>
+ </tr>
+ <tr>
+ <td><code>vattachpar</code></td>
+ <td> positive integer </td>
+ <td> 1 </td>
+ <td> 128 </td>
+ <td> Controls the parallelism of the volume package start-up and shutdown routines. On start-up, vice partitions are scanned for volumes to pre-attach using a number of worker threads, the number of which is the minimum of <code>vattachpar</code> or the number of vice partitions. On shutdown, <code>vattachpar</code> worker threads are used to detach volumes. The shutdown code is mp-scaleable well beyond the number of vice partitions. Tom Keiser (from SNA) found 128 threads for a single vice partition had a statistically significant performance improvement over 64 threads. </td>
+ </tr>
+ <tr>
+ <td><code>vhashsize</code></td>
+ <td> positive integer </td>
+ <td> 8 </td>
+ <td> 11 </td>
+ <td> This parameter controls the size of the volume hash table. The table will contain 2^( <code>vhashsize</code>) entries. Hash bucket utilization statistics are given in the <code>fileserver</code> state information as well as on shutdown. </td>
+ </tr>
+ <tr>
+ <td><code>vlrudisable</code></td>
+ <td> n/a </td>
+ <td> enabled </td>
+ <td> - </td>
+ <td> Disables the Volume Least Recently Used (VLRU) cache. </td>
+ </tr>
+ <tr>
+ <td><code>vlruthresh</code></td>
+ <td> positive integer </td>
+ <td> 120 minutes </td>
+ <td> 1440 (24 hrs) </td>
+ <td> Minutes of inactivity before a volume is eligible for soft detachment. </td>
+ </tr>
+ <tr>
+ <td><code>vlruinterval</code></td>
+ <td> positive integer </td>
+ <td> 120 seconds </td>
+ <td> - </td>
+ <td> Number of seconds between VLRU candidate queue scans </td>
+ </tr>
+ <tr>
+ <td><code>vlrumax</code></td>
+ <td> positive integer </td>
+ <td> 8 </td>
+ <td> 8 </td>
+ <td> Max number of volumes which will be soft detached in a single pass of the scanner </td>
+ </tr>
+</table>
+
+## <a name="Tools for Debugging Demand-Attac"></a> Tools for Debugging Demand-Attach File-Server
+
+Several tools aid debugging problems with demand-attach file-servers. They operate at an extremely low-level and hence require a detailed knowledge of the architecture / code.
+
+### <a name="==fssync-debug=="></a> <code>**fssync-debug**</code>
+
+%RED% **Indiscriminate use of `fssync-debug` can have extremely dire consequences. Use with care** %ENDCOLOR%
+
+`fssync-debug` provides low-level inspection and control over the volume package of the file-server. It can be used to display the file-server information associated with a volume, e.g.
+
+ ozsaw7 2# vos exam user.af -cell w.ln
+ user.af 537119916 RW 2123478 K On-line
+ ln1qaf01 /vicepb
+ RWrite 537119916 ROnly 0 Backup 537119917
+ MaxQuota 3200000 K
+ Creation Wed Sep 17 17:48:17 2003
+ Copy Thu Dec 11 18:01:37 2008
+ Backup Thu Jun 25 01:49:20 2009
+ Last Update Thu Jun 25 16:17:35 2009
+ 85271 accesses in the past day (i.e., vnode references)
+
+ RWrite: 537119916 Backup: 537119917
+ number of sites -> 1
+ server ln1qaf01 partition /vicepb RW Site
+ ozsaw7 3# /usr/afs/bin/fssync-debug query -vol 537119916 -part /vicepb
+ calling FSYNC_VolOp with command code 65543 (FSYNC_VOL_QUERY)
+ FSYNC_VolOp returned 0 (SYNC_OK)
+ protocol response code was 0 (SYNC_OK)
+ protocol reason code was 0 (0)
+ volume = {
+ hashid = 537119916
+ header = 0xf93f160
+ device = 1
+ partition = 0xf90dfb8
+ linkHandle = 0x10478400
+ nextVnodeUnique = 2259017
+ diskDataHandle = 0x104783d0
+ vnodeHashOffset = 14
+ shuttingDown = 0
+ goingOffline = 0
+ cacheCheck = 49167
+ nUsers = 0
+ needsPutBack = 0
+ specialStatus = 0
+ updateTime = 1245943107
+ vnodeIndex[vSmall] = {
+ handle = 0x104783a0
+ bitmap = 0x13f44df0
+ bitmapSize = 2792
+ bitmapOffset = 64
+ }
+ vnodeIndex[vLarge] = {
+ handle = 0x10478370
+ bitmap = 0x13c96040
+ bitmapSize = 296
+ bitmapOffset = 252
+ }
+ updateTime = 1245943107
+ attach_state = VOL_STATE_ATTACHED
+ attach_flags = VOL_HDR_ATTACHED | VOL_HDR_LOADED | VOL_HDR_IN_LRU | VOL_IN_HASH | VOL_ON_VBYP_LIST | VOL_ON_VLRU
+ nWaiters = 0
+ chainCacheCheck = 14
+ salvage = {
+ prio = 0
+ reason = 0
+ requested = 0
+ scheduled = 0
+ }
+ stats = {
+ hash_lookups = {
+ hi = 0
+ lo = 459560
+ }
+ hash_short_circuits = {
+ hi = 0
+ lo = 30
+ }
+ hdr_loads = {
+ hi = 0
+ lo = 32
+ }
+ hdr_gets = {
+ hi = 0
+ lo = 456011
+ }
+ attaches = 32
+ soft_detaches = 0
+ salvages = 1
+ vol_ops = 32
+ last_attach = 1245891030
+ last_get = 1245943107
+ last_promote = 1245891030
+ last_hdr_get = 1245943107
+ last_hdr_load = 1245891030
+ last_salvage = 1242508846
+ last_salvage_req = 1242508846
+ last_vol_op = 1245890958
+ }
+ vlru = {
+ idx = 0 (VLRU_QUEUE_NEW)
+ }
+ pending_vol_op = 0x0
+ }
+
+Note that the `volumeid` argument must be the numeric ID and the `partition` argument must be the **exact** partition name (and not an abbreviation). An explanation of all these values is beyond the scope of this document. The important fields are:
+
+- `attach_state`, which is usually
+ - `VOL_STATE_PREATTACHED` which means the volume headers have been read, but the volume is not attached
+ - `VOL_STATE_ATTACHED` which means the volume is fully attached
+ - `VOL_STATE_ERROR` which indicates that the volume cannot be attached
+- `attach_flags`
+ - `VOL_HDR_ATTACHED` means the volume headers have been read (and hence the file-server is aware of the volumes existence)
+ - `VOL_HDR_LOADED` means the volume headers are resident in memory
+ - `VOL_HDR_IN_LRU` means the volume headers are on the least-recently used queue
+ - `VOL_IN_HASH` indicates that the volume has been added to the volume linked-list
+ - `VOL_ON_VBYP_LIST` indicates that the volume is linked off the partition list
+ - `VOL_ON_VLRU` means the volume is on a VLRU queue
+- the `salvage` structure (detailed [[here|AFSLore/DemandAttach#salvsync_debug]])
+- the `stats` structure, particularly the volume operation times ( `last_*`).
+- the `vlru` structure, particularly the VLRU queue
+
+An understanding of the [volume finite-state machine](http://www.dementia.org/twiki//view/dafs-fsa.png) is required before the state of a volume should be manipulated.
+
+### <a name="==salvsync-debug=="></a> <code>**salvsync-debug**</code>
+
+%RED% **Indiscriminate use of `salvsync-debug` can have extremely dire consequences. Use with care** %ENDCOLOR%
+
+`salvsync-debug` provides low-level inspection and control of the salvageserver process, including the scheduling order of volumes.
+
+`salvsync-debug` can be used to query the current salvage status of a volume, e,g,
+
+ ozsaw7 4# /usr/afs/bin/salvsync-debug query -vol 537119916 -part /vicepb
+ calling SALVSYNC_SalvageVolume with command code 65540 (SALVSYNC_QUERY)
+ SALVSYNC_SalvageVolume returned 0 (SYNC_OK)
+ protocol response code was 0 (SYNC_OK)
+ protocol reason code was 0 (**UNKNOWN**)
+ state = {
+ state = 4 (SALVSYNC_STATE_DONE)
+ prio = 0
+ sq_len = 0
+ pq_len = 0
+ }
+
+To initiate the salvaging of a volume
+
+ ozsaw7 5# /usr/afs/bin/salvsync-debug salvage -vol 537119916 -part /vicepb
+ calling SALVSYNC_SalvageVolume with command code 65537 (SALVSYNC_SALVAGE)
+ SALVSYNC_SalvageVolume returned 0 (SYNC_OK)
+ protocol response code was 0 (SYNC_OK)
+ protocol reason code was 0 (**UNKNOWN**)
+ state = {
+ state = 1 (SALVSYNC_STATE_QUEUED)
+ prio = 0
+ sq_len = 1
+ pq_len = 0
+ }
+ ozsaw7 6# /usr/afs/bin/salvsync-debug query -vol 537119916 -part /vicepb calling SALVSYNC_SalvageVolume with command code 65540 (SALVSYNC_QUERY)
+ SALVSYNC_SalvageVolume returned 0 (SYNC_OK)
+ protocol response code was 0 (SYNC_OK)
+ protocol reason code was 0 (**UNKNOWN**)
+ state = {
+ state = 2 (SALVSYNC_STATE_SALVAGING)
+ prio = 0
+ sq_len = 0
+ pq_len = 1
+ }
+
+This is the method that should be used on demand-attach file-servers to initiate the manual salvage of volumes. It should be used with care.
+
+Under normal circumstances, the priority ( `prio`) of a salvage request is the number of times the volume has been requested by clients. %RED% Modifying the priority (and hence the order volumes are salvaged) under heavy demand-salvaging usually leads to extremely bad things happening. %ENDCOLOR% To modify the priority of a request, use
+
+ salvsync-debug priority -vol 537119916 -part /vicepb -priority 999999
+
+(where `priority` is a 32-bit integer).
+
+### <a name="==state_analyzer=="></a> <code>**state\_analyzer**</code>
+
+`state_analyzer` allows the contents of the host / callback state file ( `/usr/afs/local/fsstate.dat`) to be inspected.
+
+#### <a name="Header Information"></a> Header Information
+
+Header information is gleaned through the `hdr` command
+
+ fs state analyzer> hdr
+ loading structure from address 0xfed80000 (offset 0)
+ fs_state_header = {
+ stamp = {
+ magic = 0x62fa841c
+ version = 2
+ }
+ timestamp = "Tue Jun 23 11:51:49 2009"
+ sys_name = 941
+ server_uuid = "002e9712-ae67-1a2e-8a-42-900e866eaa77"
+ valid = 0
+ endianness = 1
+ stats_detailed = 1
+ h_offset = {
+ hi = 0
+ lo = 1024
+ }
+ cb_offset = {
+ hi = 0
+ lo = 93928
+ }
+ server_version_string = "@(#) OpenAFS 1.4.6-22 built 2009-04-18 "
+ }
+ fs state analyzer>
+
+#### <a name="Host Information"></a> Host Information
+
+Host information can be gleaned through the `h` command, e.g.
+
+ fs state analyzer> h
+ fs state analyzer: h(0)> this
+ loading structure from address 0xfed80500 (offset 1280)
+ host_state_entry_header = {
+ magic = 0xa8b9cadb
+ len = 104
+ interfaces = 2
+ hcps = 0
+ }
+ hostDiskEntry = {
+ host = "161.144.167.187"
+ port = 7001
+ hostFlags = 0x144
+ Console = 0
+ hcpsfailed = 0
+ hcps_valid = 0
+ InSameNetwork = 0
+ hcps_len = 0
+ LastCall = "Tue Jun 23 11:51:45 2009"
+ ActiveCall = "Tue Jun 23 11:51:45 2009"
+ cpsCall = "Tue Jun 23 11:51:45 2009"
+ cblist = 21133
+ index = 373
+ }
+ Interface = {
+ numberOfInterfaces = 2
+ uuid = "aae8a851-1d54-4b83-ad-17-db967bd89e1b"
+ interface[0] = {
+ addr = "161.144.167.187"
+ port = 7001
+ }
+ interface[1] = {
+ addr = "192.168.8.4"
+ port = 7001
+ }
+ }
+ fs state analyzer: h(0)> next
+ loading structure from address 0xfed80568 (offset 1384)
+ host_state_entry_header = {
+ magic = 0xa8b9cadb
+ len = 120
+ interfaces = 4
+ hcps = 0
+ }
+ hostDiskEntry = {
+ host = "10.181.34.134"
+ port = 7001
+ hostFlags = 0x144
+ Console = 0
+ hcpsfailed = 0
+ hcps_valid = 0
+ InSameNetwork = 0
+ hcps_len = 0
+ LastCall = "Tue Jun 23 11:51:08 2009"
+ ActiveCall = "Tue Jun 23 11:51:08 2009"
+ cpsCall = "Tue Jun 23 11:51:08 2009"
+ cblist = 8422
+ index = 369
+ }
+ Interface = {
+ numberOfInterfaces = 4
+ uuid = "00107e94-794d-1a3d-ae-e2-0ab52421aa77"
+ interface[0] = {
+ addr = "10.181.36.33"
+ port = 7001
+ }
+ interface[1] = {
+ addr = "10.181.36.31"
+ port = 7001
+ }
+ interface[2] = {
+ addr = "10.181.32.134"
+ port = 7001
+ }
+ interface[3] = {
+ addr = "10.181.34.134"
+ port = 7001
+ }
+ }
+ fs state analyzer: h(1)>
+
+#### <a name="Callback Information"></a> Callback Information
+
+Callback information is available through the `cb` command, e.g.
+
+ fs state analyzer> cb
+ fs state analyzer: fe(0):cb(0)> dump
+ loading structure from address 0xfed97b6c (offset 97132)
+ CBDiskEntry = {
+ cb = {
+ cnext = 0
+ fhead = 19989
+ thead = 103
+ status = 1
+ hhead = 224
+ tprev = 12276
+ tnext = 22836
+ hprev = 12276
+ hnext = 22836
+ }
+ index = 6774
+ }
+
+The `dump` command (as opposed to `this`) displays all call-backs for the current file-entry. Moving to the next file-entry can be achieved by
+
+ fs state analyzer: fe(0):cb(0)> quit
+ fs state analyzer: fe(0)> next
+ loading structure from address 0xfed97b90 (offset 97168)
+ callback_state_entry_header = {
+ magic = 0x54637281
+ len = 104
+ nCBs = 1
+ }
+ FEDiskEntry = {
+ fe = {
+ vnode = 46426
+ unique = 125874
+ volid = 537156174
+ fnext = 8880
+ ncbs = 1
+ firstcb = 23232
+ status = 0
+ }
+ index = 21352
+ }
+ fs state analyzer: fe(1)> cb
+ fs state analyzer: fe(1):cb(0)> dump
+ loading structure from address 0xfed97bd4 (offset 97236)
+ CBDiskEntry = {
+ cb = {
+ cnext = 0
+ fhead = 21352
+ thead = 103
+ status = 1
+ hhead = 382
+ tprev = 23751
+ tnext = 22772
+ hprev = 23751
+ hnext = 7824
+ }
+ index = 23232
+ }
git://git.openafs.org / openafs-wiki.git / blobdiff