1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
5 >Monitoring and Auditing AFS Performance</TITLE
8 CONTENT="Modular DocBook HTML Stylesheet Version 1.7"><LINK
10 TITLE="AFS Administration Guide"
11 HREF="book1.html"><LINK
13 TITLE="Managing File Server Machines"
14 HREF="p3023.html"><LINK
16 TITLE="Backing Up and Restoring AFS Data"
17 HREF="c15383.html"><LINK
19 TITLE="Managing Server Encryption Keys"
20 HREF="c20494.html"></HEAD
31 SUMMARY="Header navigation table"
40 >AFS Administration Guide: Version 3.6</TH
77 >Chapter 8. Monitoring and Auditing AFS Performance</H1
79 >AFS comes with three main monitoring tools: <UL
88 > program, which monitors and gathers statistics on File Server
99 > command suite, which traces Cache Manager operations in detail.</P
109 > program, which monitors and gathers statistics on both the File Server
110 and the Cache Manager.</P
115 >AFS also provides a tool for auditing AFS events on file server machines running AIX.</P
122 >Summary of Instructions</A
125 >This chapter explains how to perform the following tasks by using the indicated commands:</P
127 CLASS="informaltable"
140 >Initialize the <SPAN
158 >Display information about a trace log</TD
170 >Display information about an event set</TD
182 >Change the size of a trace log</TD
194 >Set the state of an event set</TD
206 >Dump contents of a trace log</TD
218 >Clear a trace log</TD
230 >Initialize the <SPAN
256 >Using the scout Program</A
265 > program monitors the status of the File Server process running on file server
266 machines. It periodically collects statistics from a specified set of File Server processes, displays them in a graphical
267 format, and alerts you if any of the statistics exceed a configurable threshold.</P
269 >More specifically, the <SPAN
275 > program includes the following features. <UL
278 >You can monitor, from a single location, the File Server process on any number of server machines from the local and
279 foreign cells. The number is limited only by the size of the display window, which must be large enough to display the
284 >You can set a threshold for many of the statistics. When the value of a statistic exceeds the threshold, the
291 > program highlights it (displays it in reverse video) to draw your attention to it.
292 If the value goes back under the threshold, the highlighting is deactivated. You control the thresholds, so highlighting
293 reflects what you consider to be a noteworthy situation. See <A
294 HREF="c18360.html#HDRWQ332"
295 >Highlighting Significant
307 > program alerts you to File Server process, machine, and network outages
308 by highlighting the name of each machine that does not respond to its probe, enabling you to respond more quickly.</P
312 >You can set how often the <SPAN
318 > program collects statistics from the File Server
329 >System Requirements</A
338 > program runs on any AFS client machine that has access to the <SPAN
344 > graphics package, which most UNIX distributions include as a standard utility. It can run on
345 both dumb terminals and under windowing systems that emulate terminals, but the output looks best on machines that support
346 reverse video and cursor addressing. For best results, set the TERM environment variable to the correct terminal type, or one
347 with characteristics similar to the actual ones. For machines running AIX, the recommended TERM setting is <SPAN
353 >, assuming the terminal is similar to that. For other operating systems, the wider range of
354 acceptable values includes <SPAN
386 >No privilege is required to run the <SPAN
392 > program, so any user who can access the
393 directory where its binary resides (the <SPAN
399 > directory in the conventional
400 configuration) can use it. The program's probes for collecting statistics do not impose a significant burden on the File
401 Server process, but you can restrict its use by placing the binary file in a directory with a more restrictive access control
404 >Multiple instances of the <SPAN
410 > program can run on a single client machine, each over
411 its own dedicated connection (in its own window). It must run in the foreground, so the window in which it runs does not
412 accept further input except for an interrupt signal.</P
414 >You can also run the <SPAN
420 > program on several machines and view its output on a single
421 machine, by opening telnet connections to the other machines from the central one and initializing the program in each remote
422 window. In this case, you can include the <SPAN
434 > command to make the name of each remote machine appear in the <SPAN
441 the top of the window displaying its output. See <A
442 HREF="c18360.html#HDRWQ330"
452 >Using the -basename argument to Specify a Domain Name</A
455 >As previously mentioned, the <SPAN
461 > program can monitor the File Server process on any
462 number of file server machines. If all of the machines belong to the same cell, then their hostnames probably all have the
463 same domain name suffix, such as <SPAN
469 > in the ABC Corporation cell. In this case, you can
476 > argument to the <SPAN
483 several advantages: <UL
486 >You can omit the domain name suffix as you enter each file server machine's name on the command line. The
493 > program automatically appends the domain name to each machine's name, resulting
494 in a fully-qualified hostname. You can omit the domain name suffix even when you don't include the <SPAN
500 > argument, but in that case correct resolution of the name depends on the state of your
501 cell's naming service at the time of connection.</P
505 >The machine names are more likely to fit in the appropriate column of the display without having to be truncated
506 (for more on truncating names in the display column, see <A
507 HREF="c18360.html#HDRWQ331"
508 >The Statistics Display
514 >The domain name appears in the banner line at the top of the display window to indicate the name of the cell you
526 >The Layout of the scout Display</A
535 > program can display statistics either in a dedicated window or on a plain
536 screen if a windowing environment is not available. For best results, use a window or screen that can print in reverse video
537 and do cursor addressing.</P
545 > program screen has three main regions: the <SPAN
556 >statistics display region</I
564 > line. This section describes
565 their contents, and graphic examples appear in <A
566 HREF="c18360.html#HDRWQ336"
567 >Example Commands and Displays</A
578 >By default, the string <SAMP
579 CLASS="computeroutput"
581 > appears in the banner line at the top of the window or
582 screen, to indicate that the <SPAN
588 > program is running. You can display two additional types
589 of information by include the appropriate option on the command line: <UL
598 > flag to display the local machine's name in the banner line.
599 This is particularly useful when you are running the <SPAN
606 machines but displaying the results on a single machine.</P
608 >For example, the following banner line appears when you run the <SPAN
630 CLASS="programlisting"
631 > [client1.abc.com] scout
642 > argument to display the specified cell domain name in the
643 banner line. For further discussion, see <A
644 HREF="c18360.html#HDRWQ328"
645 >Using the -basename argument to Specify a Domain
649 >For example, if you specify a value of <SPAN
661 > argument, the banner line reads:</P
663 CLASS="programlisting"
664 > scout for abc.com
676 >The Statistics Display Region</A
679 >The statistics display region occupies most of the window and is divided into six columns. The following list
680 describes them as they appear from left to right in the window. <DIV
685 CLASS="computeroutput"
690 >Displays the number of RPC connections open between the File Server process and client machines. This number
691 normally equals or exceeds the number in the fourth <SAMP
692 CLASS="computeroutput"
694 > column. It can exceed the
695 number in that column because each user on the machine can have more than one connection open at once, and one
696 client machine can handle several users.</P
700 CLASS="computeroutput"
705 >Displays the number of fetch-type RPCs (fetch data, fetch access list, and fetch status) that the File Server
706 process has received from client machines since it started. It resets to zero when the File Server process
711 CLASS="computeroutput"
716 >Displays the number of store-type RPCs (store data, store access list, and store status) that the File Server
717 process has received from client machines since it started. It resets to zero when the File Server process
722 CLASS="computeroutput"
727 >Displays the number of client machines (workstations) that have communicated with the File Server process
728 within the last 15 minutes (such machines are termed <SPAN
734 >). This number is likely to be
735 smaller than the number in the <SAMP
736 CLASS="computeroutput"
738 >) column because a single client machine can
739 have several connections open to one File Server process.</P
746 >[Unlabeled column]</B
751 >Displays the name of the file server machine on which the File Server process is running. It is 12 characters
752 wide. Longer names are truncated and an asterisk (<SAMP
753 CLASS="computeroutput"
755 >) appears as the last character
756 in the name. If all machines have the same domain name suffix, you can use the <SPAN
762 > argument to decrease the need for truncation; see <A
763 HREF="c18360.html#HDRWQ328"
765 the -basename argument to Specify a Domain Name</A
770 CLASS="computeroutput"
775 >Displays the number of kilobyte blocks available on up to 26 of the file server machine's AFS server
782 >) partitions. The display for each partition has the following format:
784 CLASS="programlisting"
785 > partition_letter:free_blocks
790 CLASS="computeroutput"
792 > indicates that partition <SPAN
798 > has 8,949 KB free. If the window is not wide enough for all partition entries to
799 appear on a single line, the <SPAN
805 > program automatically stacks the partition
806 entries into subcolumns within the sixth column.</P
808 >The label on the <SAMP
809 CLASS="computeroutput"
811 > column indicates the threshold value at which
812 entries in the column become highlighted. By default, the <SPAN
819 a partition that is over 95% full, in which case the label is as follows:</P
821 CLASS="programlisting"
822 > Disk attn: > 95% used
825 >For more on this threshold and its effect on highlighting, see <A
826 HREF="c18360.html#HDRWQ332"
828 Significant Statistics</A
835 >For all columns except the fifth (file server machine name), you can use the <SPAN
841 > argument to set a threshold value above which the <SPAN
848 program highlights the statistic. By default, only values in the fifth and sixth columns ever become highlighted. For
849 instructions on using the <SPAN
856 HREF="c18360.html#HDRWQ332"
858 Significant Statistics</A
867 >The Probe Reporting Line</A
870 >The bottom line of the display indicates how many times the <SPAN
877 the File Server processes for statistics. The statistics gathered in the latest probe appear in the statistics display
878 region. By default, the <SPAN
884 > program probes the File Servers every 60 seconds, but you can
891 > argument to specify a different probe frequency.</P
900 >Highlighting Significant Statistics</A
903 >To draw your attention to a statistic that currently exceed a threshold value, the <SPAN
909 > program displays it in reverse video (highlights it). You can set the threshold value for most
910 statistics, and so determine which values are worthy of special attention and which are normal.</P
917 >Highlighting Server Outages</A
920 >The only column in which you cannot control highlighting is the fifth, which identifies the file server machine for
921 which statistics are displayed in the other columns. The <SPAN
927 > program uses highlighting in
928 this column to indicate that the File Server process on a machine fails to respond to its probe, and automatically blanks
929 out the other columns. Failure to respond to the probe can indicate a File Server process, file server machine, or network
930 outage, so the highlighting draws your attention to a situation that is probably interrupting service to users.</P
932 >When the File Server process once again responds to the probes, its name appears normally and statistics reappear in
933 the other columns. If all machine names become highlighted at once, a possible network outage has disrupted the connection
934 between the file server machines and the client machine running the <SPAN
948 >Highlighting for Extreme Statistic Values</A
951 >To set the threshold value for one or more of the five statistics-displaying columns, use the <SPAN
957 > argument. The threshold value applies to all File Server processes you are monitoring (you
958 cannot set different thresholds for different machines). For details, see the syntax description in <A
959 HREF="c18360.html#HDRWQ335"
960 >To start the scout program</A
963 >It is not possible to change the threshold values for a running <SPAN
970 the current program and start a new one. Also, the <SPAN
976 > program does not retain threshold
977 values across restarts, so you must specify all thresholds every time you start the program.</P
986 >Resizing the scout Display</A
989 >Do not resize the display window while the <SPAN
995 > program is running. Increasing the
996 size does no harm, but the <SPAN
1002 > program does not necessarily adjust to the new dimensions.
1003 Decreasing the display's width can disturb column alignment, making the display harder to read. With any type of resizing, the
1010 > program does not adjust the display in any way until it displays the results of the
1013 >To resize the display effectively, stop the <SPAN
1019 > program, resize the window and then
1020 restart the program. Even in this case, the <SPAN
1026 > program's response depends on the accuracy
1027 of the information it receives from the display environment. Testing during development has shown that the display environment
1028 does not reliably provide information about window resizing. If you use the X windowing system, issuing the following sequence
1029 of commands before starting the <SPAN
1035 > program (or placing them in the shell initialization
1036 file) sometimes makes it adjust properly to resizing.</P
1038 CLASS="programlisting"
1050 >eval '/usr/bin/X11/resize'</B
1068 >To start the scout program</A
1074 >Open a dedicated command shell. If necessary, adjust it to the appropriate size.</P
1084 > command to start the program. <PRE
1085 CLASS="programlisting"
1106 >FileServer name(s) to monitor</VAR
1116 >base server name</VAR
1126 >poll frequency, in seconds</VAR
1142 >specify attention (highlighting) level</VAR
1152 >turn debugging output on to the named file</VAR
1158 CLASS="variablelist"
1170 >Is an optional string that accommodates the command's use of the AFS command parser. It can be omitted and
1183 >Identifies each File Server process to monitor, by naming the file server machine it is running on. Provide
1184 fully-qualified hostnames unless the <SPAN
1190 > argument is used. In that case,
1191 specify only the initial part of each machine name, omitting the domain name suffix common to all the machine
1204 >Specifies the domain name suffix common to all of the file server machines named by the <SPAN
1210 > argument. For discussion of this argument's effects, see <A
1211 HREF="c18360.html#HDRWQ328"
1212 >Using the -basename argument to Specify a Domain Name</A
1215 >Do not include the period that separates the domain suffix from the initial part of the machine name, but do
1216 include any periods that occur within the suffix itself. (For example, in the ABC Corporation cell, the proper
1241 >Sets the frequency, in seconds, of the <SPAN
1247 > program's probes to File
1248 Server processes. Specify an integer greater than 0 (zero). The default is 60 seconds.</P
1260 >Displays the name of the machine that is running the <SPAN
1267 display window's banner line. By default, no machine name is displayed.</P
1279 >Defines the threshold value at which to highlight one or more statistics. You can provide the pairs of
1280 statistic and threshold in any order, separating each pair and the parts of each pair with one or more spaces. The
1281 following list defines the syntax for each statistic.<DIV
1282 CLASS="variablelist"
1289 >conn connections</B
1294 >Highlights the value in the <SAMP
1295 CLASS="computeroutput"
1297 > (first) column when the number of
1298 connections that the File Server has open to client machines exceeds the connections value. The
1299 highlighting deactivates when the value goes back below the threshold. There is no default
1307 >fetch fetch_RPCs</B
1312 >Highlights the value in the <SAMP
1313 CLASS="computeroutput"
1315 > (second) column when the number
1316 of fetch RPCs that clients have made to the File Server process exceeds the fetch_RPCs value. The
1317 highlighting deactivates only when the File Server process restarts, at which time the value returns to
1318 zero. There is no default threshold.</P
1325 >store store_RPCs</B
1330 >Highlights the value in the <SAMP
1331 CLASS="computeroutput"
1333 > (third) column when the number of
1334 store RPCs that clients have made to the File Server process exceeds the store_RPCs value. The
1335 highlighting deactivates only when the File Server process restarts, at which time the value returns to
1336 zero. There is no default threshold.</P
1343 >ws active_clients</B
1348 >Highlights the value in the <SAMP
1349 CLASS="computeroutput"
1351 > (fourth) column when the number of
1352 active client machines (those that have contacted the File Server in the last 15 minutes) exceeds the
1353 active_clients value. The highlighting deactivates when the value goes back below the threshold. There is
1354 no default threshold.</P
1361 >disk percent_full % or disk min_blocks</B
1366 >Highlights the value for a partition in the <SAMP
1367 CLASS="computeroutput"
1370 column when either the amount of disk space used exceeds the percentage indicated by thepercent_full
1371 value, or the number of free KB blocks is less than the min_blocks value. The highlighting deactivates
1372 when the value goes back below the percent_full threshold or above the min_blocks threshold.</P
1374 >The value you specify appears in the header of the sixth column following the string
1376 CLASS="computeroutput"
1378 >. The default threshold is 95% full.</P
1380 >Acceptable values for percent_full are the integers from the range <SPAN
1392 >, and you must include the percent
1393 sign to distinguish this statistic from a min_blocks value..</P
1399 >The following example sets the threshold for the <SAMP
1400 CLASS="computeroutput"
1402 > column to 100, for
1404 CLASS="computeroutput"
1406 > column to 50, and for the <SAMP
1407 CLASS="computeroutput"
1410 column to 75%. There is no threshold for the <SAMP
1411 CLASS="computeroutput"
1415 CLASS="computeroutput"
1423 >-attention conn 100 ws 50 disk 75%</B
1427 >The following example has the same affect as the previous one except that it sets the threshold for the Disk
1428 attn column to 5000 free KB blocks:</P
1434 >-attention disk 5000 ws 50 conn 100</B
1448 >Enables debugging output and directs it into the specified file. Partial pathnames are interpreted relative
1449 to the current working directory. By default, no debugging output is produced.</P
1463 >To stop the scout program</A
1475 > in the display window. This is the proper interrupt signal even if the
1476 general interrupt signal in your environment is different.</P
1486 >Example Commands and Displays</A
1489 >This section presents examples of the <SPAN
1495 > program, combining different arguments and
1496 illustrating the screen displays that result.</P
1498 >In the first example, an administrator in the ABC Corporation issues the <SPAN
1505 without providing any optional arguments or flags. She includes the <SPAN
1512 she is providing multiple machine names. She chooses to specify on the initial part of each machine's name even though she has
1519 > argument, relying on the cell's name service to obtain the
1520 fully-qualified name that the <SPAN
1526 > program requires for establishing a connection.</P
1528 CLASS="programlisting"
1533 >scout -server fs1 fs2</B
1539 HREF="c18360.html#FIGWQ337"
1541 > depicts the resulting display. Notice first that the machine names in the fifth
1542 (unlabeled) column appear in the format the administrator used on the command line. Now consider the second line in the
1543 display region, where the machine name <SAMP
1544 CLASS="computeroutput"
1546 > appears in the fifth column. The
1548 CLASS="computeroutput"
1551 CLASS="computeroutput"
1553 > columns together show that machine <SPAN
1559 > has 144 RPC connections open to 44 client machines, demonstrating that multiple connections per
1560 client machine are possible. The <SAMP
1561 CLASS="computeroutput"
1563 > column shows that client machines have made 2,734,278
1564 fetch RPCs to machine <SPAN
1570 > since the File Server process last started and the
1572 CLASS="computeroutput"
1574 > column shows that they have made 34,066 store RPCs.</P
1576 >Six partition entries appear in the <SAMP
1577 CLASS="computeroutput"
1581 CLASS="computeroutput"
1584 CLASS="computeroutput"
1599 >). They appear on three lines in two subcolumns because of the width of the
1600 window; if the window is wider, there are more subcolumns. Four of the partition entries (<SAMP
1601 CLASS="computeroutput"
1605 CLASS="computeroutput"
1608 CLASS="computeroutput"
1611 CLASS="computeroutput"
1614 reverse video to indicate that they are more than 95% full (the threshold value that appears in the <SAMP
1615 CLASS="computeroutput"
1628 SRC="scout1.png"></P
1632 >Figure 2. First example scout display</B
1644 >In the second example, the administrator uses more of the <SPAN
1650 > program's optional
1654 >She provides the machine names in the same form as in Example 1, but this time she also uses the <SPAN
1660 > argument to specify their domain name suffix, <SPAN
1667 This implies that the <SPAN
1673 > program does not need the name service to expand the names
1674 to fully-qualified hostnames, but the name service still converts the hostnames to IP addresses.</P
1684 > flag to display in the banner line the name of the client
1685 machine where the <SPAN
1691 > program is running.</P
1701 > argument to changes the probing frequency from its
1702 default of once per minute to once every five seconds.</P
1712 > argument to changes the highlighting threshold for
1713 partitions to a 5000 KB minimum rather than the default of 95% full.</P
1718 CLASS="programlisting"
1723 >scout -server fs1 fs2 -basename abc.com -host -frequency 5 -attention disk 5000</B
1728 >The use of optional arguments results in several differences between <A
1729 HREF="c18360.html#FIGWQ338"
1732 HREF="c18360.html#FIGWQ337"
1734 >. First, because the <SPAN
1740 > flag is included, the banner
1741 line displays the name of the machine running the <SPAN
1749 CLASS="computeroutput"
1751 > along with the basename <SAMP
1752 CLASS="computeroutput"
1763 >Another difference is that two rather than four of machine <SPAN
1769 >'s partitions appear in
1770 reverse video, even though their values are almost the same as in <A
1771 HREF="c18360.html#FIGWQ337"
1774 the administrator changed the highlight threshold to a 5000 block minimum, as also reflected in the <SAMP
1775 CLASS="computeroutput"
1778 > column's header. And while machine <SPAN
1784 >'s partitions <SPAN
1796 > are still 95% full, they have more than 5000 free
1797 blocks left; partitions <SPAN
1810 because they have fewer than 5000 blocks free.</P
1812 >Note also the result of changing the probe frequency, reflected in the probe reporting line at the bottom left corner of
1813 the display. Both this example and the previous one represent a time lapse of one minute after the administrator issues the
1820 > command. In this example, however, the <SPAN
1827 has probed the File Server processes 12 times as opposed to once</P
1837 SRC="scout2.png"></P
1841 >Figure 3. Second example scout display</B
1854 HREF="c18360.html#FIGWQ339"
1856 >, an administrator in the State University cell monitors three of that cell's
1857 file server machines. He uses the <SPAN
1863 > argument to specify the <SPAN
1871 CLASS="programlisting"
1876 >scout -server server2 server3 server4 -basename stateu.edu</B
1889 SRC="scout3.png"></P
1893 >Figure 4. Third example scout display</B
1906 HREF="c18360.html#FIGWQ340"
1908 > illustrates three of the <SPAN
1915 features. First, you can monitor file server machines from different cells in a single display: <SPAN
1925 >server3.stateu.edu</B
1933 >. Because the machines belong to different cells, it is not possible to provide the
1942 >Second, it illustrates how the display must truncate machine names that do not fit in the fifth column, using an
1943 asterisk at the end of the name to show that it is shortened.</P
1945 >Third, it illustrates what happens when the <SPAN
1951 > process cannot reach a File Server
1952 process, in this case the one on the machine <SPAN
1958 >: it highlights the machine name and
1959 blanks out the values in the other columns.</P
1969 SRC="scout4.png"></P
1973 >Figure 5. Fourth example scout display</B
1984 >Using the fstrace Command Suite</A
1987 >This section describes the <SPAN
1993 > commands that system administrators employ to trace
1994 Cache Manager activity for debugging purposes. It assumes the reader is familiar with the Cache Manager concepts described in
1997 >Administering Client Machines and the Cache Manager</A
2006 > command suite monitors the internal activity of the Cache Manager and enables
2007 you to record, or trace, its operations in detail. The operations, which are termed <SPAN
2026 >. Examples of <SPAN
2033 fetching files and looking up information for a listing of files and subdirectories using the UNIX <SPAN
2041 >Following are the <SPAN
2047 > commands and their respective functions: <UL
2056 > command provides a short description of commands.</P
2066 > command clears the trace log.</P
2076 > command dumps the contents of the trace log.</P
2086 > command provides a description and syntax for commands.</P
2096 > command lists information about the trace log.</P
2106 > command lists information about the event set.</P
2116 > command changes the size of the trace log.</P
2126 > command sets the state of the event set.</P
2136 >About the fstrace Command Suite</A
2145 > command suite replaces and greatly expands the functionality formerly
2146 provided by the <SPAN
2152 > command. Its intended use is to aid in diagnosis of specific Cache
2153 Manager problems, such as client machine hangs, cache consistency problems, clock synchronization errors, and failures to
2154 access a volume or AFS file. Therefore, it is best not to keep <SPAN
2160 > logging enabled at all
2161 times, unlike the logging for AFS server processes.</P
2163 >Most of the messages in the trace log correspond to low-level Cache Manager operations. It is likely that only personnel
2164 familiar with the AFS source code can interpret them. If you have an AFS source license, you can attempt to interpret the
2165 trace yourself, or work with the AFS Product Support group to resolve the underlying problems. If you do not have an AFS
2166 source license, it is probably more efficient to contact the AFS Product Support group immediately in case of problems. They
2167 can instruct you to activate <SPAN
2173 > tracing if appropriate.</P
2175 >The log can grow in size very quickly; this can use valuable disk space if you are writing to a file in the local file
2176 space. Additionally, if the size of the log becomes too large, it can become difficult to parse the results for pertinent
2179 >When AFS tracing is enabled, each time a <SPAN
2185 > event occurs, a message is written to the
2192 >. To diagnose a problem, read the output of the trace log and analyze the
2193 operations executed by the Cache Manager. The default size of the trace log is 60 KB, but you can increase or decrease
2202 > command suite, you must first enable tracing and reserve, or
2203 allocate, space for the trace log with the <SPAN
2209 > command. With this command, you can
2216 > event set to one of three states to enable or disable tracing for the event set
2217 and to allocate or deallocate space for the trace log in the kernel: <DIV
2218 CLASS="variablelist"
2222 CLASS="computeroutput"
2227 >Enables tracing for the event set and allocates space for the trace log.</P
2231 CLASS="computeroutput"
2236 >Temporarily disables tracing for the event set; however, the event set continues to allocate space occupied by
2237 the log to which it sends data.</P
2241 CLASS="computeroutput"
2246 >Disables tracing for the event set; furthermore, the event set releases the space occupied by the log to which
2247 it sends data. When the <SPAN
2253 > event set that sends data to the <SPAN
2259 > trace log is in this state, the space allocated for that log is freed or
2266 >Both event sets and trace logs can be designated as <SPAN
2272 >, which prevents accidental resetting
2273 of an event set's state or clearing of a trace log. The designation is made as the kernel is compiled and cannot be
2276 >If an event set such as <SPAN
2282 > is persistent, you can change its state only by including
2289 > argument to the <SPAN
2295 > command. (That is,
2296 you cannot change its state along with the state of all other event sets by issuing the <SPAN
2303 > command with no arguments.) Similarly, if a trace log such as <SPAN
2310 persistent, you can clear it only by including either the <SPAN
2322 > argument to the <SPAN
2328 > command (you cannot clear it along
2329 with all other trace logs by issuing the <SPAN
2335 > command with no arguments.)</P
2337 >When a problem occurs, set the <SPAN
2343 > event set to active using the <SPAN
2349 > command. When tracing is enabled on a busy AFS client, the volume of events being
2350 recorded is significant; therefore, when you are diagnosing problems, restrict AFS activity as much as possible to minimize
2351 the amount of extraneous tracing in the log. Because tracing can have a negative impact on system performance, leave <SPAN
2357 > tracing in the dormant state when you are not diagnosing problems.</P
2359 >If a problem is reproducible, clear the <SPAN
2365 > trace log with the <SPAN
2371 > command and reproduce the problem. If the problem is not easily reproduced, keep the
2372 state of the event set active until the problem recurs.</P
2374 >To view the contents of the trace log and analyze the <SPAN
2380 > events, use the <SPAN
2386 > command to copy the content lines of the trace log to standard output (stdout) or to a
2395 >If a particular command or process is causing problems, determine its process id (PID). Search the output of the
2402 > command for the PID to find only those lines associated with the
2413 >Requirements for Using the fstrace Command Suite</A
2416 >Except for the <SPAN
2429 commands, which require no privilege, issuing the <SPAN
2435 > commands requires that the issuer
2436 be logged in as the local superuser <SPAN
2442 > on the local client machine. Before issuing an
2449 > command, verify that you have the necessary privilege.</P
2451 >The Cache Manager catalog must be in place so that logging can occur. The <SPAN
2458 command suite uses the standard UNIX catalog utilities. The default location is <SPAN
2462 >/usr/vice/etc/C/afszcm.cat</B
2464 >. It can be placed in another directory by placing the file elsewhere and
2465 using the proper NLSPATH and LANG environment variables.</P
2473 >Using fstrace Commands Effectively</A
2482 > commands most effectively, configure them as indicated: <UL
2491 > binary in a local disk directory.</P
2495 >When you dump the <SPAN
2501 > log to a file, direct it to one on the local
2506 >The trace can grow large in just a few minutes. Before attempting to dump the log to a local file, verify that you
2507 have enough room. Be particularly careful if you are using disk quotas on partitions in the local file system.</P
2511 >Attempt to limit Cache Manager activity on the AFS client machine other than the problem operation. This reduces
2512 the amount of extraneous data in the trace.</P
2522 > log for the shortest possibly period of time. If possible
2523 activate the trace immediately before performing the problem operation, deactivate it as soon as the operation
2524 completes, and dump the trace log to a file immediately.</P
2528 >If possible, obtain UNIX process ID (PID) of the command or program that initiates the problematic operation. This
2529 enables the person analyzing the trace log to search it for messages associated with the PID.</P
2540 >Activating the Trace Log</A
2543 >To start Cache Manager tracing on an AFS client machine, you must first configure <UL
2552 > kernel trace log using the <SPAN
2569 > event set using the <SPAN
2587 > command sets the size of the <SPAN
2594 kernel trace log in kilobytes. The trace log occupies 60 kilobytes of kernel by default. If the trace log already exists, it
2595 is cleared when this command is issued and a new log of the given size is created. Otherwise, a new log of the desired size is
2604 > command sets the state of the <SPAN
2611 kernel event set. The state of the <SPAN
2617 > event set determines whether information on the events
2618 in that event set is logged.</P
2620 >After establishing kernel tracing on the AFS client machine, you can check the state of the event set and the size of
2621 the kernel buffer allocated for the trace log. To display information about the state of the <SPAN
2627 > event set, issue the <SPAN
2633 > command. To display information
2640 > trace log, use the <SPAN
2647 the instructions in <A
2648 HREF="c18360.html#HDRWQ346"
2649 >Displaying the State of a Trace Log or Event Set</A
2658 >To configure the trace log</A
2664 >Become the local superuser <SPAN
2670 > on the machine, if you are not already, by issuing
2678 CLASS="programlisting"
2701 > command to set the size of the <SPAN
2707 > kernel trace log. <PRE
2708 CLASS="programlisting"
2732 >1-kilobyte_units</VAR
2739 >The following example sets the size of the <SPAN
2745 > trace log to 80 KB.</P
2747 CLASS="programlisting"
2752 >fstrace setlog cmfx 80</B
2763 >To set the event set</A
2769 >Become the local superuser <SPAN
2775 > on the machine, if you are not already, by issuing
2783 CLASS="programlisting"
2806 > command to set the state of event sets. <PRE
2807 CLASS="programlisting"
2848 >The following example activates the <SPAN
2856 CLASS="programlisting"
2861 >fstrace setset cm -active</B
2872 >Displaying the State of a Trace Log or Event Set</A
2875 >An event set must be in the <SPAN
2881 > to be included in the trace log. To display an event set's
2882 state, use the <SPAN
2888 > command. To set its state, issue the <SPAN
2894 > command as described in <A
2895 HREF="c18360.html#HDRWQ345"
2896 >To set the event set</A
2899 >To display size and allocation information for the trace log, issue the <SPAN
2906 >command with the <SPAN
2920 >To display the state of an event set</A
2926 >Become the local superuser <SPAN
2932 > on the machine, if you are not already, by issuing
2940 CLASS="programlisting"
2963 > command to display the available event set and its state.
2965 CLASS="programlisting"
2987 >The following example displays the event set and its state on the local machine.</P
2989 CLASS="programlisting"
2994 >fstrace lsset cm</B
3001 >The output from this command lists the event set and its states. The three event states for the <SPAN
3007 > event set are: <DIV
3008 CLASS="variablelist"
3020 >Tracing is enabled.</P
3032 >Tracing is disabled, but space is still allocated for the corresponding trace log (<SPAN
3050 >Tracing is disabled, and space is no longer allocated for the corresponding trace log (<SPAN
3056 >).Disables tracing for the event set.</P
3068 >To display the log size</A
3074 >Become the local superuser <SPAN
3080 > on the machine, if you are not already, by issuing
3088 CLASS="programlisting"
3111 > command to display information about the kernel trace log.
3113 CLASS="programlisting"
3150 >The following example uses the <SPAN
3156 > flag to display additional information about the
3165 CLASS="programlisting"
3170 >fstrace lslog cmfx -long</B
3174 cmfx : 60 kbytes (allocated)
3177 >The output from this command lists information on the trace log. When issued without the <SPAN
3189 > command lists only the name of the log.
3190 When issued with the <SPAN
3203 lists the log, the size of the log in kilobytes, and the allocation state of the log.</P
3205 >There are two allocation states for the kernel trace log: <DIV
3206 CLASS="variablelist"
3210 CLASS="computeroutput"
3215 >Space is reserved for the log in the kernel. This indicates that the event set that writes to this log is either
3222 > (tracing is enabled for the event set) or <SPAN
3229 temporarily disabled for the event set); however, the event set continues to reserve space occupied by the log to
3230 which it sends data.</P
3234 CLASS="computeroutput"
3239 >Space is not reserved for the log in the kernel. This indicates that the event set that writes to this log is
3246 > (tracing is disabled for the event set); furthermore, the event set releases the space
3247 occupied by the log to which it sends data.</P
3259 >Dumping and Clearing the Trace Log</A
3262 >After the Cache Manager operation you want to trace is complete, use the <SPAN
3269 command to dump the trace log to the standard output stream or to the file named by the <SPAN
3276 argument. Or, to dump the trace log continuously, use the <SPAN
3282 > argument (combine it with
3289 > argument if desired). To halt continuous dumping, press an interrupt signal such as
3298 >To clear a trace log when you no longer need the data in it, issue the <SPAN
3311 > command also clears an existing trace log automatically when you
3312 use it to change the log's size.)</P
3320 >To dump the contents of a trace log</A
3326 >Become the local superuser <SPAN
3332 > on the machine, if you are not already, by issuing
3340 CLASS="programlisting"
3363 > command to dump trace logs. <PRE
3364 CLASS="programlisting"
3398 >output_filename</VAR
3408 >seconds_between_reads</VAR
3415 >At the beginning of the output of each dump is a header specifying the date and time at which the dump began. The number
3416 of logs being dumped is also displayed if the <SPAN
3422 > argument is not specified. The header
3423 appears as follows:</P
3425 CLASS="programlisting"
3426 > AFS Trace Dump --
3437 > is the starting date of the trace log dump, <SPAN
3444 time of the trace log dump, and <SPAN
3450 > specifies the number of logs found by the <SPAN
3459 >The following is an example of trace log dump header:</P
3461 CLASS="programlisting"
3462 > AFS Trace Dump --
3463 Date: Fri Apr 16 10:44:38 1999
3467 >The contents of the log follow the header and are comprised of messages written to the log from an active event set. The
3468 messages written to the log contain the following three components: <UL
3471 >The timestamp associated with the message (number of seconds from an arbitrary start point)</P
3475 >The process ID or thread ID associated with the message</P
3479 >The message itself</P
3484 >A trace log message is formatted as follows:</P
3486 CLASS="programlisting"
3487 > time timestamp, pid pid:event message
3496 > is the number of seconds from an arbitrary start point, <SPAN
3503 the process ID number of the Cache Manager event, and <SPAN
3509 > is the Cache Manager event which
3510 corresponds with a function in the AFS source code.</P
3512 >The following is an example of a dumped trace log message:</P
3514 CLASS="programlisting"
3515 > time 749.641274, pid 3002:Returning code 2 from 19
3518 >For the messages in the trace log to be most readable, the Cache Manager catalog file needs to be installed on the local
3519 disk of the client machine; the conventional location is <SPAN
3523 >/usr/vice/etc/C/afszcm.cat</B
3526 messages that begin with the string <SAMP
3527 CLASS="computeroutput"
3529 >, like the following, indicate that the catalog is
3532 CLASS="programlisting"
3533 > raw op 232c, time 511.916288, pid 0
3534 p0:Fri Apr 16 10:36:31 1999
3537 >Every 1024 seconds, a current time message is written to each log. This message has the following format:</P
3539 CLASS="programlisting"
3540 > time timestamp, pid pid: Current time: unix_time
3543 >where timestamp is the number of seconds from an arbitrary start point, pid is the process ID number, and unix_time is
3544 the standard time format since January 1, 1970.</P
3546 >The current time message can be used to determine the actual time associated with each log message. Determine the actual
3547 time as follows: <OL
3551 >Locate the log message whose actual time you want to determine.</P
3555 >Search backward through the dump record until you come to a current time message.</P
3559 >If the current time message's <SPAN
3565 > is smaller than the log message's
3572 >, subtract the former from the latter. If the current time message's
3579 > is larger than the log message's <SPAN
3585 >, add 1024 to the latter
3586 and subtract the former from the result.</P
3590 >Add the resulting number to the current time message's <SPAN
3596 > to determine the log
3597 message's actual time.</P
3602 >Because log data is stored in a finite, circular buffer, some of the data can be overwritten before being read. If this
3603 happens, the following message appears at the appropriate place in the dump:</P
3605 CLASS="programlisting"
3606 > Log wrapped; data missing.
3615 >If this message appears in the middle of a dump, which can happen under a heavy work load, it indicates that not all
3616 of the log data is being written to the log or some data is being overwritten. Increasing the size of the log with the
3623 > command can alleviate this problem.</P
3633 >To clear the contents of a trace log</A
3639 >Become the local superuser <SPAN
3645 > on the machine, if you are not already, by issuing
3653 CLASS="programlisting"
3676 > command to clear logs by log name or by event set.
3678 CLASS="programlisting"
3709 >The following example clears the <SPAN
3715 > log used by the <SPAN
3721 > event set on the local machine.</P
3723 CLASS="programlisting"
3728 >fstrace clear cm</B
3733 >The following example also clears the <SPAN
3739 > log on the local machine.</P
3741 CLASS="programlisting"
3746 >fstrace clear cmfx</B
3757 >Examples of fstrace Commands</A
3760 >This section contains an extensive example of the use of the <SPAN
3767 which is useful for gathering a detailed trace of Cache Manager activity when you are working with AFS Product Support to
3768 diagnose a problem. The Product Support representative can guide you in choosing appropriate parameter settings for the
3771 >Before starting the kernel trace log, try to isolate the Cache Manager on the AFS client machine that is experiencing
3772 the problem accessing the file. If necessary, instruct users to move to another machine so as to minimize the Cache Manager
3773 activity on this machine. To minimize the amount of unrelated AFS activity recorded in the trace log, place both the <SPAN
3779 > binary and the dump file must reside on the local disk, not in AFS. You must be logged in as
3780 the local superuser <SPAN
3794 >Before starting a kernel trace, issue the <SPAN
3800 > command to check the state of
3809 CLASS="programlisting"
3814 >fstrace lsset cm</B
3819 >If tracing has not been enabled previously or if tracing has been turned off on the client machine, the following output
3822 CLASS="programlisting"
3823 > Available sets:
3827 >If tracing has been turned off and kernel memory is not allocated for the trace log on the client machine, the following
3828 output is displayed:</P
3830 CLASS="programlisting"
3831 > Available sets:
3832 cm inactive (dormant)
3835 >If the current state of the <SPAN
3841 > event set is <SAMP
3842 CLASS="computeroutput"
3846 CLASS="computeroutput"
3847 >inactive (dormant)</SAMP
3848 >, turn on kernel tracing by issuing the <SPAN
3855 > command with the <SPAN
3863 CLASS="programlisting"
3868 >fstrace setset cm -active</B
3873 >If tracing is enabled currently on the client machine, the following output is displayed:</P
3875 CLASS="programlisting"
3876 > Available sets:
3880 >If tracing is enabled currently, you do not need to use the <SPAN
3893 > command to clear the contents of any existing trace log, removing
3894 prior traces that are not related to the current problem.</P
3896 CLASS="programlisting"
3901 >fstrace clear cm</B
3906 >After checking on the state of the event set, issue the <SPAN
3919 > flag to check the current state and size of the kernel trace log .</P
3921 CLASS="programlisting"
3926 >fstrace lslog cmfx -long</B
3931 >If tracing has not been enabled previously or the <SPAN
3937 > event set was set to
3939 CLASS="computeroutput"
3942 CLASS="computeroutput"
3944 > previously, output similar to the
3945 following is displayed:</P
3947 CLASS="programlisting"
3948 > Available logs:
3949 cmfx : 60 kbytes (allocated)
3958 > tracing utility allocates 60 kilobytes of memory to the trace log by
3959 default. You can increase or decrease the amount of memory allocated to the kernel trace log by setting it with the <SPAN
3965 > command. The number specified with the <SPAN
3972 argument represents the number of kilobytes allocated to the kernel trace log. If you increase the size of the kernel trace
3973 log to 100 kilobytes, issue the following command.</P
3975 CLASS="programlisting"
3980 >fstrace setlog cmfx</B
3985 >After ensuring that the kernel trace log is configured for your needs, you can set up a file into which you can dump the
3986 kernel trace log. For example, create a dump file with the name <SPAN
3990 >cmfx.dump.file.1</B
3999 > command. Issue the command as a continuous process by adding the
4012 > arguments. Setting the <SPAN
4024 > dumps output from the kernel trace log to the file every 10
4027 CLASS="programlisting"
4032 >fstrace dump -follow</B
4040 > cmfx.dump.file.1 <SPAN
4048 Date: Fri Apr 16 10:54:57 1999
4050 time 32.965783, pid 0: Fri Apr 16 10:45:52 1999
4051 time 32.965783, pid 33657: Close 0x5c39ed8 flags 0x20
4052 time 32.965897, pid 33657: Gn_close vp 0x5c39ed8 flags 0x20 (returns
4054 time 35.159854, pid 10891: Breaking callback for 5bd95e4 states 1024
4056 time 35.407081, pid 10891: Breaking callback for 5c0fadc states 1024
4061 time 71.440456, pid 33658: Lookup adp 0x5bbdcf0 name g3oCKs fid (756
4062 4fb7e:588d240.2ff978a8.6)
4063 time 71.440569, pid 33658: Returning code 2 from 19
4064 time 71.440619, pid 33658: Gn_lookup vp 0x5bbdcf0 name g3oCKs (returns
4066 time 71.464989, pid 38267: Gn_open vp 0x5bbd000 flags 0x0 (returns 0x
4068 AFS Trace Dump - Completed
4078 >Using the afsmonitor Program</A
4087 > program enables you to monitor the status and performance of specified
4088 File Server and Cache Manager processes by gathering statistical information. Among its other uses, the <SPAN
4094 > program can be used to fine-tune Cache Manager configuration and load balance File
4103 > program enables you to perform the following tasks. <UL
4106 >Monitor any number of File Server and Cache Manager processes on any number of machines (in both local and foreign
4107 cells) from a single location.</P
4111 >Set threshold values for any monitored statistic. When the value of a statistic exceeds the threshold, the <SPAN
4117 > program highlights it to draw your attention. You can set threshold levels that apply to
4118 every machine or only some.</P
4122 >Invoke programs or scripts automatically when a statistic exceeds its threshold.</P
4132 >Requirements for running the afsmonitor program</A
4135 >The following software must be accessible to a machine where the <SPAN
4151 > libraries, which the <SPAN
4158 program uses to gather data</P
4168 > graphics package, which most UNIX distributions provide as a standard
4180 > screens format successfully both on so-called dumb terminals and in
4181 windowing systems that emulate terminals. For the output to looks its best, the display environment needs to support reverse
4182 video and cursor addressing. Set the TERM environment variable to the correct terminal type, or to a value that has
4183 characteristics similar to the actual terminal type. The display window or terminal must be at least 80 columns wide and 12
4192 > program must run in the foreground, and in its own separate, dedicated
4193 window or terminal. The window or terminal is unavailable for any other activity as long as the <SPAN
4199 > program is running. Any number of instances of the <SPAN
4205 > program can run on a single machine, as long as each instance runs in its own dedicated
4206 window or terminal. Note that it can take up to three minutes to start an additional instance.</P
4208 >No privilege is required to run the <SPAN
4214 > program. By convention, it is installed
4221 > directory, and anyone who can access the directory can monitor File
4222 Servers and Cache Managers. The probes through which the <SPAN
4229 statistics do not constitute a significant burden on the File Server or Cache Manager unless hundreds of people are running
4230 the program. If you wish to restrict its use, place the binary file in a directory available only to authorized users.</P
4238 >The afsmonitor Output Screens</A
4247 > program displays its data on three screens: <UL
4251 CLASS="computeroutput"
4252 >System Overview</SAMP
4253 >: This screen appears automatically when the <SPAN
4259 > program initializes. It summarizes separately for File Servers and Cache Managers the
4260 number of machines being monitored and how many of them have <SPAN
4266 > (statistics that have exceeded
4267 their thresholds). It then lists the hostname and number of alerts for each machine being monitored, indicating if
4268 appropriate that a process failed to respond to the last probe.</P
4273 CLASS="computeroutput"
4275 >: This screen displays File Server statistics for each file server
4276 machine being monitored. It highlights statistics that have exceeded their thresholds, and identifies machines that
4277 failed to respond to the last probe.</P
4282 CLASS="computeroutput"
4283 >Cache Managers</SAMP
4284 >: This screen displays Cache Manager statistics for each client
4285 machine being monitored. It highlights statistics that have exceeded their thresholds, and identifies machines that
4286 failed to respond to the last probe.</P
4291 >Fields at the corners of every screen display the following information: <UL
4294 >In the top left corner, the program name and version number.</P
4298 >In the top right corner, the screen name, current and total page numbers, and current and total column numbers.
4299 The page number (for example, <SAMP
4300 CLASS="computeroutput"
4302 >) indicates the index of the current page and
4303 the total number of (vertical) pages over which data is displayed. The column number (for example, <SAMP
4304 CLASS="computeroutput"
4307 >) indicates the index of the current leftmost column and the total number of columns in which
4308 data appears. (The symbol <SAMP
4309 CLASS="computeroutput"
4310 >>>></SAMP
4311 > indicates that there is additional data to the
4312 right; the symbol <SAMP
4313 CLASS="computeroutput"
4314 ><<<</SAMP
4315 > indicates that there is additional data to the
4320 >In the bottom left corner, a list of the available commands. Enter the first letter in the command name to run
4321 that command. Only the currently possible options appear; for example, if there is only one page of data, the
4323 CLASS="computeroutput"
4326 CLASS="computeroutput"
4328 > commands, which scroll the screen up and
4329 down respectively, do not appear. For descriptions of the commands, see the following section about navigating the
4334 >In the bottom right corner, the <SAMP
4335 CLASS="computeroutput"
4337 > field reports how many times the program
4338 has probed File Servers (<SAMP
4339 CLASS="computeroutput"
4341 >), Cache Managers (<SAMP
4342 CLASS="computeroutput"
4345 both. The counts for File Servers and Cache Managers can differ. The <SAMP
4346 CLASS="computeroutput"
4349 how often the program sends probes.</P
4358 >Navigating the afsmonitor Display Screens</B
4362 >As noted, the lower left hand corner of every display screen displays the names of the commands currently available for
4363 moving to alternate screens, which can either be a different type or display more statistics or machines of the current type.
4364 To execute a command, press the lowercase version of the first letter in its name. Some commands also have an uppercase
4365 version that has a somewhat different effect, as indicated in the following list. <DIV
4366 CLASS="variablelist"
4370 CLASS="computeroutput"
4375 >Switches to the <SAMP
4376 CLASS="computeroutput"
4377 >Cache Managers</SAMP
4378 > screen. Available only on the
4380 CLASS="computeroutput"
4381 >System Overview</SAMP
4383 CLASS="computeroutput"
4389 CLASS="computeroutput"
4394 >Switches to the <SAMP
4395 CLASS="computeroutput"
4397 > screen. Available only on the
4399 CLASS="computeroutput"
4400 >System Overview</SAMP
4402 CLASS="computeroutput"
4403 >Cache Managers</SAMP
4409 CLASS="computeroutput"
4414 >Scrolls horizontally to the left, to access the data columns situated to the left of the current set. Available
4416 CLASS="computeroutput"
4417 ><<<</SAMP
4418 > symbol appears at the top left of the screen. Press uppercase
4425 > to scroll horizontally all the way to the left (to display the first set of data
4430 CLASS="computeroutput"
4435 >Scrolls down vertically to the next page of machine names. Available when there are two or more pages of
4436 machines and the final page is not currently displayed. Press uppercase <SPAN
4443 to the final page.</P
4447 CLASS="computeroutput"
4452 >Switches to the <SAMP
4453 CLASS="computeroutput"
4454 >System Overview</SAMP
4455 > screen. Available only on the
4457 CLASS="computeroutput"
4458 >Cache Managers</SAMP
4460 CLASS="computeroutput"
4466 CLASS="computeroutput"
4471 >Scrolls up vertically to the previous page of machine names. Available when there are two or more pages of
4472 machines and the first page is not currently displayed. Press uppercase <SPAN
4479 to the first page.</P
4483 CLASS="computeroutput"
4488 >Scrolls horizontally to the right, to access the data columns situated to the right of the current set. This
4489 command is available when the <SAMP
4490 CLASS="computeroutput"
4491 >>>></SAMP
4492 > symbol appears at the upper right of the
4493 screen. Press uppercase <SPAN
4499 > to scroll horizontally all the way to the right (to display
4500 the final set of data columns).</P
4512 >The System Overview Screen</A
4516 CLASS="computeroutput"
4517 >System Overview</SAMP
4518 > screen appears automatically as the <SPAN
4524 > program initializes. This screen displays the status of as many File Server and Cache
4525 Manager processes as can fit in the current window; scroll down to access additional information.</P
4527 >The information on this screen is split into File Server information on the left and Cache Manager information on the
4528 right. The header for each grouping reports two pieces of information: <UL
4531 >The number of machines on which the program is monitoring the indicated process</P
4535 >The number of alerts and the number of machines affected by them (an <SPAN
4542 statistic has exceeded its threshold or a process failed to respond to the last probe)</P
4547 >A list of the machines being monitored follows. If there are any alerts on a machine, the number of them appears in
4548 square brackets to the left of the hostname. If a process failed to respond to the last probe, the letters
4550 CLASS="computeroutput"
4552 > (probe failure) appear in square brackets to the left of the hostname.</P
4554 >The following graphic is an example <SAMP
4555 CLASS="computeroutput"
4556 >System Overview</SAMP
4563 > program is monitoring six File Servers and seven Cache Managers. The File Server process on
4570 > and the Cache Manager on host <SPAN
4577 are each marked <SAMP
4578 CLASS="computeroutput"
4580 > to indicate that one threshold value is exceeded. The
4582 CLASS="computeroutput"
4584 > marker on host <SPAN
4590 > indicates that its File
4591 Server process did not respond to the last probe.</P
4601 SRC="overview.png"></P
4605 >Figure 6. The afsmonitor System Overview Screen</B
4623 >The File Servers Screen</A
4627 CLASS="computeroutput"
4629 > screen displays the values collected at the most recent probe for File
4630 Server statistics.</P
4632 >A summary line at the top of the screen (just below the standard program version and screen title blocks) specifies the
4633 number of monitored File Servers, the number of alerts, and the number of machines affected by the alerts.</P
4635 >The first column always displays the hostnames of the machines running the monitored File Servers.</P
4637 >To the right of the hostname column appear as many columns of statistics as can fit within the current width of the
4638 display screen or window; each column requires space for 10 characters. The name of the statistic appears at the top of each
4639 column. If the File Server on a machine did not respond to the most recent probe, a pair of dashes
4641 CLASS="computeroutput"
4643 >) appears in each column. If a value exceeds its configured threshold, it is highlighted
4644 in reverse video. If a value is too large to fit into the allotted column width, it overflows into the next row in the same
4647 >For a list of the available File Server statistics, see <A
4649 >Appendix C, The afsmonitor Program
4653 >The following graphic depicts the <SAMP
4654 CLASS="computeroutput"
4656 > screen that follows the System Overview
4657 Screen example previously discussed; however, one additional server probe has been completed. In this example, the File Server
4664 > has exceeded the configured threshold for the number of performance calls
4671 > statistic), and that field appears in reverse video. Host
4678 > did not respond to Probe 10, so dashes appear in all fields.</P
4688 SRC="fserver1.png"></P
4692 >Figure 7. The afsmonitor File Servers Screen</B
4704 >Both the File Servers and Cache Managers screen (discussed in the following section) can display hundreds of columns of
4705 data and are therefore designed to scroll left and right. In the preceding graphic, the screen displays the leftmost screen
4706 and the screen title block shows that column 1 of 235 is displayed. The appearance of the
4708 CLASS="computeroutput"
4709 >>>></SAMP
4710 > symbol in the upper right hand corner of the screen and the <SPAN
4716 > command in the command block indicate that additional data is available by scrolling right. (For
4717 information on the available statistics, see <A
4719 >Appendix C, The afsmonitor Program
4729 > command is executed, the screen looks something like the following
4730 example. Note that the horizontal scroll symbols now point both to the left (<SAMP
4731 CLASS="computeroutput"
4732 ><<<</SAMP
4734 and to the right (<SAMP
4735 CLASS="computeroutput"
4736 >>>></SAMP
4737 >) and both the <SPAN
4750 > commands appear, indicating that additional data is available by scrolling both left
4761 SRC="fserver2.png"></P
4765 >Figure 8. The afsmonitor File Servers Screen Shifted One Page to the Right</B
4783 >The Cache Managers Screen</A
4787 CLASS="computeroutput"
4788 >Cache Managers</SAMP
4789 > screen displays the values collected at the most recent probe for
4790 Cache Manager statistics.</P
4792 >A summary line at the top of the screen (just below the standard program version and screen title blocks) specifies the
4793 number of monitored Cache Managers, the number of alerts, and the number of machines affected by the alerts.</P
4795 >The first column always displays the hostnames of the machines running the monitored Cache Managers.</P
4797 >To the right of the hostname column appear as many columns of statistics as can fit within the current width of the
4798 display screen or window; each column requires space for 10 characters. The name of the statistic appears at the top of each
4799 column. If the Cache Manager on a machine did not respond to the most recent probe, a pair of dashes
4801 CLASS="computeroutput"
4803 >) appears in each column. If a value exceeds its configured threshold, it is highlighted
4804 in reverse video. If a value is too large to fit into the allotted column width, it overflows into the next row in the same
4807 >For a list of the available Cache Manager statistics, see <A
4809 >Appendix C, The afsmonitor Program
4813 >The following graphic depicts a Cache Managers screen that follows the System Overview Screen previously discussed. In
4814 the example, the Cache Manager process on host <SPAN
4820 > has exceeded the configured threshold
4821 for the number of cells it can contact (the <SPAN
4825 >numCellsContacted</B
4827 > statistic), so that field
4828 appears in reverse video.</P
4838 SRC="cachmgr.png"></P
4842 >Figure 9. The afsmonitor Cache Managers Screen</B
4861 >Configuring the afsmonitor Program</A
4864 >To customize the <SPAN
4870 > program, create an ASCII-format configuration file and use
4877 > argument to name it. You can specify the following in the configuration file:
4881 >The File Servers, Cache Managers, or both to monitor.</P
4885 >The statistics to display. By default, the display includes 271 statistics for File Servers and 570 statistics for
4886 Cache Managers. For information on the available statistics, see <A
4888 >Appendix C, The afsmonitor
4889 Program Statistics</A
4894 >The threshold values to set for statistics and a script or program to execute if a threshold is exceeded. By
4895 default, no threshold values are defined and no scripts or programs are executed.</P
4900 >The following list describes the instructions that can appear in the configuration file: <DIV
4901 CLASS="variablelist"
4905 CLASS="computeroutput"
4913 >Names a client machine for which to display Cache Manager statistics. The order of <SPAN
4919 > lines in the file determines the order in which client machines appear from top to bottom on
4921 CLASS="computeroutput"
4922 >System Overview</SAMP
4924 CLASS="computeroutput"
4925 >Cache Managers</SAMP
4931 CLASS="computeroutput"
4939 >Names a file server machine for which to display File Server statistics. The order of <SPAN
4945 > lines in the file determines the order in which file server machines appear from top to bottom
4947 CLASS="computeroutput"
4948 >System Overview</SAMP
4950 CLASS="computeroutput"
4957 CLASS="computeroutput"
4958 >thresh fs | cm <VAR
4979 >Assigns the threshold value thresh_val to the statistic field_name, for either a File Server statistic (<SPAN
4985 >) or a Cache Manager statistic (<SPAN
4992 cmd_to_execute field names a binary or script to execute each time the value of the statistic changes from being below
4993 thresh_val to being at or above thresh_val. A change between two values that both exceed thresh_val does not retrigger
4994 the binary or script. The optional arg1 through argn fields are additional values that the <SPAN
5000 > program passes as arguments to the cmd_to_execute command. If any of them include one
5001 or more spaces, enclose the entire field in double quotes.</P
5003 >The parameters <SPAN
5016 threshold_val, and arg1 through argn correspond to the values with the same name on the <SPAN
5022 > line. The host_name parameter identifies the file server or client machine where the
5023 statistic has crossed the threshold, and the actual_val parameter is the actual value of field_name that equals or
5024 exceeds the threshold value.</P
5032 > line to set either a global threshold, which applies to all file
5033 server machines listed on <SPAN
5039 > lines or client machines listed on <SPAN
5045 > lines in the configuration file, or a machine-specific threshold, which applies to only one
5046 file server or client machine. <UL
5049 >To set a global threshold, place the <SPAN
5055 > line before any of the
5068 > lines in the file.</P
5072 >To set a machine-specific threshold, place the <SPAN
5091 > line, and above any other
5104 > lines. A machine-specific threshold
5105 value always overrides the corresponding global threshold, if set. Do not place a <SPAN
5112 > line directly after a <SPAN
5125 > line directly after a <SPAN
5138 CLASS="computeroutput"
5141 >field/group/section</VAR
5146 >Specifies which individual statistic, group of statistics, or section of statistics to display on the
5148 CLASS="computeroutput"
5157 CLASS="computeroutput"
5166 >) and the order in which to display them. The
5173 > statistics in the <SPAN
5177 >IBM AFS Administration
5180 > specifies the group and section to which each statistic belongs. Include as many <SPAN
5186 > lines as necessary to customize the screen display as desired, and place them anywhere in
5187 the file. The top-to-bottom order of the <SPAN
5193 > lines in the configuration file
5194 determines the left-to-right order in which the statistics appear on the corresponding screen.</P
5196 >If there are no <SPAN
5202 > lines in the configuration file, then the screens display
5203 all statistics for both Cache Managers and File Servers. Similarly, if there are no <SPAN
5211 CLASS="computeroutput"
5213 > screen displays all file server statistics, and
5214 if there are no <SPAN
5221 CLASS="computeroutput"
5222 >Cache Managers</SAMP
5224 screen displays all client statistics.</P
5236 >Precedes a line of text that the <SPAN
5242 > program ignores because of the
5243 initial number (<SPAN
5249 >) sign, which must appear in the very first column of the line.</P
5255 >For a list of the values that can appear in the field/group/section field of a <SPAN
5264 >Appendix C, The afsmonitor Program Statistics</A
5267 >The following example illustrates a possible configuration file:</P
5269 CLASS="programlisting"
5270 > thresh cm dlocalAccesses 1000000
5271 thresh cm dremoteAccesses 500000 handleDRemote
5272 thresh fs rx_maxRtt_Usec 1000
5276 thresh cm dlocalAccesses 2000000
5277 thresh cm vcacheMisses 10000
5283 show cm numCellsContacted
5284 show cm dlocalAccesses
5285 show cm dremoteAccesses
5286 show cm vcacheMisses
5287 show cm Auth_Stats_group
5290 >Since the first three <SPAN
5296 > instructions appear before any <SPAN
5308 > instructions, they set global threshold values: <UL
5311 >All Cache Manager process in this file use <SPAN
5317 > as the threshold for the
5324 > statistic (except for the machine <SPAN
5331 which uses an overriding value of <SPAN
5341 >All Cache Manager processes in this file use <SPAN
5347 > as the threshold value for the
5354 > statistic; if that value is exceeded, the script <SPAN
5364 >All File Server processes in this file use <SPAN
5370 > as the threshold value for the
5388 > instructions monitor the Cache Manager on the machines <SPAN
5413 >. The first three use all of the global threshold values.</P
5415 >The Cache Manager on <SPAN
5421 > uses the global threshold value for the <SPAN
5427 > statistic, but a different one for the <SPAN
5434 statistic. Furthermore, <SPAN
5440 > is the only Cache Manager that uses the threshold set for the
5455 > instructions monitor the File Server on the machines <SPAN
5479 >. They all use the global threshold for the<SPAN
5488 >Because there are no <SPAN
5494 > instructions, the File Servers screen displays all File
5495 Server statistics. The Cache Managers screen displays only the statistics named in <SPAN
5502 instructions, ordering them from left to right. The <SPAN
5506 >Auth_Stats_group</B
5509 statistics, all of which are displayed (<SPAN
5525 >curr_AuthRecords</B
5531 >curr_UnauthRecords</B
5537 >curr_MaxRecordsInPAG</B
5543 >curr_LongestChain</B
5574 >HWM_MaxRecordsInPAG</B
5580 >HWM_LongestChain</B
5590 >Writing afsmonitor Statistics to a File</A
5593 >All of the statistical information collected and displayed by the <SPAN
5600 be preserved by writing it to an output file. You can create an output file by using the <SPAN
5606 > argument when you startup the <SPAN
5612 > process. You can use
5613 the output file to track process performance over long periods of time and to apply post-processing techniques to further
5614 analyze system trends.</P
5622 > program output file is a simple ASCII file that records the information
5623 reported by the File Server and Cache Manager screens. The output file has the following format:</P
5625 CLASS="programlisting"
5626 > time host_name <SPAN
5638 > list_of_measured_values
5641 >and specifies the <SPAN
5647 > at which the <SPAN
5651 >list_of_measured_values</I
5653 > were gathered from
5654 the Cache Manager (<SPAN
5660 >) or File Server (<SPAN
5666 >) process housed on
5667 host_name. On those occasion where probes fail, the value <SAMP
5668 CLASS="computeroutput"
5670 > is reported instead of the
5675 >list_of_measured_values</I
5679 >This file format provides several advantages: <UL
5682 >It can be viewed using a standard editor. If you intend to view this file frequently, use the <SPAN
5688 > flag with the <SPAN
5694 > argument. It formats the output
5695 file in a way that is easier to read.</P
5699 >It can be passed through filters to extract desired information using the standard set of UNIX tools.</P
5703 >It is suitable for long term storage of the <SPAN
5709 > program output.</P
5720 >To start the afsmonitor Program</A
5726 >Open a separate command shell window or use a dedicated terminal for each instance of the <SPAN
5732 > program. This window or terminal must be devoted to the exclusive use of the <SPAN
5738 > process because the command cannot be run in the background.</P
5742 >Initialize the <SPAN
5748 > program. The message <SAMP
5749 CLASS="computeroutput"
5750 >afsmonitor Collecting
5752 >, followed by the appearance of the <SAMP
5753 CLASS="computeroutput"
5754 >System Overview</SAMP
5756 confirms a successful start. <PRE
5757 CLASS="programlisting"
5778 >configuration file</VAR
5788 >poll frequency, in seconds</VAR
5798 >storage file name</VAR
5814 >turn debugging output on to the named file</VAR
5824 >list of file servers to monitor</VAR
5834 >list of cache managers to monitor</VAR
5836 afsmonitor Collecting Statistics...
5841 CLASS="variablelist"
5853 >Is an optional string that accommodates the command's use of the AFS command parser. It can be omitted and
5866 >Specifies the pathname of an <SPAN
5872 > configuration file, which lists the
5873 machines and statistics to monitor. Partial pathnames are interpreted relative to the current working directory.
5874 Provide either this argument or one or both of the <SPAN
5886 > arguments. You must use a configuration file to set thresholds or customize the
5887 screen display. For instructions on creating the configuration file, see <A
5888 HREF="c18360.html#HDRWQ351"
5890 afsmonitor Program</A
5903 >Specifies how often to probe the File Server and Cache Manager processes, as a number of seconds. Acceptable
5904 values range from <SPAN
5916 >; the default value
5923 >. This frequency applies to both File Server and Cache Manager probes;
5924 however, File Server and Cache Manager probes are initiated and processed independent of each other. The actual
5925 interval between probes to a host is the probe frequency plus the time needed by all hosts to respond to the
5938 >Specifies the name of an output file to which to write all of the statistical data. By default, no output file
5939 is created. For information on this file, see <A
5940 HREF="c18360.html#HDRWQ352"
5941 >Writing afsmonitor Statistics to a
5955 >Formats the output file named by the <SPAN
5961 > argument to be more easily
5968 > argument must be provided along with this flag.</P
5980 >Identifies each File Server process to monitor by specifying the host it is running on. You can identify a
5981 host using either its complete Internet-style host name or an abbreviation acceptable to the cell's naming service.
5982 Combine this argument with the <SPAN
5988 > if you wish, but not the <SPAN
6006 >Identifies each Cache Manager process to monitor by specifying the host it is running on. You can identify a
6007 host using either its complete Internet-style host name or an abbreviation acceptable to the cell's naming service.
6008 Combine this argument with the <SPAN
6014 > if you wish, but not the <SPAN
6034 >To stop the afsmonitor program</A
6043 > program session, Enter the <<SPAN
6049 >> interrupt signal or an uppercase <SPAN
6063 >The xstat Data Collection Facility</A
6072 > program uses the <SPAN
6079 facility to gather and calculate the data that it (the <SPAN
6085 > program) then uses to perform
6086 its function. You can also use the <SPAN
6092 > facility to create your own data display programs. If
6093 you do, keep the following in mind. The File Server considers any program calling its RPC routines to be a Cache Manager;
6094 therefore, any program calling the File Server interface directly must export the Cache Manager's callback interface. The
6095 calling program must be capable of emulating the necessary callback state, and it must respond to periodic keep-alive messages
6096 from the File Server. In addition, a calling program must be able to gather the collected data.</P
6104 > facility consists of two C language libraries available to user-level
6112 >/usr/afsws/lib/afs/libxstat_fs.a</B
6114 > exports calls that gather information from one or
6115 more running File Server processes.</P
6123 >/usr/afsws/lib/afs/libxstat_cm.a</B
6125 > exports calls that collect information from one or
6126 more running Cache Managers.</P
6131 >The libraries allow the caller to register <UL
6134 >A set of File Servers or Cache Managers to be examined.</P
6138 >The frequency with which the File Servers or Cache Managers are to be probed for data.</P
6142 >A user-specified routine to be called each time data is collected.</P
6147 >The libraries handle all of the lightweight processes, callback interactions, and timing issues associated with the data
6148 collection. The user needs only to process the data as it arrives.</P
6155 >The libxstat Libraries</A
6171 the callback requirements and other complications associated with the collection of data from File Servers and Cache Managers.
6172 The user provides only the means of accumulating the desired data. Each <SPAN
6179 implements three routines: <UL
6182 >Initialization (<SPAN
6195 arranges the periodic collection and handling of data.</P
6199 >Immediate probe (<SPAN
6203 >xstat_fs_ForceProbeNow</B
6209 >xstat_cm_ForceProbeNow</B
6211 >) forces the immediate collection of data, after which collection returns
6212 to its normal probe schedule.</P
6220 >xstat_fs_Cleanup</B
6226 >xstat_cm_Cleanup</B
6229 terminates all connections and removes all traces of the data collection from memory.</P
6234 >The File Server and Cache Manager each define data collections that clients can fetch. A data collection is simply a
6235 related set of numbers that can be collected as a unit. For example, the File Server and Cache Manager each define profiling
6236 and performance data collections. The profiling collections maintain counts of the number of times internal functions are
6237 called within servers, allowing bottleneck analysis to be performed. The performance collections record, among other things,
6238 internal disk I/O statistics for a File Server and cache effectiveness figures for a Cache Manager, allowing for performance
6241 >For a copy of the detailed specification which provides much additional usage information about the <SPAN
6247 > facility, its libraries, and the routines in the libraries, contact AFS Product Support.</P
6255 >Example xstat Commands</A
6258 >AFS comes with two low-level, example commands: <SPAN
6270 >. The commands allow you to experiment with the <SPAN
6277 facility. They gather information and display the available data collections for a File Server or Cache Manager. They are
6278 intended merely to provide examples of the types of data that can be collected via <SPAN
6285 they are not intended for use in the actual collection of data.</P
6292 >To use the example xstat_fs_test command</A
6298 >Issue the example <SPAN
6304 > command to test the routines in the <SPAN
6310 > library and display the data collections associated with the File Server process.
6311 The command executes in the foreground. <PRE
6312 CLASS="programlisting"
6334 >File Server name(s) to monitor</VAR
6344 >Collection(s) to fetch</VAR
6360 >poll frequency, in seconds</VAR
6370 >data collection time, in minutes</VAR
6382 CLASS="variablelist"
6394 >Must be typed in full.</P
6406 >Is an optional string that accommodates the command's use of the AFS command parser. It can be omitted and
6419 >Is the Internet host name of each file server machine on which to monitor the File Server process.</P
6431 >Specifies each data collection to return. The indicated data collection defines the type and amount of
6432 data the command is to gather about the File Server. Data is returned in the form of a predefined data structure
6433 (refer to the specification documents referenced previously for more information about the data
6436 >There are two acceptable values: <UL
6445 > reports various internal performance statistics related to the
6446 File Server (for example, vnode cache entries and <SPAN
6463 > reports all of the internal performance statistics provided by
6470 > setting, plus some additional, detailed performance figures about
6471 the File Server (for example, minimum, maximum, and cumulative statistics regarding File Server RPCs, how
6472 long they take to complete, and how many succeed).</P
6487 >Directs the command to gather statistics just one time. Omit this option to have the command continue to
6488 probe the File Server for statistics every 30 seconds. If you omit this option, you can use the <<SPAN
6494 >> interrupt signal to halt the command at any time.</P
6506 >Sets the frequency in seconds at which the program initiates probes to the File Server. If you omit this
6507 argument, the default is 30 seconds.</P
6519 >Sets how long the utility runs before exiting, as a number of minutes. If you omit this argument, the
6520 default is 10 minutes.</P
6532 >Displays additional information as the command runs.</P
6546 >To use the example xstat_cm_test command</A
6552 >Issue the example <SPAN
6558 > command to test the routines in the <SPAN
6564 > library and display the data collections associated with the Cache Manager. The
6565 command executes in the foreground. <PRE
6566 CLASS="programlisting"
6588 >Cache Manager name(s) to monitor</VAR
6598 >Collection(s) to fetch</VAR
6614 >poll frequency, in seconds</VAR
6624 >data collection time, in minutes</VAR
6636 CLASS="variablelist"
6648 >Must be typed in full.</P
6660 >Is an optional string that accommodates the command's use of the AFS command parser. It can be omitted and
6673 >Is the host name of each client machine on which to monitor the Cache Manager.</P
6685 >Specifies each data collection to return. The indicated data collection defines the type and amount of
6686 data the command is to gather about the Cache Manager. Data is returned in the form of a predefined data
6687 structure (refer to the specification documents referenced previously for more information about the data
6690 >There are two acceptable values: <UL
6699 > provides profiling information about the numbers of times
6700 different internal Cache Manager routines were called since the Cache manager was started.</P
6710 > reports various internal performance statistics related to the
6711 Cache manager (for example, statistics about how effectively the cache is being used and the quantity of
6712 intracell and intercell data access).</P
6722 > reports all of the internal performance statistics provided by
6729 > setting, plus some additional, detailed performance figures about
6730 the Cache Manager (for example, statistics about the number of RPCs sent by the Cache Manager and how long
6731 they take to complete; and statistics regarding things such as authentication, access, and PAG information
6732 associated with data access).</P
6747 >Directs the command to gather statistics just one time. Omit this option to have the command continue to
6748 probe the Cache Manager for statistics every 30 seconds. If you omit this option, you can use the <<SPAN
6754 >> interrupt signal to halt the command at any time.</P
6766 >Sets the frequency in seconds at which the program initiates probes to the Cache Manager. If you omit this
6767 argument, the default is 30 seconds.</P
6779 >Sets how long the utility runs before exiting, as a number of minutes. If you omit this argument, the
6780 default is 10 minutes.</P
6792 >Displays additional information as the command runs.</P
6808 >Auditing AFS Events on AIX File Servers</A
6811 >You can audit AFS events on AIX File Servers using an AFS mechanism that transfers audit information from AFS to the AIX
6812 auditing system. The following general classes of AFS events can be audited. For a complete list of specific AFS audit events,
6815 >Appendix D, AIX Audit Events</A
6819 >Authentication and Identification Events</P
6827 >Privilege Required Events</P
6831 >Object Creation and Deletion Events</P
6835 >Attribute Modification Events</P
6839 >Process Control Events</P
6850 >This section assumes familiarity with the AIX auditing system. For more information, see the <SPAN
6857 > for the version of AIX you are using.</P
6866 >Configuring AFS Auditing on AIX File Servers</A
6869 >The directory <SPAN
6873 >/usr/afs/local/audit</B
6875 > contains three files that contain the information
6876 needed to configure AIX File Servers to audit AFS events: <UL
6885 > file contains information on auditable AFS events. The contents
6886 of this file are integrated into the corresponding AIX events file (<SPAN
6890 >/etc/security/audit/events</B
6902 > file defines the six classes of AFS audit events and the events
6903 that make up each class. It also defines the classes of AFS audit events to audit for the File Server, which runs as the
6904 local superuser <SPAN
6910 >. The contents of this file must be integrated into the
6911 corresponding AIX config file (<SPAN
6915 >/etc/security/audit/config</B
6927 > file contains a list of information about audited files. You
6928 must only audit files in the local file space. The contents of this file must be integrated into the corresponding AIX
6933 >/etc/security/audit/objects</B
6940 >Once you have properly configured these files to include the AFS-relevant information, use the AIX auditing system to
6941 start up and shut down the auditing.</P
6949 >To enable AFS auditing</A
6955 >Create the following string in the file <SPAN
6959 >/usr/afs/local/Audit</B
6961 > on each File Server on
6962 which you plan to audit AFS events: <PRE
6963 CLASS="programlisting"
6968 >AFS_AUDIT_AllEvents</B
6981 > command (with the <SPAN
6988 to stop and restart all server processes on each File Server. For instructions on using this command, see <A
6989 HREF="c6449.html#HDRWQ170"
6990 >Stopping and Immediately Restarting Processes</A
7001 >To disable AFS auditing</A
7007 >Remove the contents of the file <SPAN
7011 >/usr/afs/local/Audit</B
7013 > on each File Server for which
7014 you are no longer interested in auditing AFS events.</P
7024 > command (with the <SPAN
7031 to stop and restart all server processes on each File Server. For instructions on using this command, see <A
7032 HREF="c6449.html#HDRWQ170"
7033 >Stopping and Immediately Restarting Processes</A
7045 SUMMARY="Footer navigation table"
7084 >Backing Up and Restoring AFS Data</TD
7098 >Managing Server Encryption Keys</TD
git://git.openafs.org / openafs.git / blob