3 fileserver - Initializes the File Server component of the fs process
10 B<fileserver> S<<< [B<-d> <I<debug level>>] >>>
11 S<<< [B<-p> <I<number of processes>>] >>>
12 S<<< [B<-spare> <I<number of spare blocks>>] >>>
13 S<<< [B<-pctspare> <I<percentage spare>>] >>> S<<< [B<-b> <I<buffers>>] >>>
14 S<<< [B<-l> <I<large vnodes>>] >>> S<<< [B<-s> <I<small nodes>>] >>>
15 S<<< [B<-vc> <I<volume cachesize>>] >>> S<<< [B<-w> <I<call back wait interval>>] >>>
16 S<<< [B<-cb> <I<number of call backs>>] >>> [B<-banner>] [B<-novbc>]
17 S<<< [B<-implicit> <I<admin mode bits: rlidwka>>] >>> [B<-readonly>]
18 S<<< [B<-hr> <I<number of hours between refreshing the host cps>>] >>>
19 S<<< [B<-busyat> <I<< redirect clients when queue > n >>>] >>>
20 [B<-nobusy>] S<<< [B<-rxpck> <I<number of rx extra packets>>] >>>
21 [B<-rxdbg>] [B<-rxdbge>] S<<< [B<-rxmaxmtu> <I<bytes>>] >>>
22 S<<< [B<-rxbind> <I<address to bind the Rx socket to>>] >>>
23 S<<< [B<-vattachpar> <I<number of volume attach threads>>] >>>
24 S<<< [B<-m> <I<min percentage spare in partition>>] >>>
25 [B<-lock>] [B<-L>] [B<-S>] S<<< [B<-k> <I<stack size>>] >>>
26 S<<< [B<-realm> <I<Kerberos realm name>>] >>>
27 S<<< [B<-udpsize> <I<size of socket buffer in bytes>>] >>>
28 S<<< [B<-sendsize> <I<size of send buffer in bytes>>] >>>
29 S<<< [B<-abortthreshold> <I<abort threshold>>] >>>
30 S<<< [B<-auditlog> <I<path to log file>>] >>>
31 [B<-enable_peer_stats>] [B<-enable_process_stats>] [B<-help>]
38 The B<fileserver> command initializes the File Server component of the
39 C<fs> process. In the conventional configuration, its binary file is
40 located in the F</usr/afs/bin> directory on a file server machine.
42 The B<fileserver> command is not normally issued at the command shell
43 prompt, but rather placed into a database server machine's
44 F</usr/afs/local/BosConfig> file with the B<bos create> command. If it is
45 ever issued at the command shell prompt, the issuer must be logged onto a
46 file server machine as the local superuser C<root>.
48 The File Server creates the F</usr/afs/logs/FileLog> log file as it
49 initializes, if the file does not already exist. It does not write a
50 detailed trace by default, but the B<-d> option may be used to
51 increase the amount of detail. Use the B<bos getlog> command to
52 display the contents of the log file.
54 The command's arguments enable the administrator to control many aspects
55 of the File Server's performance, as detailed in L<OPTIONS>. By default
56 the B<fileserver> command sets values for many arguments that are suitable
57 for a medium-sized file server machine. To set values suitable for a small
58 or large file server machine, use the B<-S> or B<-L> flag
59 respectively. The following list describes the parameters and
60 corresponding argument for which the B<fileserver> command sets default
61 values, and the table below summarizes the setting for each of the three
68 The maximum number of lightweight processes (LWPs) the File Server uses to
69 handle requests for data; corresponds to the B<-p> argument. The File
70 Server always uses a minimum of 32 KB of memory for these processes.
74 The maximum number of directory blocks the File Server caches in memory;
75 corresponds to the B<-b> argument. Each cached directory block (buffer)
76 consumes 2,092 bytes of memory.
80 The maximum number of large vnodes the File Server caches in memory for
81 tracking directory elements; corresponds to the B<-l> argument. Each large
82 vnode consumes 292 bytes of memory.
86 The maximum number of small vnodes the File Server caches in memory for
87 tracking file elements; corresponds to the B<-s> argument. Each small
88 vnode consumes 100 bytes of memory.
92 The maximum volume cache size, which determines how many volumes the File
93 Server can cache in memory before having to retrieve data from disk;
94 corresponds to the B<-vc> argument.
98 The maximum number of callback structures the File Server caches in
99 memory; corresponds to the B<-cb> argument. Each callback structure
100 consumes 16 bytes of memory.
104 The maximum number of Rx packets the File Server uses; corresponds to the
105 B<-rxpck> argument. Each packet consumes 1544 bytes of memory.
109 The default values are:
111 Parameter (Argument) Small (-S) Medium Large (-L)
112 ---------------------------------------------------------------------
113 Number of LWPs (-p) 6 9 12
114 Number of cached dir blocks (-b) 70 90 120
115 Number of cached large vnodes (-l) 200 400 600
116 Number of cached small vnodes (-s) 200 400 600
117 Maximum volume cache size (-vc) 200 400 600
118 Number of callbacks (-cb) 20,000 60,000 64,000
119 Number of Rx packets (-rxpck) 100 150 200
121 To override any of the values, provide the indicated argument (which can
122 be combined with the B<-S> or B<-L> flag).
124 The amount of memory required for the File Server varies. The approximate
125 default memory usage is 751 KB when the B<-S> flag is used (small
126 configuration), 1.1 MB when all defaults are used (medium configuration),
127 and 1.4 MB when the B<-L> flag is used (large configuration). If
128 additional memory is available, increasing the value of the B<-cb> and
129 B<-vc> arguments can improve File Server performance most directly.
131 By default, the File Server allows a volume to exceed its quota by 1 MB
132 when an application is writing data to an existing file in a volume that
133 is full. The File Server still does not allow users to create new files in
134 a full volume. To change the default, use one of the following arguments:
140 Set the B<-spare> argument to the number of extra kilobytes that the File
141 Server allows as overage. A value of C<0> allows no overage.
145 Set the B<-pctspare> argument to the percentage of the volume's quota the
146 File Server allows as overage.
150 By default, the File Server implicitly grants the C<a> (administer) and
151 C<l> (lookup) permissions to system:administrators on the access control
152 list (ACL) of every directory in the volumes stored on its file server
153 machine. In other words, the group's members can exercise those two
154 permissions even when an entry for the group does not appear on an ACL. To
155 change the set of default permissions, use the B<-implicit> argument.
157 The File Server maintains a I<host current protection subgroup> (I<host
158 CPS>) for each client machine from which it has received a data access
159 request. Like the CPS for a user, a host CPS lists all of the Protection
160 Database groups to which the machine belongs, and the File Server compares
161 the host CPS to a directory's ACL to determine in what manner users on the
162 machine are authorized to access the directory's contents. When the B<pts
163 adduser> or B<pts removeuser> command is used to change the groups to
164 which a machine belongs, the File Server must recompute the machine's host
165 CPS in order to notice the change. By default, the File Server contacts
166 the Protection Server every two hours to recompute host CPSs, implying
167 that it can take that long for changed group memberships to become
168 effective. To change this frequency, use the B<-hr> argument.
170 The File Server stores volumes in partitions. A partition is a
171 filesystem or directory on the server machine that is named C</vicepX>
172 or C</vicepXX> where XX is "a" through "z" or "aa" though "zz". The
173 File Server expects that the /vicepXX directories are each on a
174 dedicated filesystem. The File Server will only use a /vicepXX if it's
175 a mountpoint for another filesystem, unless the file
176 C</vicepXX/AlwaysAttach> exists. The data in the partition is a
177 special format that can only be access using OpenAFS commands or an
180 The File Server generates the following message when a partition is nearly
183 No space left on device
185 This command does not use the syntax conventions of the AFS command
186 suites. Provide the command name and all option names in full.
190 Do not use the B<-k> and B<-w> arguments, which are intended for use
191 by the AFS Development group only. Changing them from their default
192 values can result in unpredictable File Server behavior. In any case,
193 on many operating systems the File Server uses native threads rather
194 than the LWP threads, so using the B<-k> argument to set the number of
195 LWP threads has no effect.
197 Do not specify both the B<-spare> and B<-pctspare> arguments. Doing so
198 causes the File Server to exit, leaving an error message in the
199 F</usr/afs/logs/FileLog> file.
201 Options that are available only on some system types, such as the B<-m>
202 and B<-lock> options, appear in the output generated by the B<-help>
203 option only on the relevant system type.
209 =item B<-auditlog> <I<log path>>
211 Set and enable auditing.
213 =item B<-d> <I<debug level>>
215 Sets the detail level for the debugging trace written to the
216 F</usr/afs/logs/FileLog> file. Provide one of the following values, each
217 of which produces an increasingly detailed trace: C<0>, C<1>, C<5>, C<25>,
218 and C<125>. The default value of C<0> produces only a few messages.
220 =item B<-p> <I<number of processes>>
222 Sets the number of threads to run. Provide a positive integer. The File
223 Server creates and uses five threads for special purposes, in addition to
224 the number specified (but if this argument specifies the maximum possible
225 number, the File Server automatically uses five of the threads for its own
228 The maximum number of threads can differ in each release of AFS. Consult
229 the I<IBM AFS Release Notes> for the current release.
231 =item B<-spare> <I<number of spare blocks>>
233 Specifies the number of additional kilobytes an application can store in a
234 volume after the quota is exceeded. Provide a positive integer; a value of
235 C<0> prevents the volume from ever exceeding its quota. Do not combine
236 this argument with the B<-pctspare> argument.
238 =item B<-pctspare> <I<percentage spare>>
240 Specifies the amount by which the File Server allows a volume to exceed
241 its quota, as a percentage of the quota. Provide an integer between C<0>
242 and C<99>. A value of C<0> prevents the volume from ever exceeding its
243 quota. Do not combine this argument with the B<-spare> argument.
245 =item B<-b> <I<buffers>>
247 Sets the number of directory buffers. Provide a positive integer.
249 =item B<-l> <I<large vnodes>>
251 Sets the number of large vnodes available in memory for caching directory
252 elements. Provide a positive integer.
254 =item B<-s> <I<small nodes>>
256 Sets the number of small vnodes available in memory for caching file
257 elements. Provide a positive integer.
259 =item B<-vc> <I<volume cachesize>>
261 Sets the number of volumes the File Server can cache in memory. Provide a
264 =item B<-w> <I<call back wait interval>>
266 Sets the interval at which the daemon spawned by the File Server performs
267 its maintenance tasks. Do not use this argument; changing the default
268 value can cause unpredictable behavior.
270 =item B<-cb> <I<number of callbacks>>
272 Sets the number of callbacks the File Server can track. Provide a positive
277 Prints the following banner to F</dev/console> about every 10 minutes.
279 File Server is running at I<time>.
283 Prevents the File Server from breaking the callbacks that Cache Managers
284 hold on a volume that the File Server is reattaching after the volume was
285 offline (as a result of the B<vos restore> command, for example). Use of
286 this flag is strongly discouraged.
288 =item B<-implicit> <I<admin mode bits>>
290 Defines the set of permissions granted by default to the
291 system:administrators group on the ACL of every directory in a volume
292 stored on the file server machine. Provide one or more of the standard
293 permission letters (C<rlidwka>) and auxiliary permission letters
294 (C<ABCDEFGH>), or one of the shorthand notations for groups of permissions
295 (C<all>, C<none>, C<read>, and C<write>). To review the meaning of the
296 permissions, see the B<fs setacl> reference page.
298 =item B<-hr> <I<number of hours between refreshing the host cps>>
300 Specifies how often the File Server refreshes its knowledge of the
301 machines that belong to protection groups (refreshes the host CPSs for
302 machines). The File Server must update this information to enable users
303 from machines recently added to protection groups to access data for which
304 those machines now have the necessary ACL permissions.
306 =item B<-busyat> <I<< redirect clients when queue > n >>>
308 Defines the number of incoming RPCs that can be waiting for a response
309 from the File Server before the File Server returns the error code
310 C<VBUSY> to the Cache Manager that sent the latest RPC. In response, the
311 Cache Manager retransmits the RPC after a delay. This argument prevents
312 the accumulation of so many waiting RPCs that the File Server can never
313 process them all. Provide a positive integer. The default value is
316 =item B<-rxpck> <I<number of rx extra packets>>
318 Controls the number of Rx packets the File Server uses to store data for
319 incoming RPCs that it is currently handling, that are waiting for a
320 response, and for replies that are not yet complete. Provide a positive
325 Writes a trace of the File Server's operations on Rx packets to the file
326 F</usr/afs/logs/rx_dbg>.
330 Writes a trace of the File Server's operations on Rx events (such as
331 retransmissions) to the file F</usr/afs/logs/rx_dbg>.
333 =item F<-m> <I<min percentage spare in partition>>
335 Specifies the percentage of each AFS server partition that the AIX version
336 of the File Server creates as a reserve. Specify an integer value between
337 C<0> and C<30>; the default is 8%. A value of C<0> means that the
338 partition can become completely full, which can have serious negative
343 Prevents any portion of the fileserver binary from being paged (swapped)
344 out of memory on a file server machine running the IRIX operating system.
348 Sets values for many arguments in a manner suitable for a large file
349 server machine. Combine this flag with any option except the B<-S> flag;
350 omit both flags to set values suitable for a medium-sized file server
355 Sets values for many arguments in a manner suitable for a small file
356 server machine. Combine this flag with any option except the B<-L> flag;
357 omit both flags to set values suitable for a medium-sized file server
360 =item B<-k> <I<stack size>>
362 Sets the LWP stack size in units of 1 kilobyte. Do not use this argument,
363 and in particular do not specify a value less than the default of C<24>.
365 =item B<-realm> <I<Kerberos realm name>>
367 Defines the Kerberos realm name for the File Server to use. If this
368 argument is not provided, it uses the realm name corresponding to the cell
369 listed in the local F</usr/afs/etc/ThisCell> file.
371 =item B<-udpsize> <I<size of socket buffer in bytes>>
373 Sets the size of the UDP buffer, which is 64 KB by default. Provide a
374 positive integer, preferably larger than the default.
376 =item B<-enable_peer_stats>
378 Activates the collection of Rx statistics and allocates memory for their
379 storage. For each connection with a specific UDP port on another machine,
380 a separate record is kept for each type of RPC (FetchFile, GetStatus, and
381 so on) sent or received. To display or otherwise access the records, use
382 the Rx Monitoring API.
384 =item B<-enable_process_stats>
386 Activates the collection of Rx statistics and allocates memory for their
387 storage. A separate record is kept for each type of RPC (FetchFile,
388 GetStatus, and so on) sent or received, aggregated over all connections to
389 other machines. To display or otherwise access the records, use the Rx
394 Prints the online help for this command. All other valid options are
401 The following B<bos create> command creates an fs process on the file
402 server machine C<fs2.abc.com> that uses the large configuration size, and
403 allows volumes to exceed their quota by 10%. Type the command on a single
406 % bos create -server fs2.abc.com -instance fs -type fs \
407 -cmd "/usr/afs/bin/fileserver -pctspare 10 \
408 -L" /usr/afs/bin/volserver /usr/afs/bin/salvager
411 =head1 TROUBLESHOOTING
413 Sending process signals to the File Server Process can change its
414 behavior in the following ways:
416 Process Signal OS Result
417 ---------------------------------------------------------------------
419 File Server XCPU Unix Prints a list of client IP
422 File Server USR2 Windows Prints a list of client IP
425 File Server POLL HPUX Prints a list of client IP
428 Any server TSTP Any Increases Debug level by a power
429 of 5 -- 1,5,25,125, etc.
430 This has the same effect as the
431 -debug XXX command-line option.
433 Any Server HUP Any Resets Debug level to 0
435 File Server TERM Any Run minor instrumentation over
436 the list of descriptors.
438 Other Servers TERM Any Causes the process to quit.
440 File Server QUIT Any Causes the File Server to Quit.
441 Bos Server knows this.
443 The basic metric of whether an AFS file server is doing well is the number
444 of connections waiting for a thread,
445 which can be found by running the following command:
447 % rxdebug <server> | grep waiting_for | wc -l
449 Each line returned by C<rxdebug> that contains the text "waiting_for"
450 represents a connection that's waiting for a file server thread.
452 If the blocked connection count is ever above 0, the server is having
453 problems replying to clients in a timely fashion. If it gets above 10,
454 roughly, there will be noticable slowness by the user. The total number of
455 connections is a mostly irrelevant number that goes essentially
456 monotonically for as long as the server has been running and then goes back
457 down to zero when it's restarted.
459 The most common cause of blocked connections rising on a server is some
460 process somewhere performing an abnormal number of accesses to that server
461 and its volumes. If multiple servers have a blocked connection count, the
462 most likely explanation is that there is a volume replicated between those
463 servers that is absorbing an abnormally high access rate.
465 To get an access count on all the volumes on a server, run:
467 % vos listvol <server> -long
469 and save the output in a file. The results will look like a bunch of B<vos
470 examine> output for each volume on the server. Look for lines like:
472 40065 accesses in the past day (i.e., vnode references)
474 and look for volumes with an abnormally high number of accesses. Anything
475 over 10,000 is fairly high, but some volumes like root.cell and other
476 volumes close to the root of the cell will have that many hits routinely.
477 Anything over 100,000 is generally abnormally high. The count resets about
480 Another approach that can be used to narrow the possibilities for a
481 replicated volume, when multiple servers are having trouble, is to find all
482 replicated volumes for that server. Run:
484 % vos listvldb -server <server>
486 where <server> is one of the servers having problems to refresh the VLDB
489 % vos listvldb -server <server> -part <partition>
491 to get a list of all volumes on that server and partition, including every
492 other server with replicas.
494 Once the volume causing the problem has been identified, the best way to
495 deal with the problem is to move that volume to another server with a low
496 load or to stop any runaway programs that are accessing that volume
497 unnecessarily. Often the volume will be enough information to tell what's
500 If you still need additional information about who's hitting that server,
501 sometimes you can guess at that information from the failed callbacks in the
502 F<FileLog> log in F</var/log/afs> on the server, or from the output of:
504 % /usr/afsws/etc/rxdebug <server> -rxstats
506 but the best way is to turn on debugging output from the file server.
507 (Warning: This generates a lot of output into FileLog on the AFS server.)
508 To do this, log on to the AFS server, find the PID of the fileserver
513 where <pid> is the PID of the file server process. This will raise the
514 debugging level so that you'll start seeing what people are actually doing
515 on the server. You can do this up to three more times to get even more
516 output if needed. To reset the debugging level back to normal, use (The
517 following command will NOT terminate the file server):
521 The debugging setting on the File Server should be reset back to normal when
522 debugging is no longer needed. Otherwise, the AFS server may well fill its
523 disks with debugging output.
525 The lines of the debugging output that are most useful for debugging load
528 SAFS_FetchStatus, Fid = 2003828163.77154.82248, Host 171.64.15.76
529 SRXAFS_FetchData, Fid = 2003828163.77154.82248
531 (The example above is partly truncated to highlight the interesting
532 information). The Fid identifies the volume and inode within the volume;
533 the volume is the first long number. So, for example, this was:
535 % vos examine 2003828163
536 pubsw.matlab61 2003828163 RW 1040060 K On-line
537 afssvr5.Stanford.EDU /vicepa
538 RWrite 2003828163 ROnly 2003828164 Backup 2003828165
540 Creation Mon Aug 6 16:40:55 2001
541 Last Update Tue Jul 30 19:00:25 2002
542 86181 accesses in the past day (i.e., vnode references)
544 RWrite: 2003828163 ROnly: 2003828164 Backup: 2003828165
546 server afssvr5.Stanford.EDU partition /vicepa RW Site
547 server afssvr11.Stanford.EDU partition /vicepd RO Site
548 server afssvr5.Stanford.EDU partition /vicepa RO Site
550 and from the Host information one can tell what system is accessing that
553 Note that the output of L<vos_examine(1)> also includes the access count, so
554 once the problem has been identified, vos examine can be used to see if the
555 access count is still increasing. Also remember that you can run vos
556 examine on the read-only replica (e.g., pubsw.matlab61.readonly) to see the
557 access counts on the read-only replica on all of the servers that it's
560 =head1 PRIVILEGE REQUIRED
562 The issuer must be logged in as the superuser C<root> on a file server
563 machine to issue the command at a command shell prompt. It is conventional
564 instead to create and start the process by issuing the B<bos create>
580 IBM Corporation 2000. <http://www.ibm.com/> All Rights Reserved.
582 This documentation is covered by the IBM Public License Version 1.0. It was
583 converted from HTML to POD by software written by Chas Williams and Russ
584 Allbery, based on work by Alf Wachsmann and Elizabeth Cassell.