2 # Copyright (c) 1996, 2001 Carnegie Mellon University
5 # See CMU_copyright.ph for use and distribution information
7 package OpenAFS::wrapper;
11 OpenAFS::wrapper - AFS command wrapper
16 %result = &wrapper($cmd, \@args, \@pspec, \%options);
20 This module provides a generic wrapper for calling an external program and
21 parsing its output. It is primarily intended for use by AFStools for calling
22 AFS commands, but is general enough to be used for running just about any
23 utility program. The wrapper is implemented by a single function,
24 B<OpenAFS::wrapper::wrapper>, which takes several arguments:
30 The command to run. This can be a full path, or it can be a simple command
31 name, in which case B<wrapper()> will find the binary on its internal path.
35 A reference to the list of arguments to be passed to the command. Each
36 element of the list is passed as a single argument, as in B<exec()>.
40 A reference to the list describing how to parse the command's output.
41 See below for details.
45 A reference to a table of command execution and parsing options.
49 On success, B<wrapper()> returns an associative array of data gathered
50 from the command's output. The exact contents of this array are
51 caller-defined, and depend on the parsing instructions given. On failure,
52 an exception will be thrown (using B<die>), describing the reason for the
55 The I<%options> table may be used to pass any or all of the following
56 options into B<wrapper()>, describing how the command should be executed
57 and its output parsed:
63 If specified and nonzero, the command's stderr will be passed directly
64 to the calling program's, instead of being parsed. This is useful when
65 we want to process the command's output, but let the user see any
66 diagnostic output or error messages.
70 If specified and nonzero, the command's stdout will be passed directly
71 to the calling program's, instead of being parsed. This is useful when
72 the command being run produces diagnostic or error messages on stderr
73 that we want to parse, but provides bulk data on stdout that we don't
74 want to touch (e.g. B<vos dump> when the output file is stdout).
78 If specified, the path to be used for the program to execute, instead of
79 deriving it from the command name. This is useful when we want the
80 command's argv[0] (which is always I<$cmd}) to be different from the
85 If specified and nonzero, the built-in instructions for catching errors
86 from the command will be added to the end of the instructions in @pspec
87 instead of to the beginning.
91 =head1 PARSING COMMAND OUTPUT
93 The I<@pspec> list describes how to parse command output. Each element
94 of the list acts like an "instruction" describing how to parse the command's
95 output. As each line of output is received from the program, the parsing
96 instructions are run over that line in order. This process continues for
97 every line of output until the program terminates, or the process is
98 aborted early by flow-control operators.
100 Each parsing instruction is a reference to a list, which consists of a
101 regular expression and a list of "actions". As a line of output is
102 processed, it is compared to each instruction's regexp in turn. Whenever
103 a match is found, the actions associated with that instruction are taken,
104 in order. Each instruction's regexp may contain one or more parenthesized
105 subexpressions; generally, each "action" uses up one subexpression, but there
106 are some exceptions. Due to the current design of B<wrapper()>, each regexp
107 must have at least one subexpression, even if it is not used.
109 The acceptable actions are listed below, each followed by a number in brackets
110 indicating how many subexpressions are "used" by this action. It is an error
111 if there are not enough subexpressions left to satisfy an action. In the
112 following descriptions, I<$action> is the action itself (typically a string or
113 reference), I<$value> is the value of the subexpression that will be used, and
114 I<%result> is the result table that will be returned by B<wrapper> when the
121 Sets $result{$action} to $value. Note that several specific strings have
122 special meaning, and more may be added in the future. To ensure compatibility
123 with future versions of B<wrapper>, use only valid Perl identifiers as
128 Sets $$action to $value.
132 Pushes the remaining subexpression values onto @$action. This action uses
133 all remaining subexpression values.
137 Sets $$action{$value0} to $value1.
141 Calls the referenced function, with all remaining subexpression values as
142 its arguments. Any values returned by the function will be used to refill
143 the (now empty) subexpression value list, and thus may be used as arguments
144 by subsequent actions. If only a few values are required, use a function
147 sub usetwo { # uses two values and preserves the rest
148 my($val1, $val2, @rest) = @_;
150 print STDOUT "Got $val1, $val2\n";
156 End processing for this line of output, ignoring any remaining instructions.
157 Remaining actions in this instruction will be processed.
161 Skip the next I<n> instructions. This, along with the '.' action, can be
162 used to build simple flow-control constructs based on the contents of
167 Signal an error after this instruction. Remaining actions in this instruction
168 will be processed, but no further instructions will be processed for this
169 line, and no further lines of output will be processed. If I<x> is given,
170 it will be used as a regexp to match against the B<previous> line of output,
171 and the first parenthesized subexpression resulting from that match will be
172 used as the error string. Otherwise, one subexpression from the current
173 line will be used up as the error string.
177 Prints $value to STDOUT.
183 use OpenAFS::CMU_copyright;
184 use OpenAFS::util qw(:DEFAULT :afs_internal);
191 @EXPORT = qw(&wrapper);
192 @EXPORT_OK = qw(&wrapper &fast_wrapper);
195 my($cmd, $args, $instrs, $options) = @_;
196 my($prevline, $pid, $exception);
197 my(@instrs, $instr, $action, @values, $path);
199 my(@werrinstrs) = ([ '^(wrapper\:.*)', '-' ]);
200 my(@cerrinstrs) = ([ '^(' . $cmd . '\:.*)', '-' ],
201 [ '^(' . $path . '\:.*)', '-' ]);
203 if ($options->{errors_last}) {
204 @instrs = (@werrinstrs, @$instrs, @cerrinstrs);
206 @instrs = (@werrinstrs, @cerrinstrs, @$instrs);
209 if ($options->{path}) {
210 $path = $options->{path};
211 } elsif ($cmd =~ /^\//) {
214 $path = $AFScmd{$cmd};
217 if ($AFS_Trace{wrapper}) {
218 print STDERR "Instructions:\n";
219 foreach $instr (@$instrs) {
220 print STDERR " /", $instr->[0], "/\n";
221 if ($AFS_Trace{wrapper} > 2) {
222 my(@actions) = @$instr;
225 join(', ', map { ref($_) ? "<" . ref($_) . " reference>"
233 if ($options->{pass_stdout}) {
234 open(REALSTDOUT, ">&STDOUT");
236 $pid = open(AFSCMD, "-|");
237 if (!defined($pid)) {
238 die "wrapper: Fork failed for $cmd: $!\n";
241 ## Run the appropriate program
244 if ($AFS_Trace{wrapper} > 1) {
245 print STDERR "Command: $path ", join(' ', @$args), "\n";
248 open(STDERR, ">&STDOUT") if (!$options{pass_stderr});
249 if ($options{pass_stdout}) {
250 open(STDOUT, ">&REALSTDOUT");
254 { exec($path $cmd, @$args); }
255 # Need to be careful here - we might be doing "vos dump" to STDOUT
256 if ($options{pass_stdout}) {
257 print STDERR "wrapper: Exec failed for $cmd: $!\n";
259 print STDOUT "wrapper: Exec failed for $cmd: $!\n";
263 if ($options{pass_stdout}) {
267 ## Now, parse the output
272 print STDERR $_ if ($AFS_Trace{wrapper} > 3);
276 foreach $instr (@instrs) {
277 my($dot, $action, @actions);
285 @values = ($_ =~ $instr->[0]);
286 next instr if (!@values);
294 foreach $action (@actions) {
295 if (ref($action) eq 'SCALAR') {
297 $$action = shift(@values);
301 } elsif (ref($action) eq 'ARRAY') {
302 push(@$action, @values);
304 } elsif (ref($action) eq 'HASH') {
306 $$action{$values[0]} = $values[1];
307 shift(@values); shift(@values);
309 $$action{shift @values} = '';
314 } elsif (ref($action) eq 'CODE') {
315 @values = &$action(@values);
316 } elsif (ref($action)) {
317 $exception = "Unknown reference to " . ref($action)
318 . "in parse instructions";
320 } else { ## Must be a string!
321 if ($action eq '.') {
323 } elsif ($action =~ /\+(\d+)/) {
325 } elsif ($action =~ /-(.*)/) {
328 if ($pat && $prevline) {
329 ($exception) = ($prevline =~ $pat);
331 $exception = shift(@values);
335 } elsif ($action eq '?') {
336 print STDOUT (@values ? shift(@values) : $_), "\n";
338 $result{$action} = shift(@values);
345 last line if ($exception);
346 last instr if ($dot);
351 $exception .= "\n" if ($exception && $exception !~ /\n$/);
352 die $exception if ($exception);
357 ## Generate code for a fast wrapper (see example below)
359 my($instrs, $refs) = @_;
360 my($SRC, $N, $N1, $X, $instr, $pattern, @actions, $action);
368 my($prevline, @values, $skip, $exception);
370 line: while (<$FD>) {
373 $SRC .= " print STDERR \$_;\n" if ($AFS_Trace{'wrapper'} > 3);
376 foreach $instr (@$instrs) {
377 ($pattern, @actions) = (@$instr);
378 $SRC .= ($pattern ? <<"#####" : <<"#####");
381 die \$exception if \$exception;
382 if (\$skip) { \$skip-- } else {
383 \@values = (\$_ =~ /$pattern/);
388 die \$exception if \$exception;
389 if (\$skip) { \$skip-- } else {
394 foreach $action (@actions) {
395 if (ref($action) eq 'SCALAR') {
396 $refs[++$X] = $action;
399 if (\@values) { \${\$refs[$X]} = shift (\@values) }
400 else { goto instr_$N1 }
403 } elsif (ref($action) eq 'ARRAY') {
404 $refs[++$X] = $action;
407 push(\@{\$refs[$X]}, \@values);
411 } elsif (ref($action) eq 'HASH') {
412 $refs[++$X] = $action;
416 \$refs[$X]{\$values[0]} = shift(\$values[1]);
417 shift(\@values); shift(\@values);
419 \$refs[$X]{shift(\@values)} = '';
426 } elsif (ref($action) eq 'CODE') {
427 $refs[++$X] = $action;
428 $SRC .= "\n \@values = \$refs[$X]->(\@values);\n";
430 } elsif (ref($action)) {
431 die "Unknown reference to " . ref($action) . "in parse instructions\n";
433 } elsif ($action eq '.') {
434 $SRC .= "\n next line;\n";
436 } elsif ($action eq '?') {
439 if (\@values) { print STDOUT shift(\@values), "\\n" }
440 else { print STDOUT \$_, "\\n" }
443 } elsif ($action =~ /\+(\d+)/) {
444 $SRC .= "\n \$skip = $1;\n";
446 } elsif ($action =~ /-(.*)/) {
447 $SRC .= $1 ? <<"#####" : <<"#####";
449 if (\$prevline) { (\$exception) = (\$prevline =~ /$1/) }
450 elsif (\@values) { \$exception = shift(\@values) }
451 else { \$exception = \$_ }
454 if (\@values) { \$exception = shift(\@values) }
455 else { \$exception = \$_ }
461 if (\@values) { \$result{"\Q$action\E"} = shift(\@values) }
462 else { goto instr_$N1 }
476 die $exception if $exception;
485 ####################### Example code #######################
487 # my($FD, $refs) = @_;
488 # my($prevline, @values, $skip, $exception);
490 # line: while (<$FD>) {
491 # print STDERR $_; ## if ($AFS_Trace{'wrapper'} > 3);
494 # ## Following block repeated for each instruction
496 # die $exception if $exception;
497 # if ($skip) { $skip-- } else {
498 # @values = ($_ =~ /## pattern ##/); ## () if no pattern
499 # if (@values) { ## 1 if no pattern
500 # ## For each action, include one of the following blocks:
503 # if (@values) { ${$refs[X]} = shift (@values) }
504 # else { goto instr_N+1 }
507 # push(@{$refs[X]}, @values);
512 # $refs[X]{shift(@values)} = shift(@values);
513 # } elsif (@values) {
514 # $refs[X]{shift(@values)} = '';
521 # @values = $refs[X]->(@values);
527 # if (@values) { print STDOUT shift(@values), "\n" }
528 # else { print STDOUT $_, "\n" }
534 # if ($prevline) { ($exception) = ($prefline =~ /XXX/) }
535 # elsif (@values) { $exception = shift(@values) }
536 # else { $exception = $_ }
539 # if (@values) { $exception = shift(@values) }
540 # else { $exception = $_ }
543 # if (@values) { $result{XXX} = shift(@values) }
544 # else { goto instr_N+1 }
549 # die $exception if $exception;
553 ############################################################
556 ## The following does exactly the same thing as wrapper(),
557 ## but should be considerably faster. Instead of interpreting
558 ## parsing instructions, it translates them into perl code,
559 ## which is then compiled into the interpreter. The chief
560 ## benefit to this approach is that we no longer compile
561 ## one RE per instruction per line of input.
564 my($cmd, $args, $instrs, $options) = @_;
565 my(@instrs, $SRC, $CODE, $path, $pid, $refs, $FD, $exception);
567 my(@werrinstrs) = ([ '^(wrapper\:.*)', '-' ]);
568 my(@cerrinstrs) = ([ '^(' . $cmd . '\:.*)', '-' ],
569 [ '^(' . $path . '\:.*)', '-' ]);
573 if ($options->{errors_last}) {
574 @instrs = (@werrinstrs, @$instrs, @cerrinstrs);
576 @instrs = (@werrinstrs, @cerrinstrs, @$instrs);
578 $SRC = _fastwrap_gen(\@instrs, $refs);
581 if ($options->{path}) {
582 $path = $options->{path};
583 } elsif ($cmd =~ /^\//) {
586 $path = $AFScmd{$cmd};
589 if ($AFS_Trace{'wrapper'}) {
590 print STDERR "Instructions:\n";
591 foreach $instr (@$instrs) {
592 print STDERR " /", $instr->[0], "/\n";
593 if ($AFS_Trace{'wrapper'} > 2) {
594 my(@actions) = @$instr;
597 join(', ', map { ref($_) ? "<" . ref($_) . " reference>"
604 if ($AFS_Trace{'wrapper'} > 2) { print STDERR "Input parse code:\n$SRC\n" }
607 if ($options->{pass_stdout}) {
608 open(REALSTDOUT, ">&STDOUT");
610 $pid = open($FD, "-|");
611 if (!defined($pid)) {
612 die "wrapper: Fork failed for $cmd: $!\n";
615 ## Run the appropriate program
617 if ($AFS_Trace{'wrapper'} > 1) {
618 print STDERR "Command: $path ", join(' ', @$args), "\n";
621 open(STDERR, ">&STDOUT") if (!$options{pass_stderr});
622 if ($options{pass_stdout}) {
623 open(STDOUT, ">&REALSTDOUT");
627 { exec($path $cmd, @$args) }
628 # Need to be careful here - we might be doing "vos dump" to STDOUT
629 if ($options{pass_stdout}) {
630 print STDERR "wrapper: Exec failed for $cmd: $!\n";
632 print STDOUT "wrapper: Exec failed for $cmd: $!\n";
636 if ($options{pass_stdout}) {
640 ## Now, parse the output
641 eval { $CODE->($FD, $refs) };
646 $exception .= "\n" if ($exception && $exception !~ /\n$/);
647 die $exception if ($exception);
656 The following set of instructions is used by B<wrapper> to detect errors
657 issued by the command, or by the child process spawned to invoke the command.
658 I<$cmd> is the name of the command to run, and I<$path> is the path to the
659 binary actually invoked.
661 [ '^(wrapper\:.*)', '-' ]
662 [ '^(' . $cmd . '\:.*)', '-' ]
663 [ '^(' . $path . '\:.*)', '-' ]
665 The following instruction is added by the B<OpenAFS::vos> module to catch errors
666 generated by B<vos> commands, which often take the form of a generic error
667 message (Error in vos XXX command), with a description of the specific problem
668 on the preceeding line:
670 [ 'Error in vos (.*) command', '-(.*)' ]
672 If the AFStools parameter I<vostrace> is nonzero, the following instruction
673 is added to force all lines of output to be copied to STDOUT. Note that this
674 is different from specifying the I<pass_stdout> option, which would pass the
675 command's STDOUT directly to ours without parsing it.
679 B<OpenAFS::vos::AFS_vos_listvldb> uses the following instructions to parse the
680 output of "vos listvldb". This is a fairly complex example, which illustrates
681 many of the features of B<wrapper>.
683 1 ['^(VLDB|Total) entries', '.']
685 my(%vinfo) = %OpenAFS::wrapper::result;
687 $vinfo{rosites} = [@rosites] if (@rosites);
688 $vlist{$vinfo{name}} = \%vinfo;
690 %OpenAFS::wrapper::result = ();
693 3 ['^(\S+)', 'name' ],
694 4 ['RWrite\:\s*(\d+)', 'rwid' ],
695 5 ['ROnly\:\s*(\d+)', 'roid' ],
696 6 ['Backup\:\s*(\d+)', 'bkid' ],
697 7 ['Volume is currently (LOCKED)', 'locked' ],
698 8 ['server (\S+) partition /vicep(\S+) RW Site', 'rwserv', 'rwpart'],
699 9 ['server (\S+) partition /vicep(\S+) RO Site', sub {
700 push(@rosites, [$_[0], $_[1]]);
703 Instruction 1 matchees the header and trailer lines printed out by B<vos>, and
704 terminates processing of those lines before instructions 2 and 3 have a chance
705 to match it. This is a simple example of a conditional - the next two
706 instructions are used only if this one doesn't match. If we wanted to consider
707 additional instructions even on lines that do match this one, we could place
708 them above this one, or use '+2' instead of '.', which would skip only the next
709 two instructions and allow remaining ones to be processed.
711 Instruction 2 matches the first line printed for each volume, stores away any
712 information that has been collected about the previous volume, and prepares for
713 the new one. Besides being a good example of use of a code reference as an
714 action, this instruction also takes advantage of the fact that B<wrapper>'s
715 %result array is a dynamically-scoped variable, and so can be modified by code
716 referenced in parsing instructions.
718 The remaining instructions are fairly simple. Instructions 3 through 8 use
719 simple strings to add information about the volume to %result. Instruction 9
720 is a bit more complicated; it uses a function to add a server/partition pair
721 to the current volume's list of RO sites.
725 The CMUCS AFStools, including this module are
726 Copyright (c) 1996, 2001 Carnegie Mellon University. All rights reserved.
727 For use and redistribution information, see CMUCS/CMU_copyright.pm