--- /dev/null
+ Writing TAP Tests
+
+Introduction
+
+ This is a guide for users of the C TAP Harness package or similar
+ TAP-based test harnesses explaining how to write tests. If your
+ package uses C TAP Harness as the test suite driver, you may want to
+ copy this document to an appropriate file name in your test suite as
+ documentation for contributors.
+
+About TAP
+
+ TAP is the Test Anything Protocol, a protocol for communication
+ between test cases and a test harness. This is the protocol used by
+ Perl for its internal test suite and for nearly all Perl modules,
+ since it's the format used by the build tools for Perl modules to run
+ tests and report their results.
+
+ A TAP-based test suite works with a somewhat different set of
+ assumptions than an xUnit test suite. In TAP, each test case is a
+ separate program. That program, when run, must produce output in the
+ following format:
+
+ 1..4
+ ok 1 - the first test
+ ok 2
+ # a diagnostic, ignored by the harness
+ not ok 3 - a failing test
+ ok 4 # skip a skipped test
+
+ The output should all go to standard output. The first line specifies
+ the number of tests to be run, and then each test produces output that
+ looks like either "ok <n>" or "not ok <n>" depending on whether the
+ test succeeded or failed. Additional information about the test can
+ be provided after the "ok <n>" or "not ok <n>", but is optional.
+ Additional diagnostics and information can be provided in lines
+ beginning with a "#".
+
+ Processing directives are supported after the "ok <n>" or "not ok <n>"
+ and start with a "#". The main one of interest is "# skip" which says
+ that the test was skipped rather than successful and optionally gives
+ the reason. Also supported is "# todo", which normally annotates a
+ failing test and indicates that test is expected to fail, optionally
+ providing a reason for why.
+
+ There are three more special cases. First, the initial line stating
+ the number of tests to run, called the plan, may appear at the end of
+ the output instead of the beginning. This can be useful if the number
+ of tests to run is not known in advance. Second, a plan in the form:
+
+ 1..0 # skip entire test case skipped
+
+ can be given instead, which indicates that this entire test case has
+ been skipped (generally because it depends on facilities or optional
+ configuration which is not present). Finally, if the test case
+ encounters a fatal error, it should print the text:
+
+ Bail out!
+
+ on standard output, optionally followed by an error message, and then
+ exit. This tells the harness that the test aborted unexpectedly.
+
+ The exit status of a successful test case should always be 0. The
+ harness will report the test as "dubious" if all the tests appeared to
+ succeed but it exited with a non-zero status.
+
+Writing TAP Tests
+
+ Environment
+
+ One of the special features of C TAP Harness is the environment that
+ it sets up for your test cases. If your test program is called under
+ the runtests driver, the environment variables SOURCE and BUILD will
+ be set to the top of the source tree and the top of the build tree,
+ respectively. You can use those environment variables to locate
+ additional test data, programs and libraries built as part of your
+ software build, and other supporting information needed by tests.
+
+ Perl
+
+ Since TAP is the native test framework for Perl, writing TAP tests in
+ Perl is very easy and extremely well-supported. If you've never
+ written tests in Perl before, start by reading the documentation for
+ Test::Tutorial and Test::Simple, which walks you through the basics,
+ including the TAP output syntax. Then, the best Perl module to use
+ for serious testing is Test::More, which provides a lot of additional
+ functions over Test::Simple including support for skipping tests,
+ bailing out, and not planning tests in advance. See the documentation
+ of Test::More for all the details and lots of examples.
+
+ C TAP Harness can run Perl test scripts directly and interpret the
+ results correctly, and similarly the Perl Test::Harness module can run
+ TAP tests written in other languages using, for example, the TAP
+ library that comes with C TAP Harness. However, the "prove" tool that
+ comes with Perl and runs tests makes some Perl-specific assumptions
+ that aren't always appropriate for packages that aren't written in
+ Perl.
+
+ C
+
+ C TAP Harness provides a basic TAP library that takes away most of the
+ pain of writing TAP test cases in C. A C test case should start with
+ a call to plan(), passing in the number of tests to run. Then, each
+ test should use is_int(), is_string(), is_double(), or is_hex() as
+ appropriate to compare expected and seen values, or ok() to do a
+ simpler boolean test. The is_*() functions take expected and seen
+ values and then a printf-style format string explaining the test
+ (which may be NULL). ok() takes a boolean and then the printf-style
+ string.
+
+ Here's a complete example test program that uses the C TAP library:
+
+ #include <tap/basic.h>
+
+ int
+ main(void)
+ {
+ plan(4);
+
+ ok(1, "the first test");
+ is_int(42, 42, NULL);
+ diag("a diagnostic, ignored by the harness");
+ ok(0, "a failing test");
+ skip("a skipped test");
+
+ return 0;
+ }
+
+ This test program produces the output shown above in the section on
+ TAP and demonstrates most of the functions. The other functions of
+ interest are sysdiag() (like diag() but adds strerror() results),
+ bail() and sysbail() for fatal errors, skip_block() to skip a whole
+ block of tests, and skip_all() which is called instead of plan() to
+ skip an entire test case.
+
+ The C TAP library also provides plan_lazy(), which can be called
+ instead of plan(). If plan_lazy() is called, the library will keep
+ track of how many test results are reported and will print out the
+ plan at the end of execution of the program. This should normally be
+ avoided since the test may appear to be successful even if it exits
+ prematurely, but it can make writing tests easier in some
+ circumstances.
+
+ Complete API documentation for the basic C TAP library that comes with
+ C TAP Harness is available at:
+
+ <http://www.eyrie.org/~eagle/software/c-tap-harness/>
+
+ It's common to need additional test functions and utility functions
+ for your C tests, particularly if you have to set up and tear down a
+ test environment for your test programs, and it's useful to have them
+ all in the libtap library so that you only have to link your test
+ programs with one library. Rather than editing tap/basic.c and
+ tap/basic.h to add those additional functions, add additional *.c and
+ *.h files into the tap directory with the function implementations and
+ prototypes, and then add those additional objects to the library.
+ That way, you can update tap/basic.c and tap/basic.h from subsequent
+ releases of C TAP Harness without having to merge changes with your
+ own code.
+
+ Libraries of additional useful TAP test functions are available in
+ rra-c-util at:
+
+ <http://www.eyrie.org/~eagle/software/rra-c-util/>
+
+ Some of the code there is particularly useful when testing programs
+ that require Kerberos keys.
+
+ If you implement new test functions that compare an expected and seen
+ value, it's best to name them is_<something> and take the expected
+ value, the seen value, and then a printf-style format string and
+ possible arguments to match the calling convention of the functions
+ provided by C TAP Harness.
+
+ Shell
+
+ C TAP Harness provides a library of shell functions to make it easier
+ to write TAP tests in shell. That library includes much of the same
+ functionality as the C TAP library, but takes its parameters in a
+ somewhat different order to make better use of shell features.
+
+ The libtap.sh file should be installed in a directory named tap in
+ your test suite area. It can then be loaded by tests written in shell
+ using the environment set up by runtests with:
+
+ . "$SOURCE"/tap/libtap.sh
+
+ Here is a complete test case written in shell which produces the same
+ output as the TAP sample above:
+
+ #!/bin/sh
+
+ . "$SOURCE"/tap/libtap.sh
+ cd "$BUILD"
+
+ plan 4
+ ok 'the first test' true
+ ok '' [ 42 -eq 42 ]
+ diag a diagnostic, ignored by the harness
+ ok '' false
+ skip 'a skipped test'
+
+ The shell framework doesn't provide the is_* functions, so you'll use
+ the ok function more. It takes a string describing the text and then
+ treats all of its remaining arguments as a condition, evaluated the
+ same way as the arguments to the "if" statement. If that condition
+ evaluates to true, the test passes; otherwise, the test fails.
+
+ The plan, plan_lazy, diag, and bail functions work the same as with
+ the C library. skip takes a string and skips the next test with that
+ explanation. skip_block takes a count and a string and skips that
+ many tests with that explanation. skip_all takes an optional reason
+ and skips the entire test case.
+
+ Since it's common for shell programs to want to test the output of
+ commands, there's an additional function ok_program provided by the
+ shell test library. It takes the test description string, the
+ expected exit status, the expected program output, and then treats the
+ rest of its arguments as the program to run. That program is run with
+ standard error and standard output combined, and then its exit status
+ and output are tested against the provided values.
--- /dev/null
+This directory contains a test harness and the beginnings of what is
+intended to become the primary OpenAFS test suite. The tests in this
+directory are run when "make check" is run at the top level of the OpenAFS
+tree.
+
+runtests.c is the test harness, and TESTS is the list of tests that it
+will run. If you add a new test, add it to TESTS as well. All tests must
+be executables (possibly shell scripts or Perl scripts) that end in either
+".t" or "-t", but should be listed in TESTS without that suffix.
+
+Tests should be organized into subdirectories, and where it makes sense to
+do so, those subdirectories should match the subdirectory names under src
+in the AFS source tree. In other words, tests for src/util/* should go in
+a directory named util, tests for the libkopenafs library should go in a
+directory named kopenafs, and so forth. To integrate with the build
+system, each subdirectory will need to have its own Makefile.in and be
+added to the list of generated Makefiles in configure.in at the top
+level. The Makefile.in in this directory will also need to be modified
+to recurse into any new directories. See util/Makefile.in for an example
+of how to write a Makefile.in for a new test directory.
+
+runtests.c, tap/basic.c, tap/basic.h, tap/libtap.sh, and HOWTO come from
+the C TAP Harness distribution at:
+
+ http://www.eyrie.org/~eagle/software/c-tap-harness/
+
+but feel free to propose modifications directly through OpenAFS Gerrit.
+Russ Allbery will take care of merging modifications upstream. However,
+OpenAFS-specific modifications should not be made to those files. To add
+additional OpenAFS-specific code to the TAP library, add additional *.c
+and *.h (or *.sh) files to the tap directory rather than modifying
+basic.c, basic.h, or libtap.sh.
git://git.openafs.org / openafs.git / commitdiff