5 This is a guide for users of the C TAP Harness package or similar
6 TAP-based test harnesses explaining how to write tests. If your
7 package uses C TAP Harness as the test suite driver, you may want to
8 copy this document to an appropriate file name in your test suite as
9 documentation for contributors.
13 TAP is the Test Anything Protocol, a protocol for communication
14 between test cases and a test harness. This is the protocol used by
15 Perl for its internal test suite and for nearly all Perl modules,
16 since it's the format used by the build tools for Perl modules to run
17 tests and report their results.
19 A TAP-based test suite works with a somewhat different set of
20 assumptions than an xUnit test suite. In TAP, each test case is a
21 separate program. That program, when run, must produce output in the
27 # a diagnostic, ignored by the harness
28 not ok 3 - a failing test
29 ok 4 # skip a skipped test
31 The output should all go to standard output. The first line specifies
32 the number of tests to be run, and then each test produces output that
33 looks like either "ok <n>" or "not ok <n>" depending on whether the
34 test succeeded or failed. Additional information about the test can
35 be provided after the "ok <n>" or "not ok <n>", but is optional.
36 Additional diagnostics and information can be provided in lines
39 Processing directives are supported after the "ok <n>" or "not ok <n>"
40 and start with a "#". The main one of interest is "# skip" which says
41 that the test was skipped rather than successful and optionally gives
42 the reason. Also supported is "# todo", which normally annotates a
43 failing test and indicates that test is expected to fail, optionally
44 providing a reason for why.
46 There are three more special cases. First, the initial line stating
47 the number of tests to run, called the plan, may appear at the end of
48 the output instead of the beginning. This can be useful if the number
49 of tests to run is not known in advance. Second, a plan in the form:
51 1..0 # skip entire test case skipped
53 can be given instead, which indicates that this entire test case has
54 been skipped (generally because it depends on facilities or optional
55 configuration which is not present). Finally, if the test case
56 encounters a fatal error, it should print the text:
60 on standard output, optionally followed by an error message, and then
61 exit. This tells the harness that the test aborted unexpectedly.
63 The exit status of a successful test case should always be 0. The
64 harness will report the test as "dubious" if all the tests appeared to
65 succeed but it exited with a non-zero status.
71 One of the special features of C TAP Harness is the environment that
72 it sets up for your test cases. If your test program is called under
73 the runtests driver, the environment variables SOURCE and BUILD will
74 be set to the top of the source tree and the top of the build tree,
75 respectively. You can use those environment variables to locate
76 additional test data, programs and libraries built as part of your
77 software build, and other supporting information needed by tests.
81 Since TAP is the native test framework for Perl, writing TAP tests in
82 Perl is very easy and extremely well-supported. If you've never
83 written tests in Perl before, start by reading the documentation for
84 Test::Tutorial and Test::Simple, which walks you through the basics,
85 including the TAP output syntax. Then, the best Perl module to use
86 for serious testing is Test::More, which provides a lot of additional
87 functions over Test::Simple including support for skipping tests,
88 bailing out, and not planning tests in advance. See the documentation
89 of Test::More for all the details and lots of examples.
91 C TAP Harness can run Perl test scripts directly and interpret the
92 results correctly, and similarly the Perl Test::Harness module can run
93 TAP tests written in other languages using, for example, the TAP
94 library that comes with C TAP Harness. However, the "prove" tool that
95 comes with Perl and runs tests makes some Perl-specific assumptions
96 that aren't always appropriate for packages that aren't written in
101 C TAP Harness provides a basic TAP library that takes away most of the
102 pain of writing TAP test cases in C. A C test case should start with
103 a call to plan(), passing in the number of tests to run. Then, each
104 test should use is_int(), is_string(), is_double(), or is_hex() as
105 appropriate to compare expected and seen values, or ok() to do a
106 simpler boolean test. The is_*() functions take expected and seen
107 values and then a printf-style format string explaining the test
108 (which may be NULL). ok() takes a boolean and then the printf-style
111 Here's a complete example test program that uses the C TAP library:
113 #include <tap/basic.h>
120 ok(1, "the first test");
121 is_int(42, 42, NULL);
122 diag("a diagnostic, ignored by the harness");
123 ok(0, "a failing test");
124 skip("a skipped test");
129 This test program produces the output shown above in the section on
130 TAP and demonstrates most of the functions. The other functions of
131 interest are sysdiag() (like diag() but adds strerror() results),
132 bail() and sysbail() for fatal errors, skip_block() to skip a whole
133 block of tests, and skip_all() which is called instead of plan() to
134 skip an entire test case.
136 The C TAP library also provides plan_lazy(), which can be called
137 instead of plan(). If plan_lazy() is called, the library will keep
138 track of how many test results are reported and will print out the
139 plan at the end of execution of the program. This should normally be
140 avoided since the test may appear to be successful even if it exits
141 prematurely, but it can make writing tests easier in some
144 Complete API documentation for the basic C TAP library that comes with
145 C TAP Harness is available at:
147 <http://www.eyrie.org/~eagle/software/c-tap-harness/>
149 It's common to need additional test functions and utility functions
150 for your C tests, particularly if you have to set up and tear down a
151 test environment for your test programs, and it's useful to have them
152 all in the libtap library so that you only have to link your test
153 programs with one library. Rather than editing tap/basic.c and
154 tap/basic.h to add those additional functions, add additional *.c and
155 *.h files into the tap directory with the function implementations and
156 prototypes, and then add those additional objects to the library.
157 That way, you can update tap/basic.c and tap/basic.h from subsequent
158 releases of C TAP Harness without having to merge changes with your
161 Libraries of additional useful TAP test functions are available in
164 <http://www.eyrie.org/~eagle/software/rra-c-util/>
166 Some of the code there is particularly useful when testing programs
167 that require Kerberos keys.
169 If you implement new test functions that compare an expected and seen
170 value, it's best to name them is_<something> and take the expected
171 value, the seen value, and then a printf-style format string and
172 possible arguments to match the calling convention of the functions
173 provided by C TAP Harness.
177 C TAP Harness provides a library of shell functions to make it easier
178 to write TAP tests in shell. That library includes much of the same
179 functionality as the C TAP library, but takes its parameters in a
180 somewhat different order to make better use of shell features.
182 The libtap.sh file should be installed in a directory named tap in
183 your test suite area. It can then be loaded by tests written in shell
184 using the environment set up by runtests with:
186 . "$SOURCE"/tap/libtap.sh
188 Here is a complete test case written in shell which produces the same
189 output as the TAP sample above:
193 . "$SOURCE"/tap/libtap.sh
197 ok 'the first test' true
199 diag a diagnostic, ignored by the harness
201 skip 'a skipped test'
203 The shell framework doesn't provide the is_* functions, so you'll use
204 the ok function more. It takes a string describing the text and then
205 treats all of its remaining arguments as a condition, evaluated the
206 same way as the arguments to the "if" statement. If that condition
207 evaluates to true, the test passes; otherwise, the test fails.
209 The plan, plan_lazy, diag, and bail functions work the same as with
210 the C library. skip takes a string and skips the next test with that
211 explanation. skip_block takes a count and a string and skips that
212 many tests with that explanation. skip_all takes an optional reason
213 and skips the entire test case.
215 Since it's common for shell programs to want to test the output of
216 commands, there's an additional function ok_program provided by the
217 shell test library. It takes the test description string, the
218 expected exit status, the expected program output, and then treats the
219 rest of its arguments as the program to run. That program is run with
220 standard error and standard output combined, and then its exit status
221 and output are tested against the provided values.