README -- Tcl test suite design document.

RCS: @(#) $Id: README,v 1.1.1.1 2007/07/10 15:04:24 duncan Exp $

Contents:
---------

    1. Introduction
    2. Running tests
    3. Adding tests
    4. Incompatibilities with prior Tcl versions

1. Introduction:
----------------

This directory contains a set of validation tests for the Tcl commands
and C Library procedures for Tcl.  Each of the files whose name ends
in ".test" is intended to fully exercise the functions in the C source
file that corresponds to the file prefix.  The C functions and/or Tcl
commands tested by a given file are listed in the first line of the
file.

2. Running tests:
-----------------

We recommend that you use the "test" target of Tcl's Makefile to run
the test suite.  From the directory in which you build Tcl, simply
type "make test".  This will create a special executable named
tcltest in which the testing scripts will be evaluated.  To create
the tcltest executable without running the test suite, simple type
"make tcltest".

All the configuration options of the tcltest package are available
during a "make test" by defining the TESTFLAGS environment variable.
For example,if you wish to run only those tests in the file append.test,
you can type:

	make test TESTFLAGS="-file append.test"

For interactive testing, the Tcl Makefile provides the "runtest" target.
Type "make runtest" in your build directory, and the tcltest executable
will be created, if necessary, then it will run interactively.  At the
command prompt, you may type any Tcl commands.  If you type
"source ../tests/all.tcl", the test suite will run.  You may use the
tcltest::configure command to configure the test suite run as an
alternative to command line options via TESTFLAGS.  You might also
wish to use the tcltest::testConstraint command to select the constraints
that govern which tests are run.  See the documentation for the tcltest
package for details.

3. Adding tests:
----------------

Please see the tcltest man page for more information regarding how to
write and run tests.

Please note that the all.tcl file will source your new test file if
the filename matches the tests/*.test pattern (as it should).  The
names of test files that contain regression (or glass-box) tests
should correspond to the Tcl or C code file that they are testing.
For example, the test file for the C file "tclCmdAH.c" is
"cmdAH.test".  Test files that contain black-box tests may not
correspond to any Tcl or C code file so they should match the pattern
"*_bb.test". 

Be sure your new test file can be run from any working directory.

Be sure no temporary files are left behind by your test file.
Use [tcltest::makeFile], [tcltest::removeFile], and [tcltest::cleanupTests]
properly to be sure of this.

Be sure your tests can run cross-platform in both a build environment
as well as an installation environment.  If your test file contains
tests that should not be run in one or more of those cases, please use
the constraints mechanism to skip those tests.

4. Incompatibilities of package tcltest 2.1 with 
   testing machinery of very old versions of Tcl:
------------------------------------------------

1) Global variables such as VERBOSE, TESTS, and testConfig of the
   old machinery correspond to the [configure -verbose], 
   [configure -match], and [testConstraint] commands of tcltest 2.1,
   respectively.

2) VERBOSE values were longer numeric.  [configure -verbose] values
   are lists of keywords.

3) When you run "make test", the working dir for the test suite is now
   the one from which you called "make test", rather than the "tests"
   directory.  This change allows for both unix and windows test
   suites to be run simultaneously without interference with each
   other or with existing files.  All tests must now run independently
   of their working directory.

4) The "all" file is now called "all.tcl"

5) The "defs" and "defs.tcl" files no longer exist.

6) Instead of creating a doAllTests file in the tests directory, to
   run all nonPortable tests, just use the "-constraints nonPortable"
   command line flag.  If you are running interactively, you can run
   [tcltest::testConstraint nonPortable 1] (after loading the tcltest
   package).

7) Direct evaluation of the *.test files by the "source" command is no
   longer recommended.  Instead, "source all.tcl" and use the "-file" and
   "-notfile" options of tcltest::configure to control which *.test files
   are evaluated.