Lisp-Unit-Tester — a portable unit-tester for Lisp

Lisp-Unit-Tester (aka LUT)

[Note: this document is a revised version of the original written by Chris Riesbeck.  It has been modified to reflect changes made to the lisp-unit package (including a name change to lisp-unit-tester) at the pleasure of Neil Haven. When I posted this description yesterday, 25 May 2008, I neglected to include a link to download the package, my apologies -- it is now included below where it says download here.]

lisp-unit-tester (with nickname :lut) is a Common Lisp library that supports unit testing (download here).  Unit testing is the coding practice wherein a library of functionality tests is developed along with, or even ahead of, production code.  Unit testing helps detect regressions (code changes that break old functionality), helps focus design efforts on meeting practical and well-defined goals, and helps detect bugs.

There is a long history of testing packages in Lisp, traditionally called “regression” testers, see, for example,  RT (ca 1990), CLUnit (ca 2000), Heute (ca 2007).  Unit testing packages in Lisp and other languages benefit from inspiration by eXtreme Programming practices and JUnit for Java.

This page has two parts:

• An overview with examples
• A reference section with all forms and functions

Overview

Chris Riesbeck’s main goal for the original unit-tester package was to make it simple to use.  It is well-suited to both beginning and advanced Lisp programmers. The advantages of unit-tester, and the revised version lisp-unit-tester, are:

• Written in portable Common Lisp.  The package has been run under sbcl, clisp, corman common lisp, lispworks lisp, and so on…
• Just one file to load.
• Dead-simple to define and run tests. See example.
• Supports redefining functions and even macros without reloading tests.
• Supports test-first programming.
• Supports testing return values, printed output, macro expansions, and error conditions.
• Produces short readable output with a reasonable level of detail.
• Groups tests by package for modularity.

The changes from Chris Reisbeck’s original package now present in lisp-unit-tester are the following:

• Unexpected errors in test code no longer abort tests;
• All tests are automatically numbered sequentially for ease of reference;
• Groups of tests are executed in the order defined rather than alphabetically;
• The package now works under CormanLisp;
• All exported functions are now documented;
• The user has the option to set a verbose success mode so that successful tests are not silent;
• Unexpected errors in tests are tallied and reported.

How to Use lisp-unit-tester

1. Load (or compile and load) lisp-unit-tester.lisp.
2. Evaluate (use-package :lut) or, if you prefer a lot of typing, (use-package :lisp-unit-tester) 
3. Load a file of tests. See below for how to define tests. An example file (test-lisp-unit.lisp) is provided with the distribution.
4. Run the tests with test.

Uniquely numbered test results will be printed, along with a summary of how many tests were run, how many passed, how many failed, and how many error conditions were generated.  (It is possible to customize the behavior of :lut so that only test failures will be printed; it is also possible to customize the behavior of :lut so that unexpected error conditions halt testing.)

You define a test with define-test:

(define-test name exp₁ exp₂ …)

This defines a test called name. The expressions can be anything, but typically most will be assertion forms.

Tests can be defined before the code they test, even if they’re testing macros. This is to support test-first
programming.

After defining your tests and the code they test, run the tests with

(test)

This runs every test defined in the current package. To run just a specific test, use

(test name₁)

To run multiple specific tests, use

(test (name₁ name₂ …))

To run tests in a particular package named mypackage, use

(test (name₁ name₂ …) :in-package mypackage)

e.g., (test (greater summit) :in-package mypackage).

The following example

• defines some tests to see if pick-greater returns the larger of two arguments
• defines a deliberately broken version of pick-greater
• runs the tests

First, we define some tests.

> (define-test pick-greater
    (assert-equal 5 (pick-greater 2 5))
    (assert-equal 5 (pick-greater 5 2))
    (assert-equal 10 (pick-greater 10 10))
    (assert-equal 0 (pick-greater -5 0)) )
PICK-GREATER

Following good test-first programming practice, we run these tests before writing any code.

> (test pick-greater)
PICK-GREATER:1 (PICK-GREATER 2 5) failed:
   Expected 5 but saw #<Undefined-Function #x1719AD0>; …
PICK-GREATER:2 (PICK-GREATER 5 2) failed:
  Expected 5 but saw #<Undefined-Function #x175ACC0>; …
PICK-GREATER:3 (PICK-GREATER 10 10) failed:
  Expected 10 but saw #<Undefined-Function #x177A640>; …
PICK-GREATER:4 (PICK-GREATER -5 0) failed:
  Expected 0 but saw #<Undefined-Function #x179A170>; …
PICK-GREATER: 0 assertions passed, 4 failed; including 4 execution errors.

This shows that we need to do some work. So we define our broken version of pick-greater.

> (defun pick-greater (x y) x) ;; deliberately wrong
PICK-GREATER

Now we run the tests again:

> (test pick-greater)
PICK-GREATER:1 (PICK-GREATER 2 5) failed: Expected 5 but saw 2
PICK-GREATER:2 (PICK-GREATER 5 2) succeeded: Got 5
PICK-GREATER:3 (PICK-GREATER 10 10) succeeded: Got 10
PICK-GREATER:4 (PICK-GREATER -5 0) failed: Expected 0 but saw -5
PICK-GREATER: 2 assertions passed, 2 failed; including 0 execution errors.

This shows two failures. The first test failed because (pick-greater 2 5) returned 2 when 5 was expected, and the fourth test case failed because (pick-greater -5 0) returned -5 when 0 was expected.

Assertion Forms

The most commonly used assertion form is

(assert-equal value form)

This tallies a failure if form returns a value not #’equal to value. Both value and test are evaluated in the local lexical environment. This means that you can use local variables in tests. In particular, you can write loops that run many tests at once:

> (define-test my-sqrt
    (dotimes (i 5)
     (assert-equal i (my-sqrt (* i i)))))
MY-SQRT
> (defun my-sqrt (n) (/ n 2)) ;; wrong definition!!
> (set-verbose-success nil) ;; only show failures
> (test my-sqrt)
MY-SQRT:2 (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
MY-SQRT:4 (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
MY-SQRT:5 (MY-SQRT (* I I)) failed: Expected 4 but saw 8
MY-SQRT: 2 assertions passed, 3 failed; including 0 execution errors.

Notice that the above output doesn’t tell us for which values of i the code failed. Fortunately, you can fix this by adding expressions at the end of the assert-equal. These expression and their values will be printed when the results of an assertion are announced.

> (define-test my-sqrt
    (dotimes (i 5)
     (assert-equal i (my-sqrt (* i i)) i))) ;; added i at the end
MY-SQRT
> (run-tests my-sqrt)
MY-SQRT:2 (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
   I => 1
MY-SQRT:4 (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
   I => 3
MY-SQRT:5 (MY-SQRT (* I I)) failed: Expected 4 but saw 8
   I => 4
MY-SQRT: 2 assertions passed, 3 failed; including 0 execution errors.

The next most useful assertion form is

(assert-true test)

This tallies a failure if test returns false. Again, if you need to print out extra information, just add expressions after test.

There are also assertion forms to test what code prints, what errors code returns, or what a macro expands into. A complete list of assertion forms is in the reference section.

How to Organize Tests with Packages

Tests are grouped internally by the current package, so that a set of tests can be defined for one package of code without interfering with tests for other packages.

If your code is being defined in cl-user, which is common when learning Common Lisp, but not for production-level code, then you should define your tests in cl-user as well.

If your code is being defined in its own package, you should define your tests either in that same package, or in another package for test code. The latter approach has the advantage of making sure that your tests have access to only the exported symbols of your code package.

For example, if you were defining a date package, your date.lisp file would look like this:

(defpackage :date
   (:use :common-lisp)
   (:export #:date->string #:string->date))
 (in-package :date)
(defun date->string (date) …)
(defun string->date (string) …)

Your date-tests.lisp file would look like this:

(defpackage :date-tests
  (:use :common-lisp :lisp-unit-tester :date))
(in-package :date-tests)
(define-test date->string
   (assert-true (string= ... (date->string ...)))
    ...)
...  

You could then run all your date tests in the test package:

(in-package :date-tests) (test)   

Alternately, you could run all your date tests from any package with:

(lisp-unit-tester:test :in-package :date-tests)

Reference Section

Here is a list of the functions and macros exported by lisp-unit-tester.

Functions for managing tests

(define-test name exp₁ exp₂ …)

This macro defines a test called name with the expressions specified, in the package specified by the value of *package* in effect when define-test is executed. The expresssions are assembled into runnable code whenever needed by run-tests. Hence you can define or redefine macros without reloading tests using those macros.

(remove-tests names [package])

This function removes the tests named for the given package. If no package is given, the value of *package* is used.

(remove-all-tests [package])

This function removes the tests for the given package. If no package is given, it removes all tests for the current package. If nil is given, it removes all tests for all packages.

(test [test | (test*)] [:in-package package])

Arguments: test – (not evaluated) a symbol designating a test defined with define-test
    package – the package to run the tests in, defaults to *package*
Semantics: run the named test or tests in the package.  If the tests are not provided or are
  nil, then all tests in <package> are run sequentially.
Examples: 
(test test1)                    
(test (test1 test2) :in-package :cl-user)
See Also: define-test
Returns: no values 

(use-debugger flag)

 If <flag> is nil, errors will be caught by lisp-unit-tester and the debugger will not be invoked.  If <flag> is t, lisp-unit-tester will not attempt to intercept errors.  By default, lisp-unit-tester is configured to intercept errors so that testing can proceed without the debugger.

(set-verbose-success flag)

Set <flag> to t when you want lisp-unit-tester to announce successful tests as well as failures.  Set <flag> to nil when you want lisp-unit-tester only to announce failures.  By default, lisp-unit-tester is configured to announce successes as well as failures.

(set-abort-on-conditions flag)

Set <flag> to t if you want an unexpected error signal to abort an entire test.  Set <flag> to nil if you want a test to continue after encountering an unexpected error.  By default, lisp-unit-tester is configured to continue after encountering unexpected errors.

(set-tag number)

 By default, the assertions within a particular test suite are numbered sequentially starting at 1.  You can change the origin of the running sequence by calling 'set-tag.  This can be useful if you need to have a unique identifying tag for each assertion within a test.

Forms for assertions

All of the assertion forms are macros. They tally a failure if the associated predication returns false. Assertions can be made about return values, printed output, macro expansions, and even expected errors. Assertion form arguments are evaluated in the local lexical environment.

All assertion forms allow you to include additional expressions at the end of the form. These expressions and their values will be printed only when the test fails.

Return values are unspecified for all assertion forms.

Assertions Involving Equality

(assert-eq value form [form₁ form₂ ...])
(assert-eql value form [form₁ form₂ ...])
(assert-equal value form [form₁ form₂ ...])
(assert-equalp value form [form₁ form₂ ...])
(assert-equality predicate value form [form₁ form₂ ...])

These macros tally a failure if value is not equal to the result returned by form, using the specified equality predicate.  In general, assert-equal is used for most tests.

 

Example use of assert-equality:
  (assert-equality #’set-equal ‘(a b c) (unique-atoms ‘((b c) a ((b a) c))))

Assertions Involving Boolean Value

(assert-true test [form₁form₂ ...]) 
(assert-false test [form₁form₂ ...])

assert-true tallies a failure if test returns false.
assert-false tallies a failure if test returns true.

Assertions of What Gets Printed

(assert-prints “output” form [form₁ form₂ ...])

This macro tallies a failure if form does not print to standard output stream a string equal to the given string, ignoring differences in beginning and ending newlines.

Assertions Involving Macroexpansion

(assert-expands expansion form [form₁ form₂ ...])

This macro tallies a failure if (macroexpand-1 form) does not produce a value equal to expansion.

Assertions of Error Conditions

(assert-no-error form [form₁ form₂ ...])

This macro tallies a failure if form signals an unhandled condition.

 (assert-error condition-type form [form₁ form₂ ...])

This macro tallies a failure if form does not signal an error that is equal to or a subtype of condition-type. Use error to refer to any kind of error. See condition types in
the Common Lisp Hyperspec for other possible names.

 

For example, (assert-error ‘arithmetic-error (foo 0)) would assert that
foo is supposed to signal an arithmetic error when passed zero.

Utility predicates

Several predicate functions are exported that are often useful in writing
tests with assert-equality.

(logically-equal value₁value₂)

This predicate returns true of the two values are either both true, i.e.,
non-NIL, or both false.

(set-equal list₁list₂ [:test])

This predicate returns true the first list is a subset of the second and
vice versa. :test can be used to specify an equality predicate.
The default is eql.