Peter Siebel - Practical Common Lisp

In this chapter you'll return to cutting code and develop a simple unit testing framework for Lisp. This will give you a chance to use some of the features you've learned about since Chapter 3, including macros and dynamic variables, in real code.

The main design goal of the test framework will be to make it as easy as possible to add new tests, to run various suites of tests, and to track down test failures. For now you'll focus on designing a framework you can use during interactive development.

The key feature of an automated testing framework is that the framework is responsible for telling you whether all the tests passed. You don't want to spend your time slogging through test output checking answers when the computer can do it much more quickly and accurately. Consequently, each test case must be an expression that yields a boolean value—true or false, pass or fail. For instance, if you were writing tests for the built-in + function, these might be reasonable test cases: [100] This is for illustrative purposes only—obviously, writing test cases for built-in functions such as + is a bit silly, since if such basic things aren't working, the chances the tests will be running the way you expect is pretty slim. On the other hand, most Common Lisps are implemented largely in Common Lisp, so it's not crazy to imagine writing test suites in Common Lisp to test the standard library functions.

(= (+ 1 2) 3)

(= (+ 1 2 3) 6)

(= (+ -1 -3) -4)

Functions that have side effects will be tested slightly differently—you'll have to call the function and then check for evidence of the expected side effects. [101] Side effects can include such things as signaling errors; I'll discuss Common Lisp's error handling system in Chapter 19. You may, after reading that chapter, want to think about how to incorporate tests that check whether a function does or does not signal a particular error in certain situations. But in the end, every test case has to boil down to a boolean expression, thumbs up or thumbs down.

If you were doing ad hoc testing, you could enter these expressions at the REPL and check that they return T . But you want a framework that makes it easy to organize and run these test cases whenever you want. If you want to start with the simplest thing that could possibly work, you can just write a function that evaluates the test cases and AND s the results together.

(defun test-+ ()

(and

(= (+ 1 2) 3)

(= (+ 1 2 3) 6)

(= (+ -1 -3) -4)))

Whenever you want to run this set of test cases, you can call test-+.

CL-USER> (test-+)

T

As long as it returns T , you know the test cases are passing. This way of organizing tests is also pleasantly concise—you don't have to write a bunch of test bookkeeping code. However, as you'll discover the first time a test case fails, the result reporting leaves something to be desired. When test-+returns NIL , you'll know something failed, but you'll have no idea which test case it was.

So let's try another simple—even simpleminded—approach. To find out what happens to each test case, you could write something like this:

(defun test-+ ()

(format t "~:[FAIL~;pass~] ... ~a~%" (= (+ 1 2) 3) '(= (+ 1 2) 3))

(format t "~:[FAIL~;pass~] ... ~a~%" (= (+ 1 2 3) 6) '(= (+ 1 2 3) 6))

(format t "~:[FAIL~;pass~] ... ~a~%" (= (+ -1 -3) -4) '(= (+ -1 -3) -4)))

Now each test case will be reported individually. The ~:[FAIL~;pass~]part of the FORMAT directive causes FORMAT to print "FAIL" if the first format argument is false and "pass" otherwise. [102] I'll discuss this and other FORMAT directives in more detail in Chapter 18. Then you label the result with the test expression itself. Now running test-+shows you exactly what's going on.

CL-USER> (test-+)

pass ... (= (+ 1 2) 3)

pass ... (= (+ 1 2 3) 6)

pass ... (= (+ -1 -3) -4)

NIL

This time the result reporting is more like what you want, but the code itself is pretty gross. The repeated calls to FORMAT as well as the tedious duplication of the test expression cry out to be refactored. The duplication of the test expression is particularly grating because if you mistype it, the test results will be mislabeled.

Another problem is that you don't get a single indicator whether all the test cases passed. It's easy enough, with only three test cases, to scan the output looking for "FAIL"; however, when you have hundreds of test cases, it'll be more of a hassle.

What you'd really like is a way to write test functions as streamlined as the first test-+that return a single T or NIL value but that also report on the results of individual test cases like the second version. Since the second version is close to what you want in terms of functionality, your best bet is to see if you can factor out some of the annoying duplication.

The simplest way to get rid of the repeated similar calls to FORMAT is to create a new function.

(defun report-result (result form)

(format t "~:[FAIL~;pass~] ... ~a~%" result form))

Now you can write test-+with calls to report-resultinstead of FORMAT . It's not a huge improvement, but at least now if you decide to change the way you report results, there's only one place you have to change.

(defun test-+ ()

(report-result (= (+ 1 2) 3) '(= (+ 1 2) 3))

(report-result (= (+ 1 2 3) 6) '(= (+ 1 2 3) 6))

(report-result (= (+ -1 -3) -4) '(= (+ -1 -3) -4)))

Next you need to get rid of the duplication of the test case expression, with its attendant risk of mislabeling of results. What you'd really like is to be able to treat the expression as both code (to get the result) and data (to use as the label). Whenever you want to treat code as data, that's a sure sign you need a macro. Or, to look at it another way, what you need is a way to automate writing the error-prone report-resultcalls. You'd like to be able to say something like this:

(check (= (+ 1 2) 3))

and have it mean the following:

(report-result (= (+ 1 2) 3) '(= (+ 1 2) 3))

Writing a macro to do this translation is trivial.

(defmacro check (form)

`(report-result ,form ',form))

Now you can change test-+to use check.

(defun test-+ ()

(check (= (+ 1 2) 3))

(check (= (+ 1 2 3) 6))

(check (= (+ -1 -3) -4)))

Since you're on the hunt for duplication, why not get rid of those repeated calls to check? You can define checkto take an arbitrary number of forms and wrap them each in a call to report-result.

(defmacro check (&body forms)

`(progn

,@(loop for f in forms collect `(report-result ,f ',f))))

This definition uses a common macro idiom of wrapping a PROGN around a series of forms in order to turn them into a single form. Notice also how you can use ,@to splice in the result of an expression that returns a list of expressions that are themselves generated with a backquote template.