Background
The cl-ansi-tests application (I am using the July 2005 version) is a set of over 20,000 individual tests of various aspects of a lisp’s conformance with the Common Lisp standard. It was written by Paul Dietz over the span of many years. Running cl-ansi-tests is a good way to discover things about a particular lisp implementation, but, as Paul Dietz says, the “tests have not been selected to reflect the importance or relative frequency of different CL features”, so the numerical results of the tests must be interpreted carefully. It is possible for a lisp implementation to fail many trivial conformance tests, but succeed at all the important ones. Likewise, it is possible for a lisp implementation to pass nearly all the tests, but to fail a few critically important ones.
Since the cl-ansi-tests suite is a moderately-large, stressful application (~180,000 lines of code contained in ~850 files), running the cl-ansi-tests suite serves as a good stability test as well as a standards conformance test. I will describe my conformance testing results separate from the stability testing results. Then I will present my (subjective) interpretation of the tests.
The Procedure
Everything necessary to run the cl-ansi-tests suite under Corman Lisp is distributed here: ansi-tests. The distribution includes source code for a modified version of the cl-ansi-tests suite (see ansi-tests/cl-ansi-tests-20050704.ccl/), some ANSI-compatibility patches for Corman Lisp (see ansi-tests/corman-patches/ and ansi-tests/compatibility.lisp), directions for running the tests (see the header for ansi-tests/run-cl-ansi-tests.lisp), and the results from running the tests (see ansi-tests/cl-ansi-tests.24052008.results).
The cl-ansi-tests suite from CLOCC will not load under Corman Lisp out of the box; modification to six files was required to get it running with Corman Lisp. You can see a list of modifications by grepping “cormanlisp” on the source code located at ansi-tests/cl-ansi-tests-20050704.ccl/. I wrote a replacement test driver for Corman Lisp called ‘run-cl-ansi-tests.lisp’; you should use this driver instead of the ‘doit.lsp’ file which is distributed with the cl-ansi-tests suite.
I ran the tests under Windows VISTA on a dual-core Pentium, using Corman Lisp version 3.02 (beta, pre-release). I made a number of changes to the Corman Lisp release. The changes are detailed in the file ‘compatibility.lisp’ from the distribution.
Usage instructions are available in the comments at the head of the ‘run-cl-ansi-tests.lisp’ file in the distribution. If you attempt to run the tests via the usual ‘doit.lsp’, you will crash.
I ran as many of the tests as I could, given my time constraints. In the end, I was able to run 17,271 tests out of a total of more than 21,000 tests.
The distribution includes a copy of the test output with a listing of all failed tests in the file ‘cl-ansi-tests.24052008.results’.
Results and Interpretation of the ANSI Conformance Tests
The result of my testing, as of 25 May 2008, using Corman Lisp version 3.02 (beta, pre-release) and a modified version of the cl-ansi-tests-20050704.orig test suite downloaded from the UBUNTU archives, was that ccl failed 4,643 tests and passed 12,628 tests. I was able to run 17,271 tests; I was unable to run approximately 4,000 tests.
The runnable tests were in the following categories:
symbols
iteration
cons
arrays
hash-tables
types-and-classes
strings
characters
pathnames
files
streams
conditions
packages
sequences
system-construction
objects
numbers
eval-and-compile
data-and-control-flow
structures
I was unable to run the following categories of tests:
printer
reader
environment
miscellaneous
I did not run those categories of tests because either the tests would not load, or, once loaded, the tests would hang or crash my system repeatedly. I ran out of time to debug them.
The largest category of test failures resulted from non-conforming error-signaling behaviors: specifically, Corman Lisp often signals an error at the correct spot, but signals the incorrect error according to the spec. For example, Corman Lisp will typically signal a ‘SIMPLE-ERROR if the wrong number of arguments is supplied to a function; the specification is for a ‘PROGRAM-ERROR.
The second largest category of tests failures was caused by deficiencies in ccl’s type system with
regard to the specification.
The next largest category of test failures was caused by improper handling of the :allow-other-keys
keyword: specifically, the spec states that any function or macro accepting any keyword must also
accept the :allow-other-keys keyword. Corman Lisp incorrectly signals an error if :allow-other-keys
is used without being explicitly declared in the function or macro lambda list.
These three categories of failures accounted for approximately half of all test failures.
Other significant categories of test failures arose from ccl’s type system, incomplete implementation
of the crufty and byzantine ‘loop’ facility, and the failure of the :default-initargs option to
defclass. You can peruse the complete listing of program output in the file
‘cl-ansi-tests.24052008.results’ from the distribution.
Users should be aware that a small number of test cases failed due to non-ANSI scoping rules for
special variables in certain unusual cases.
Based on the results of this round of testing, it appears that significant improvement could be made
very quickly to Corman Lisp’s ANSI compatibility by attending to the following (in order of sheer
numbers of tests):
1. Error signaling in accordance with CLtL2 (> 1150 cases);
2. Harmonizing ccl’s type system with the spec (> 700 cases);
3. Fixing :allow-other-keys (> 300 cases);
4. Implementing the rest of the LOOP spec (> 150 cases);
5. Fixing the :default-initargs option to defclass.
I estimate that fixing these issues would reduce the number of failed tests by an order of magnitude.
I should emphasize that the large majority of failures exhibited by ccl were rather trivial: improper error signaling, a nit-picking problem with :allow-other-keys, missing LOOP features that nobody uses anyway, etc. A few of the failures were more serious in terms of incorrect code generation or missing, but useful, language features. (For example, due to a code-generation error, #’with-package-iterator is broken.) Generally, I would say the results of the testing were quite positive: ccl is a nearly ANSI compliant implementation of lisp, none of the non-compliant behaviors are difficult to fix, none of the non-compliant behaviors are terribly important, and a little effort can yield a big improvement.
Results of Stability Testing
Unfortunately, considered as a stability test, the results of running the test suite were disappointing. I encountered context-dependent and intermittent instabilities in ccl during the course of testing. I found several cases where, separately, I could run Program A, or I could run Program B, but if I tried to run Program A followed directly by Program B, ccl would hang or crash.
I also encountered several non-reproducible problems where ccl would run a program perfectly well part of the time, but would hang or crash other times. Specifically, I cannot run the entire test suite at once: I must load and execute parts of the suite file-by-file. If I try to load the entire test suite, ccl will hang.
Summary & Caveats
Corman Lisp conforms substantially to the CLtL2 spec, although significant areas of non-conformance exist. I would estimate that the most significant areas of non-compliance in terms of numbers of pages of the CLtL2 specification violated are (in some rough order):
1. Error signaling with incorrect condition types
2. The type system mishandles many objects
3. The LOOP specification is not completely implemented
I would estimate that the most significant areas of non-conformance in practical terms of creating bugs and gotchas in code you are writing or trying to port are:
1. The :default-initargs option to defclass does not work
2. The :test-not option to several functions (member, adjoin, etc.) does not work
3. No support for synonym streams or broadcast streams
4. Some FORMAT directives (notably ~<~>) are broken
Subjectively, the most important current issue standing in the way of Corman Lisp’s suitability for product-deployment is not *any* of the ANSI conformance issues, which are relatively minor; the most important current issue is the stability of Corman Lisp measured by the frequency of opaque crashes and system hangs (hang = processor runs away and never comes back). The stability issues must be fixed before Corman Lisp can be considered production-ready.
The strengths of Corman Lisp are its source-code (which is both readily-available and cleanly-written), its compatibility with Microsoft’s C/C++ compiler (which is the compiler of choice for a vast number of commercial applications), and its intuitive and versatile FFI (compare, for instance, how difficult it is to call Lisp from C in SBCL vs. how easy it is in Corman Lisp and you will see what I mean).
Based on my experience over the past few months with Corman Lisp, including writing approximately 5000 lines of cormanlisp lisp, I would say the support I’ve received from Roger has been outstanding and the lisp implementation is generally reliable and fast. If the stability issues with the Corman Lisp implementation can be fixed, I would not hesitate to recommend Corman Lisp for anyone who needs a tightly-integrated lisp/C/C++ solution at a quite reasonable cost.
May 2008
RSS 