Test suite - A progress report

Just a quick progress report on the R3 test suite...

The new test suite is running, as a first draft design. It will be published soon (maybe today?) for the R3-Alpha test group (a private test group). In addition, I will be providing documentation to explain how it works, and how users can extend the tests simply by modifying the test suite data files. Once the R3-Alpha group approves the tests, they will be released more openly.

The test engine is built mainly on the reflective test approach, where thousands of test vectors (singular test expressions) are generated and then tested, driven by small test specification files -- each typically only a page in length.

As a result, the tests require validation through inspection of their log files. Once inspected, a hash checksum tracks if changes have occurred, in either the results (test failures) or from the introduction of new tests.

The method seems sound, but I think there could be a problem with the quantity of tests generated. Currently, with only five (of 56) datatypes tested, there are more than 22'000 test vectors, and they build exponentially. We may end up with more than a million test vectors for the final suite.

Although the hash approach works well for detecting failures and other variations, one must compare the test log files to determine specific differences. And, I am a bit concerned that the comparison required may be beyond the capabilities of most "diff" (change differencing) programs.

This will require some thought. Since the log output is a set of vectors itself, it may be possible to develop a specific diff method that works well for these larger data sets. (If not, then we may need to borrow La Sorbonne's supercomputer to analyze our results each time.)

Once the tests are released, let me know your opinion in this area.

1 Comments