Test for fun and profit, Part 3: The XML test framework

Do you enjoy using defective software? Of course, you don’t; no one does, and that’s why testing is so important. Testing not only helps identify the ways in which an implementation veers away from stated requirements, but also helps prevent defects that have aleady been identified and squashed from creeping back in.

For the last two months, I’ve described testing, I’ve explained its terminology, and I’ve explained how it works. This month, I move from the abstract to the concrete by presenting a simple framework for unit and component testing. If you’re joining us for the first time, you might want to read the first two articles in the series before launching into our sample code.

A simple example

For the purposes of this article, we will build and test a simple scanner. Our scanner will search a string for javadoc tags (that is, lines that begin with the @ character).

Let’s take a look at our scanner’s code:

import java.util.Vector;
public
class Scanner {
  public
  int
  scan(Vector vector, String string, int nBegin, int nLength) {
    if (vector == null)
      throw new NullPointerException("vector");
    if (string == null)
      throw new NullPointerException("string");
    if (nBegin + nLength > string.length())
      throw new StringIndexOutOfBoundsException("nBegin + nLength > string.length()");
    if (nBegin < 0)
      throw new StringIndexOutOfBoundsException("nBegin < 0");
    if (nLength < 0) nLength = string.length() - nBegin;
    vector.removeAllElements();
    int n1 = string.indexOf("/**", nBegin);
    if (n1 < nBegin)
      return -1;
    int n2 = string.indexOf("*/", n1 + 3);
    if (n2 < 0 || n2 > nBegin + nLength)
      return -1;
    StringBuffer stringbuffer = null;
    int n = n1 + 3;
    while (n < n2) {
      int nEOL = string.indexOf("rn", n);
      int nNextLine = nEOL + 2;
      if (nEOL < 0) {
        nEOL = string.indexOf('r', n);
        nNextLine = nEOL + 1;
      }
      if (nEOL < 0) {
        nEOL = string.indexOf('n', n);
        nNextLine = nEOL + 1;
      }
      if (nEOL < 0) {
        nEOL = n2;
        nNextLine = nEOL;
      }
      if (nEOL > n2) {
        nEOL = n2;
        nNextLine = nEOL;
      }
      if (nEOL > n) {
        char c = string.charAt(n);
        while ((c == ' ' || c == 't' || c == '*') && n < n2)
          c = string.charAt(++n);
        if (string.charAt(n) == '@') {
          if (stringbuffer != null) vector.addElement(stringbuffer);
          stringbuffer = new StringBuffer();
        }
        if (stringbuffer != null) stringbuffer.append(string.substring(n, nEOL));
      }
      n = nNextLine;
    }
    if (stringbuffer != null) vector.addElement(stringbuffer);
    return n2 + 2;
  }
}

The scan() method searches the supplied string for a javadoc-style comment (that is, a comment that begins with /** and ends with */). If it finds one, it searches the body of the comment for lines that begin with the @ character. These lines begin javadoc tags. Upon finding such a tag, scan() copies it, along with all subsequent lines up to the next tag or up to the end of the comment, into a collection. Once it has found all such tags, it returns both the collection and the point at which it stopped searching.

The tests

Even though I’ve given you the code first, in general it’s a good idea to take a stab at writing some tests before you begin actually programming. While it is difficult to generate a thorough and comprehensive test suite at this point, there are several good reasons for taking a first stab at it:

• It helps you refine the program’s design in your mind
• It helps you identify program requirements that are incomplete or missing

After you’ve written some code, it’s easy to return to the test suite and add additional tests. In addition to testing any new requirements you’ve discovered, these tests should test all paths through the unit under consideration (keeping in mind the discussion of black-box versus white-box testing two months ago).

Now, let’s look at the test code:

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE testsuite SYSTEM "test.dtd">
<testsuite name="Scanner Test">
  <preamble>
    <set id="scanner"><constructor class="Scanner"/></set>
  </preamble>
  <test name="Test 1">
    <preamble>
      <set id="vector"><constructor class="java.util.Vector"/></set>
    </preamble>
    <action>
      <set id="return">
        <method id="scanner" name="scan">
          <get id="vector"/>
          <primitive>
/**
 * The _Role_ interface.
 *
 * A role organizes a set of permits into a package.
 *
 * @author Todd Sundsted
 * @version 0.1
 *
 */
          </primitive>
          <primitive>0</primitive>
          <primitive>-1</primitive>
        </method>
      </set>
    </action>
    <result>
      <equal>
        <get id="return"/>
        <primitive>136</primitive>
      </equal>
      <equal>
        <method id="vector" name="size"/>
        <primitive>2</primitive>
      </equal>
      <equal>
        <method id="vector" name="elementAt"><primitive>0</primitive></method>
        <primitive>@author Todd Sundsted</primitive>
      </equal>
      <equal>
        <method id="vector" name="elementAt"><primitive>1</primitive></method>
        <primitive>@version 0.1</primitive>
      </equal>
    </result>
  </test>
</testsuite>

The tests above are written in XML. I’ll explain the syntax later; let’s first concentrate on understanding what it is we’re testing.

Tests are grouped into units called test suites. Each test has three sections. The action section contains the action or behavior being tested. It typically contains either a constructor (if the construction of the unit is what is being tested) or a method. The result section contains the conditions that determine whether or not the unit passed the test. The preamble provides a place in which to set up the objects used in the test.

The suite of tests above is by no means complete. A complete test suite can include dozens or even hundreds of individual tests. But our sample tests tests do demonstrate how to use the testing framework and how to construct tests.

Oops, we’ve made a mistake!

If you look closely at the code in the Java class above, you’ll notice a mistake. Let’s see what happens when we run the test suite.

Test Suite: Scanner Test
Test: Test 1
 ...passed
 ...passed
 ...failed
 ...failed

The output indicates which test suite ran and which test identified a defect. Defects are identified by comparing the generated output with the expected output as described in the <result> section of each test (more on which below). The failed test indicates which input caused the unit to fail. In our case, problems arose with the last two conditions of the first test.

Now that the defect has been identified, it will be easy enough to go back into the code and fix it. After doing so, we should run the tests again in order to make sure that we didn’t break anything else in the course of our repair work.

Look once more at the Java source code above. You’ll see that we’ve mistakenly added the StringBuffer instance to the output vector instead of its string representation (an easy enough mistake to make — and one that I’ve made before). The listing below identifies the offending lines and suggest replacements.

In line 58...
 replace: if (stringbuffer != null) vector.addElement(stringbuffer);
    with: if (stringbuffer != null) vector.addElement(stringbuffer.toString());
In line 65...
 replace: if (stringbuffer != null) vector.addElement(stringbuffer);
    with: if (stringbuffer != null) vector.addElement(stringbuffer.toString());

The framework

Now that you’ve seen the framework in action, I’d like to describe the framework itself.

First, the framework is XML-based. I selected XML because it seems destined to become the standard language for describing information. It is also very clean, and tools for creating it are beginning to appear.

Second, the framework doesn’t require coding. The test-description language is declarative, not procedural. I believe that this reduces the complexity of the tests and lowers the level of programming skill required to create them.

We next turn our attention to the Document Type Definition (DTD). A DTD defines a format that valid XML must follow. It specifies the complete set of acceptable tags and their relationships, as well as a number of other details. The power of XML lies in the fact that there is not a single official XML DTD (as is the case with HTML). Instead, XML DTDs can be tailored to the problem at hand — in our case, testing:

<?xml version="1.0" encoding="US-ASCII"?>
<!-- test.dtd -->
<!ELEMENT testsuite (preamble?,test+)> 
<!ATTLIST testsuite
          name CDATA #IMPLIED
          >
<!ELEMENT test (preamble?,action,result)>
<!ATTLIST test
          name CDATA #IMPLIED
          >
<!ELEMENT preamble (constructor|method|field|primitive|null|get|set)+>
<!ELEMENT action (constructor|method|field|primitive|null|get|set)>
<!ELEMENT result (equal)*>
<!ELEMENT equal ((constructor|method|field|primitive|null|get|set),
                 (constructor|method|field|primitive|null|get|set))
          >
<!ELEMENT constructor (constructor|method|field|primitive|null|get|set)*>
<!ATTLIST constructor
          class CDATA #REQUIRED
          >
<!ELEMENT method (constructor|method|field|primitive|null|get|set)*>
<!ATTLIST method
          id IDREF #REQUIRED
          name CDATA #REQUIRED
          >
<!ELEMENT field EMPTY>
<!ATTLIST field
          id IDREF #REQUIRED
          name CDATA #REQUIRED
          >
<!ELEMENT primitive (#PCDATA)>
<!ATTLIST primitive
          type (string|byte|short|int|long|float|double|boolean|char) #IMPLIED
          >
<!ELEMENT null EMPTY>
<!ELEMENT get EMPTY>
<!ATTLIST get
          id IDREF #REQUIRED
          >
<!ELEMENT set (constructor|method|field|primitive|null|get|set)>
<!ATTLIST set
          id ID #REQUIRED
          >

Below is a definition for each of the DTD tags.

<testsuite>

Groups together a set of tests into a suite.

<preamble>

Contains all the code necessary to set up a single test or an entire test suite. In it, supporting objects are created, objects are, if need be, initialized, and any necessary operations are performed.

<test>

Defines a single test. A test consists (roughly) of an action and a result; it can have a descriptive name.

<action>

Contains the operation to be tested — typically, a constructor or a method call.

<result>

Can have numerous subsections, each checking the value of a specific type of output.

<equals>

Checks the specified values.

<constructor>

Creates an instance of the specified class.

<method>

Invokes a method on a specified bound value. See <set> below.

<field>

Reads a field on a specified bound value. See <set> below.

<set>

Binds an ID to a value. Other elements can then refer to this bound value.

<get>

Accesses a bound value.

<primitive>

Creates an instance of a primitive value.

Conclusion

I’m sure by now you’d like to get your hands on some live code. Download xaf.jar, xml4j.jar, and (if you’re not using the Java 2 Platform) collections.jar. xml4j.jar contains IBM’s XML parser, collections.jar contains Java 2-compatible collection classes, and xaf.jar contains a free XML framework (written by yours truly) and the testing infrastructure. Add all of the jar files to your CLASSPATH.

You’ll also need something to test. Download <a href=" and compile it, then download test.xml and test.dtd. The former contains the test definitions in the format I described above.

Go ahead and give it a try. At the command line, type:

java Test test.xml

This will load and run the tests.

You now have an understanding of the importance of performing tests, and can use the tools necessary to do so. Remember to keep testing and testability in mind at all stages of the development process — even if you won’t be the one doing the testing. If you do, you’ll produce better software and your customers will love you for it.

Source: www.infoworld.com