Doclet your servlet!

So you’ve written that killer servlet, and now it’s been discovered. Congratulations!

But alas, your inbox is flooded with messages: What do the parameters mean? What beans does the servlet put in the session, and how can I access those beans from a JavaServer Page (JSP)? What URL variants can I use to invoke the servlet? And how can I tell which JSP is going to be called to render the response? Too bad you didn’t write any documentation because now, rather than sitting back and reaping the rewards of your accomplishment, you’re getting carpal tunnel syndrome dealing with that email flood. Doclets to the rescue!

How can doclets help? With great wisdom, the Java team understood that developers don’t like to produce documentation because writing documentation takes them away from coding. The JDK team made complete documentation much easier to produce and maintain by integrating documentation production with the routine production of commented code. They produced an elegant documentation engine, JavaDoc, which scans code for the comments that developers have always written at the beginning of each class and method. Upon completing the scan, JavaDoc outputs a set of HTML pages that present the information derived from the scanned code and comments. Further, JavaDoc looks for a special set of tags within the comments and uses them to structure the resulting documentation into important categories, such as method parameters or hyperlinks to a related method.

With the onset of Java 1.2, JavaDoc became an extensible tool. Developers could produce alternative forms of documentation by writing a Doclet — a class that takes as input the documentation-related information that JavaDoc can extract from a set of files, and writes documentation files in a format specified by the developer. Most example doclets stay pretty close to the original JavaDoc mission. For example, Sun provides a doclet that can write code documentation in RTF (rich text format, which most word processors can read) instead of HTML format. Some more adventuresome doclets actually generate new Java code files; for example, the Swing team internally uses a special doclet to automatically create BeanInfo classes from Bean classes.

In this article, I introduce another kind of doclet extension; producing documentation intended to be read by HTML or JSP designers instead of programmers. In particular, the ServletDoclet class described below produces HTML files that organize the information that any Webpage author would want to know about a servlet. How do I invoke the servlet from a link or a form? What parameters can I use, and what do they do? What session beans change state as a result of calling that servlet? What JSPs are associated with the servlet and render its output? You cannot find that information in standard JavaDoc documentation for a servlet class, and Webpage authors shouldn’t have to scan your source code to deduce what parameters it takes.

Using the techniques described below, you can provide solid documentation to your Webpage authors with just a simple extension to your normal coding practice. By adding a specific set of tags to your servlet’s JavaDoc comments, and running the ServletDoclet, you can produce comprehensive documentation for your servlet at the touch of a key.

How doclets work: A quick overview

Using a doclet with JavaDoc requires only a small change in the command line. In the normal JavaDoc command line, you indicate the files or packages you wish to document, a path upon which to find other related source files (the -sourcepath), and an output directory. To use a doclet, you simply add -doclet and the fully qualified class name. For example, the ServletDoclet can be invoked on the BookmarkServlet.java file with this command:

javadoc -doclet javaworld.ServletDoclet -d docs -sourcepath src
     src/javaworld/BookmarkServlet.java

In addition, for documenting servlets, you should have the Servlet APIs available to JavaDoc for parsing (otherwise, you will get warnings from JavaDoc). You can download the source code for the javax.servlet and javax.servlet.http packages from Apache’s Jakarta project, and then put the source into the -sourcepath indicated on the command line.

JavaDoc execution

First, regardless of which doclet you use, JavaDoc scans all the targeted files or packages and produces an internal representation of the classes and comments contained therein. Each class is represented by an instance of ClassDoc, which contains information about the class. Every method is represented by an instance of MethodDoc as well. When JavaDoc completes its scan, each ClassDoc and MethodDoc contains information representing the comments and tags found relating to the class or method. That bundle of ClassDoc instances is collected in a RootDoc, along with information about any additional, custom command-line arguments.

Second, JavaDoc loads the requested doclet and calls its start() method, passing in the RootDoc (which contains all the ClassDoc and MethodDoc instances, as well as the command-line options). The doclet then writes out documentation using whatever techniques it likes, extracting information from the RootDoc, ClassDoc, and MethodDoc instances along the way. Generally speaking, that involves creating PrintStream instances that write to disk, and writing out formatted content information to them based on the ClassDoc and MethodDoc instances.

The ServletDoclet approach

This article follows my prior article about using reflection with servlets “Untangle Your Servlet Code with Reflection”). In that article, I described a technique for using reflection to map URLs onto methods within the servlet that execute a particular behavior of that servlet. Those methods are called controllers, following the model-view-controller (MVC) convention. ServletDoclet detects controller methods and their URLs, and produces documentation for each.

Before diving into code, it’s worthwhile to sketch the ServletDoclet approach at a high level. First, ServletDoclet specifies an additional set of tags, which you can use in ServletDoclet comments. The following table lists those tags and what they mean.

Second, ServletDoclet takes the ClassDoc passed to it by JavaDoc and figures out which classes correspond to servlets. It then creates an instance of ServletDoc for each servlet, with instances of ControllerDoc for each controller within the servlet. (Note that the ServletDoclet class is the overall doclet, whereas the ServletDoc instance is used to represent an individual servlet’s documentation.)

Finally, ServletDoclet creates an instance of a formatting class called HtmlDocletWriter, and passes an array of ServletDoc instances into it. The HtmlDocletWriter instance then writes out the actual documentation in HTML format.

The code presented here is based on an earlier version produced collaboratively with Wei Cao, who was an intern at SRI International, my current employer, in the summer of 2000. You can download the complete source code from Resources.

Below is an example of using the tags to comment a particular controller method within a servlet class:

/** adds a url to the bookmark list
   @requestParameter url the url to add
   @requestParameter name the name to associate with the added url
   @requestBean String message will say "added 1 bookmark" 
   @requestBean String mode will be "view" when called by this URL
   @requestBean BookmarkList list contains the current bookmarks 
   @jsp /javaworld/bookmarks.jsp displays the bookmark list
*/
public void add(HttpServletRequest request, 
                 HttpServletResponse response) {
} 

The resulting documentation looks like this:

<insert ServletDocletSnippet.html here, perhaps as an image from a browser ??>

For a full example of a commented servlet, see BookmarkList.java. For a full example of a resulting HTML page, see BookmarkListInfo.html.

Getting into the code

The entry point from JavaDoc into any doclet is the static start() method. In the case of ServletDoclet, the start() method reads options from the command line (to be discussed in more detail later), constructs ServletDoc instances for each servlet ClassDoc, and then writes an HTML document for each ServletDoc. Returning “true” indicates success.

public static boolean start(RootDoc root){
    readOptions(root.options());
    ServletDoc[] docs = constructServletDocs(root.classes());
    for(int i = 0; i < docs.length; ++i) {
        writeDoc(docs[i]);
    }
    return true;
}    

An array of ServletDoc is constructed by iterating through the given array of classes and determining which are servlets. The isServlet method (not shown) works recursively through the superclasses of the given class until it finds either the Servlet class or a class with no superclass. In the former case, that class extends Servlet, so the method returns true. In the latter case, it returns false.

static ServletDoc[] constructServletDocs(ClassDoc[] classes) {
    Vector vector = new Vector();
    for (int i=0; i < classes.length; i++) {
        if (isServlet(classes[i]) ){
             vector.add(new ServletDoc(classes[i]));                           
        }
    }
    ServletDoc[] array = new ServletDoc[vector.size()];
    vector.copyInto(array);
    return array;
}

Representing a servlet with ServletDoc

To create reasonable documentation, you only need to know a few things about the servlet itself. Of course, it’s good to know the servlet’s name and description. The name is taken from the class name, and the description from the JavaDoc comment for the class. The ServletDoc class maintains a member variable referencing the ClassDoc from which it is derived, and can easily get that information from the ClassDoc‘s name and commentText() methods. In addition, you will want to know the baseURL that is used to invoke that servlet on the Web server. Since the Web server might have varying domain names, you report this as a relative URL, starting with a “/”.

ServletDoc discovers its baseURL from the @baseURL tag in the class comment. The Doclet API makes it easy to extract that information, given the ClassDoc. Calling the tag() method with the name of the desired tag (without the “@” prefix) results in an array of zero or more tags. Each tag’s text() method returns the text associated with the tag. Note that the full baseURL() method, below, checks for a missing tag and returns the baseURL “/servletName” in that case.

public String baseURL() {
    if (mBaseURL == null) {
        Tag[] tag = mClassDoc.tags(ServletDoclet.sBaseURLTag);
        mBaseURL = tag.length == 0 ? "/" + name() : tag[0].text();
    }
    return mBaseURL;
}

The final useful piece of information about a servlet is its list of controller methods; a controller method is a method that takes a servlet request and a servlet response as its parameters. ServletDoc‘s initialize() method constructs a Vector of ControllerDoc instances to represent each controller method it finds. It does so by iterating through the methods and making a new ControllerDoc for each method it finds with the right parameters, as shown below.

public Enumeration controllers() {
    return mControllers.elements();
}
    
private void initialize() {
    mControllers = constructControllers();
}
    
private Vector constructControllers() {
    Vector results = new Vector();
    MethodDoc[] methods = mClassDoc.methods();
    for(int i = 0; i < methods.length;++i) {
        if (isControllerMethod(methods[i] )) {
            results.addElement(new ControllerDoc(methods[i], baseURL()));
        }
    }
    return results;
}
private boolean isControllerMethod(MethodDoc methoddoc) {
    Parameter[] params = methoddoc.parameters();
    return params.length == 2 
                && params[0].type().typeName().equals("HttpServletRequest") 
                && params[1].type().typeName().equals("HttpServletResponse");
}

A ControllerDoc wraps a particular MethodDoc, providing easy access to the information useful for servlet documentation. Basic useful information includes the name, description, and URL for the controller method, extracted in similar fashion as the related ServletDoc() methods. In addition, ControllerDoc provides methods for getting an array of tags for requestParameters, requestBeans, sessionBeans, and jsps — which work in a similar fashion to the baseURL() method described above, and thus will not be further explained here; the ControllerDoc.java file provides full details.

Now you have seen how information relevant to servlet documentation is extracted from the ClassDoc. The next step is to write that information into an HTML file.

Writing the HTML documentation

The first step in writing HTML documentation is to create a PrintStream that writes to a file. In this case, the filename is derived from the servlet name, with the suffix “Info.html” added by default. If the user designated a folder on the command line, the file is created in that folder; otherwise, it is created in the working directory. The code that creates the PrintStream appears below:

static PrintWriter makePrintStream(ServletDoc doc) 
                           throws IOException {
    return new PrintWriter(
                      new FileOutputStream(makeFile(doc)));
}
     
static File makeFile(ServletDoc doc) {
    String filename = doc.name() + sFilenameSuffix;
    if (sFolder != null && sFolder.exists() && sFolder.isDirectory()) {
        return new File(sFolder,filename); 
    }
         
    if (sFolder != null) {
        warn("warning: folder does not exist " + sFolder);
    }
         
    return new File(filename); 
}

For each servlet, ServletDoclet creates a PrintStream and then passes the ServletDoc and the PrintStream to a HtmlDocletWriter, which actually writes out the content. By separating the servlet documentation data from the servlet documentation formatting, you make it easier to change the formatting. As described later, that can be done via a command-line option that specifies an alternative class to HtmlDocletWriter.

The write() method of HtmlDocletWriter goes through all the steps of creating a well-formed HTML document:

public void write(ServletDoc doc, PrintWriter out) {
    writeHtmlHeader(doc, out);
    writeHtmlBodyHeader(doc,out);
    writeOverview(doc, out);
    
    
    Enumeration controllers = doc.controllers();
    while (controllers.hasMoreElements()) {
        writeController((ControllerDoc)controllers.nextElement(),
                         doc, 
                         out);
    }
   
    writeHtmlBodyFooter(out);
    writeHtmlFooter(out);             
}

Of this code, the writeHtmlHeader(), writeHtmlBodyHeader(), writeHtmlBodyFooter(), and writeHtmlFooter() methods are pretty obvious. They consist of statements such as:

    out.println("<html>");

The interesting methods write the servlet overview and each controller. Since the overview is simpler, let’s start with that method. The writeOverview() method begins by writing a level-one header containing the servlet name (from the ServletDoc‘s name() method), which will tell the user what servlet the documentation is for. Then the writeOverview() method creates a paragraph containing the description of the servlet. As you can see, writing HTML documentation is pretty easy once you have the right information in a Doc class.

protected void writeOverview(ServletDoc doc, PrintWriter out) {
    out.println("<h1>" + doc.name() + " Documentation</h1>");
    out.println("
");
    out.println(doc.description());
    out.println("
"); 
}

Writing the documentation for a controller is more involved but not really any more difficult. The writeController() method lists the steps:

protected void writeController( ControllerDoc cdoc, 
                                ServletDoc doc, 
                                PrintWriter out) {
    writeControllerOverview(cdoc, doc, out);
    writeParameters(cdoc, doc, out);
    writeBeans(cdoc, doc, out);
    writeJsps(cdoc, doc, out);   
}

In the overview section, you write out the controller’s URL and its description, followed by samples of how to use it in a hyperlink or form. You can directly copy those usage samples into an HTML or JSP Webpage, and edit them to specify the parameters. To write the usage sample, you must know the URL that invokes the servlet as well as the parameters that can be used with it. The ControllerDoc provides that information in its URL and requestParameter() methods, respectively. In the case of a hyperlink, the code below iterates through the parameters and builds up a URL, adding a &param=value for each parameter. In the case of a form, the code iterates through the parameters and adds an <input> for each parameter, with the type of input and value to be specified by the user. Note that because you are outputting sample HTML code that you want to appear literally in the Webpage, you must use “<” and “>” instead of “”. Likewise, the funny construction “&” will appear in the Webpage as “&”.

protected void writeControllerOverview(ControllerDoc cdoc, 
                                       ServletDoc doc, 
                                       PrintWriter out) {
    String url = cdoc.url();
    Tag[] paramTag = cdoc.requestParameters();
    out.println("<hr>");
    out.println("<h2>URL: " + url + "</h2>");
    out.println("
" + cdoc.description() + "
");
    out.println("<h3>Sample Usage</h3>
");
    out.print("<a href="");
    out.print(url);
    if (paramTag.length > 0) {
        out.print("?");
        for(int i = 0; i < paramTag.length; ++i) {
            if (i > 0) out.print("&");
                out.print(wordFromString(paramTag[i].text(),1));
                out.print("=");
                out.print("value");
                out.print(i + 1);
        }
    }
    out.println("">link</a>");
    out.println("<br><br>");
    out.println("<form method="post" action="" + 
                    url + ""><br>");
    for(int i = 0; i < paramTag.length; ++i) {
        out.println("<input name="" + 
            wordFromString(paramTag[i].text(),1) +
                "" type="" value=""><br>");
      }
    out.println("<input type="submit" name="submit" " +
                  value="submit"><br>");
    out.println("</form>");
    out.println("
"); 
}

The wordFromString() method here extracts a particular word (“1” is the first word) from the tag. Recall that the first word of a requestParameter tag is the name of the parameter and that the remaining words are comment. The remainderOfString() method could be used to retrieve the comment without the parameter name (although that information is not needed here). Those two methods are available in the source code and use a StringTokenizer to do their work.

Now I’ll skip to the writeBeans() method, which writes out the information about the JavaBeans made available to the JSP as either request attributes or session beans. For each bean, HtmlDocletWriter writes out a jsp:useBean tag that illustrates its use in a JSP. It also writes out a comment describing the bean. Since the handling of request and session beans are mostly the same, most of the work is delegated to the writeBeanDetail() method with the scope of the bean (request or session) as a parameter. Again the wordFromString() method is used to extract the two first parts of the comment for a bean, the bean’s type and name. The remainderOfString() method is used to extract the rest of the comment, which is, in fact, the proper comment.

protected void writeBeans(ControllerDoc cdoc, 
    ServletDoc doc, PrintWriter out) {
    out.println("<h3>Beans</h3>");
    Tag[] requestBean = cdoc.requestBeans();
    for(int i = 0; i < requestBean.length; ++i) {
            writeBeanDetail(requestBean[i],"request",doc, out);
    }
    
    Tag[] sessionBean = cdoc.sessionBeans();
    for(int i = 0; i < sessionBean.length; ++i) {
        writeBeanDetail(sessionBean[i],"session", doc, out);
    }
}
protected void writeBeanDetail( Tag beanTag, 
                                String scope, 
                                ServletDoc doc, 
                                PrintWriter out) {
    String   text = beanTag.text(),
             beanType = wordFromString(text,1),
             beanName = wordFromString(text,2),
             description = remainderOfString(text,2),
             link = getBeanLink(beanType, doc);
    out.println("
");
    writeUseBean(beanName, beanType, scope, link, out);
    out.println("<br>");
    out.println(description);
    out.println("
");
}
protected void writeUseBean(    String name, 
                                String type, 
                                String scope, 
                                String link, 
                                PrintWriter out) {
    out.println("<code>&quot;);
    out.print(&quot;&lt;jsp:useBean id=&quot;&quot; + name + 
                &quot;&quot; scope=&quot;&quot; + scope +&quot;&quot; class=&quot;&quot;);
    if (link != null) {
         out.print(&quot;&lt;a href=&quot;&quot; + link + &quot;&quot;&gt;&quot;);
         out.print(type);
         out.print(&quot;&lt;/a&gt;&quot;);
    }
    else {
         out.print(type);
    }
    out.print(&quot;&quot;/&gt;&quot;);
    out.println(&quot;</code>");
}

I will not go into detail on the other methods that write out parts of the documentation for a controller; they are very similar to the above methods.

Linking to external JavaDoc

The writeUseBean() method accomplishes a neat trick, which deserves further discussion. The trick is constructing a hyperlink in the documentation from the class of a bean to the regular JavaDoc documentation for that class. That is useful because when you have a jsp:useBean tag in a JSP, you typically want to write additional tags that access or set that bean’s properties. You could, with much effort, extend your doclet to also document such beans. Fortunately, there is an easier way: You can link from your doclet-based documentation to standard JavaDoc documentation.

JavaDoc makes a provision for such linking in its -link option. That option is followed by a URL to the root of an external documentation directory. In that directory, there is (by convention) a file called package-list, with one package name per line. The package-list lists the packages that are documented within the external directory. Using the package-list, you can determine whether a particular class is documented in an external directory and, thus, whether a link is possible. Since multiple -link options can be provided, each for a different external documentation directory, the package-list helps you find the right baseURL for a given package and class.

Unfortunately, JavaDoc does not process the -link option for a doclet; you have to do all the work yourself. That work begins in the readOptions() method, which reads and acts upon command-line options. ServletDoclet supports three options: a default directory into which you can write the documentation, a class to be used as the writer (in place of HtmlDocletWriter), and the link option. The readOptions() method is passed a two-dimensional array of strings. The first dimension has one array per command-line option. The second dimension has all the words that were part of a given option, starting with the name of the option. Hence, the method below iterates through the options, checking to see if the first word of the option is a known option type. If it is a known option type, the method dispatches on the option type, using the second word as a parameter to the option. In the case of -link, the second word is the URL, and you call the addLink() method to store it. The addLink() method makes an instance of ExternalLink with the given URL and stores it in a Vector link. I will discuss ExternalLink next.

private static void readOptions(String[][] options) {
    for (int i = 0; i < options.length; i++) {
        String[] opt = options[i];
        if (opt[0].equals("-d")) {
            sFolder = new File(opt[1]);
        }
        else if (opt[0].equals("-link")) {
            addLink(opt[1]);
        }
        else if (opt[0].equals("-writer")) {
            sDocletWriterClass = opt[1];
        }
    }
}
static void addLink(String link) {
    try {
        sExternalLinks.addElement(
                  new ExternalLink(new URL(link)));
  
    }
    catch (java.net.MalformedURLException ex) {
 
    }
 }

The ExternalLink class provides two essential methods. First, its containsPackage() method determines if that link contains a particular package. Second, if containsPackage() returns true, makeExternalDocURL() can be called with a package and class to create a URL that links to the documentation for that class. Upon initialization of ExternalLink, you read the list of packages from the package-list file by constructing a URL to the package-list, opening a stream to it, and reading each line with a BufferedReader. You store the list of packages in a Vector, which has the satisfying consequence of making the containsPackage() method implementation a one-liner.

public ExternalLink(URL url) {
    mBaseURL = url;
    mPackageList = new Vector();
    initialize();
}
private void initialize() {
    // try/catch/finally omitted for readability
    BufferedReader reader = null;
    InputStream stream = null;
    
    URL packageListURL = new URL(mBaseURL,"package-list");
    stream = packageListURL.openStream();
    reader = new BufferedReader(new InputStreamReader(stream));
    for(    String thePackage = reader.readLine(); 
           thePackage != null; 
           thePackage = reader.readLine()) {
        mPackageList.addElement(thePackage);
    }
}
public boolean containsPackage(String packageName) {
    return mPackageList.contains(packageName); 
}

Constructing a URL that points to external documentation requires extending the base URL with a path consisting of all the directories in the class’s package name, followed by the class name and the .html extension.

public String makeExternalDocURL(String packageName, String className) {
    StringBuffer buffer = new StringBuffer();
    buffer.append(mBaseURL.toString());
    buffer.append(packageName.replace('.', slash()));
    buffer.append(slash());
    buffer.append(className);
    buffer.append(".html");
    return buffer.toString();
}

Now you finally have all the tools to do the work. On doclet initialization, you construct ExternalLinks for the external documentation sources. While writing requestBean and sessionBean tags in your HtmlDocletWriter, you check to see if there is a class that can be represented by an external link. To do that, you find the ClassDoc for the bean type. That step is necessary to get the package that the given type is in, using the context of the file to expand simple names such as String into fully qualified names such as java.lang.String. Once you have a package name and a class name, you call a static ServletDoclet method, which checks every ExternalLink looking for the package. Upon finding the ExternalLink that contains the desired package, the ServletDoclet method calls makeExternalDocURL() to build a URL to the documentation for the target class.

protected String getBeanLink(String type, ServletDoc doc) {
    ClassDoc cd = doc.findClassDoc(type);
    if (cd == null) return null;
    
    PackageDoc pack = cd.containingPackage();
    return ServletDoclet.findExternalDocURL(pack.name(),cd.name());  
}
static String findExternalDocURL(String packageName, String className) {
    Enumeration en = sExternalLinks.elements();
    while(en.hasMoreElements()) {
        ExternalLink link = (ExternalLink)en.nextElement();
        if (link.containsPackage(packageName)) {
            return link.makeExternalDocURL(packageName,className);
        }
    }
    return null;
}

Conclusion

Your New Year’s resolutions were to get more exercise, take your vitamins, and write good documentation for the Webpage designers who will use your servlet. Will you settle for 1 out of 3?

The ServletDoclet presented herein can make it much easier to write and maintain useful servlet documentation for Webpage designers who may not appreciate reading normal JavaDoc, and couldn’t possibly scan your code to figure out what parameters you support. JavaDoc is an elegant solution that integrates documenting with coding, and the ServletDoclet extends JavaDoc to write Webpages your designers can read.

In addition, that example of the Doclet API’s simplicity might embolden you to customize ServletDoclet further to your own needs (perhaps adding some more tags that fit your situation), or even to write similar doclets for different purposes. You could, for instance, write a doclet that scans your code for tags that indicate “to do” notes you left for yourself in your code, using custom tags you designed. Or you could write a doclet that reminds you to write documentation by listing all the methods you forgot to document. Better yet, why not write a doclet that counts how many new classes and methods you wrote since yesterday, and reminds you to take your vitamins and get some exercise!

Source: www.infoworld.com