Build database-powered mobile applications on the Java platform

Internet-connected mobile devices are important tools in our daily lives. The ability to access information in realtime and in real-life circumstances has enabled us to work more efficiently. To fully harness mobile commerce’s potential, we need to integrate wireless devices with the rest of the Internet. In particular, wireless devices should be able to access the vast amounts of data on Internet backend databases and utilize data processing power from enterprise-level database applications.

To achieve that functionality, Java is our best bet. It offers the ideal platform for developing complex mobile commerce applications. Java programs can deploy on both the wireless device and the application server, independent of host hardware and operation systems. Client-side and server-side Java applications can integrate seamlessly under consistent API designs.

In this article, we explore ways to access Internet backend databases from Java-enabled mobile devices. We use a simplified example to illustrate our approach and explain the design and implementation decisions we made.

The Java platform for small mobile devices

The standard Java APIs and virtual machine (VM) are feature rich. But standard Java applications are simply too big and too slow to run directly on resource-constrained mobile devices. Small runtime memory footprint and fast execution speed prove essential for mobile commerce developers. In response to this need, Sun created a small-footprint edition of Java: J2ME (Java 2 Platform, Micro Edition).

J2ME is divided into configurations and profiles. You can use different combinations for different memory and processing power requirements. The Connected Limited Device Configuration (CLDC) and Mobile Information Device Profile (MIDP) combination is designed specially for resource-constrained mobile devices, such as wireless phones and personal digital assistants. CLDC provides class libraries to support a limited set of Java core language APIs; MIDP provides device-specific implementations for application-level APIs, such as GUI components, network connection, and persistent storage. CLDC and MIDP work together to provide a complete development and runtime environment for mobile devices. Each MIDP-supported mobile device has a MIDP VM that can run compiled MIDP byte code. Thus, MIDP applications run on all MIDP devices.

Unlike applications built around browser-based technologies such as WML (Wireless Markup Language) and other HTML variants, MIDP applications are real programs that can handle interactive user interfaces, manipulate data, and allow offline user interaction. In contrast with platform-native applications, MIDP applications are portable across all mobile platforms and can deploy over the Internet. MIDP developers can take advantage of Java’s sound object-oriented design and the many third-party libraries/components available for the Java platform.

A MIDP application’s basic component is the MIDlet. A MIDlet can have its own event handlers and user interface. Several MIDlets and their supporting classes work together in a MIDlet suite to provide a MIDP application’s complete functionalities (for more information about MIDP development, see Resources). However, MIDP lacks a crucial functionality for database-powered applications: Java Database Connectivity (JDBC). In the next section, we discuss how to access databases from MIDP applications.

J2EE middleware

In theory, MIDP applications on wireless devices can directly communicate with backend databases using raw TCP/IP network sockets. However, the mobile application must therefore handle all database-dependent communication, resulting in an application that cannot be ported across different backend database servers. On top of that, the mobile application needs to process both the business logic and the presentation logic — a huge burden for resource-constrained mobile devices, thus producing inefficient, unexpandable, nonportable applications. In a sound design, the mobile application should handle only the interaction with the user (presentation logic) and leave everything else to the server.

The server-side requirements necessitate another server-side layer between the databases and MIDP applications. J2EE (Java 2 Platform, Enterprise Edition) provides the perfect tools for such server-side middleware. JavaServer Pages (JSPs) can communicate with MIDP applications via the standard HTTP protocol; JSPs can also interact directly with databases through database-independent JDBC APIs or integrate with other J2EE components, such as Enterprise JavaBeans (EJBs). For more information on JSP, JDBC, and EJB development, please refer to Resources.

Architecture

Figure 1 shows the basic architecture discussed above. The MIDP application runs on mobile devices. MIDlets communicate with JSPs through the HTTP protocol using data formatted as XML (more on this later). The JSP then passes the MIDlet request to a middle layer of application software that handles business logic and communicates with databases through JDBC. The middle layer can be a set of helper classes in the JSP container or a separate EJB server. Since the focus of this article is MIDlet-JSP communication, in our simplified example, we combine middle-layer business-logic handling and database access functions into JSPs.

Example

This article’s example is a notes/comments collection program. We want to gather user feedback on a new product. First, the user needs to log in and start a session. Then, on the mobile device, he records any notes/comments he has on that product during realtime usage. At any time, the user can send the time-stamped notes to a central database for further processing. In this example, the central database has only one table, Notes. The SQL listing below shows the table schema:

CREATE TABLE Notes (
  Notes.NoteID    BIGINT(20) NOT NULL AUTO_INCREMENT PRIMARY KEY,
  Notes.UserName  TEXT,
  Notes.Time      DATETIME,
  Notes.Note      TEXT
);

Note: You can download the source code for both the client- and server-side Java example programs from Resources. The MIDlets directory contains all the MIDlets, and the jsp directory contains all the JSPs. The support directory contains supporting Java classes that need to be deployed on both MIDP and JSP server sides. See the sidebar below, “Deploy the Example MIDP Application on Palm OS Devices,” for more information.

The application works as follows:

1. The MIDlet Session manages the HTTP session information through the JSPs updateSession.jsp and getSession.jsp. In the example, for illustration purposes, only one string variable, username, is stored with the session. No password check or username validation is performed. Real user authentication reaches beyond this article’s scope.
2. The MIDlet Record receives user notes in realtime and stores them in a record management system (RMS) record store (database) with time stamps.
3. The MIDlet SendData reads the notes from the record store, sends them to updateDB.jsp — which stores the notes — and time-stamps them with username in a backend relational database.

Figure 2 shows the screen shots of a complete session on the MIDP side, from login to upload of all notes to a database.

Figure 3 shows the content in the backend database after the session completes.

We want to emphasize that this is only a simplified example to illustrate our approach. In the real world, a mobile application should avoid any lengthy user input. You could replace the note-taking process with interactive multiple-choice questionnaires. The questions and choices could be generated dynamically from databases and sent to MIDlets via JSPs as well.

We developed and tested the example system with the following toolkits/environments:

• On the MIDP client side: CLDC 1.0, MIDP 1.03, MIDP4Palm 1.0 on a Palm V running Palm OS 3.5
• On the server side: J2EE 1.3, Jakarta Tomcat 4.0.1, MySQL 3.23.44, and the MM.MySQL JDBC driver 2.0.4

We compiled and packaged the Java programs using standard Java 2 1.3.1 tools.

To follow the rest of this article, you should have a basic understanding of how to implement standalone MIDP or JSP applications. For an earlier JavaWorld series that can help you get started with MIDP, see Resources below. In the following sections, we will address some implementation issues and show you how to make client- and server-side Java applications work together.

Persistent storage on MIDP

One of the most important features of MIDP is its ability to store data in on-device databases. It facilitates client-server communication by providing a data cache. That cache allows the wireless application to work on its own for a period of time without assistance from the server side (JSPs). It drastically improves mobile applications’ flexibility and speed. In this section, we discuss how to implement persistent storage in our example application.

We use RMS record stores to store user notes and associated time stamps in the MIDlet Record. RMS record stores resemble files on filesystems. They can be opened and accessed by their names and can survive soft resets (reboots). A record store can store a number of Records. Each record has a unique RecordID as the primary key, and the record itself contains data in binary format.

The MIDP API offers many methods to manipulate stores and records. The following method opens a record store:

RecordStore noteStore = RecordStore.openRecordStore("JWTestStore", true);

The first parameter in openRecordStore() is a name that uniquely identifies the record store. The second parameter indicates whether or not a new store should be created if the named store does not exist.

To add a new record containing the string note to the record store, we use the addRecord() method:

noteStore.addRecord( note.getBytes(), 0, note.length() );

After we gather a few notes, we can send them to the database using SendData. In the MIDlet SendData, we transverse all the records in a store using the enumeration object and then delete the processed records:

RecordEnumeration re = noteStore.enumerateRecords(null, null, false);
while ( re.hasNextElement() ) {
  int id = re.nextRecordId();
  byte [] b = noteStore.getRecord(id);
  // Do something about the data in array b.
  // Assemble it into the XML document to be send to server.
  noteStore.deleteRecord(id);
}

The first parameter in method enumerateRecords() is a RecordFilter, which decides which records to include in the returned enumeration. The second parameter is a RecordComparator, which specifies the order in which the records are returned. In the above example, we used the null value for both parameters, so the enumeration contains all records in the store in no particular order. The third parameter indicates whether the enumeration should be kept up-to-date with changes in the record store.

The above name-based record-store access methods are convenient. However, there is a problem: you can install multiple MIDlets suites on the same device, and chances are that some suites might choose to use the same record store names. To avoid that confusion, the MIDP RecordStore class has a built-in security model. Any MIDlet in a suite can access only RMS databases created by MIDlets in the same suite. The static method RecordStore.listRecordStores() returns an array of record store names accessible to the current MIDlet.

Data connections and session management

In this section, we look at how to establish data connections on both the client and the server side. On the MIDP client side, open a new connection to a specific URL with the following code:

HttpConnection conn = (HttpConnection) Connector.open(url);

To send data over HTTP, use conn.setRequestMethod(HttpConnection.POST) for the POST method or conn.setRequestMethod(HttpConnection.GET) for the GET method.

Now we can send data to the server using the output stream from the connection object:

DataOutputStream os = conn.openDataOutputStream();
os.write(StringToPost.getBytes());

Finally, we can retrieve the server response data from the input stream:

DataInputStream is = conn.openDataInputStream();

On the JSP server side, the client’s request data can be retrieved in the input stream reader:

Reader reader = request.getReader();

The server talks with the database through JDBC connections (Do not confuse JDBC connection objects with HTTP connection objects!):

Connection conn = DriverManager.getConnection("jdbc:mysql://computer/DB", DBusername, passwd);

Then, the server can send response data back to the MIDP client through the output stream. In a JSP, any text outside the scriptlet (embedded Java code) sections automatically writes to the output stream.

Session management

Since our server application needs to support simultaneous multiple users, we must track each user’s state information to store notes with the appropriate usernames. However, HTTP is a stateless protocol, and each HTTP connection conducts a separate transaction with no knowledge of the context. A standard way to solve this problem is to have the server group relevant connections into sessions and store persistent state information with the sessions. Every MIDlet and JSP in our example application uses the technique discussed in this section to manage sessions.

The most common way to use sessions in HTTP applications is to embed unique identification text called cookies in HTTP headers. If the cookie from a request header matches a cookie on the server side, the server associates the current connection with a session through that cookie. On the server side, each session is an HttpSession object. The cookie handling is automatic in JSP containers.

The code below retrieves the HttpSession object for the current connection. If the current connection is not associated with any active session (i.e., if there is no valid cookie), a new HttpSession object will be created, and a new cookie associated with it will be sent back to the client for future identification.

HttpSession sess = request.getSession(true);

We can bind an arbitrary object containing persistent session-state data to a session with a unique name:

sess.setAttribute("Username", username);

We can then retrieve the bound object:

username = (String) sess.getAttribute("Username");

But for the method getSession() to get the correct HttpSession object, the MIDP client must remember to send the correct cookies. The small-footprint implementation of the MIDP HttpConnection class lacks built-in cookie support. However, HttpConnection does provide several methods to manage HTTP headers directly.

Before a MIDP client sends out an HTTP request to a server, it uses the following method to set a cookie in the request header:

conn.setRequestProperty("cookie", cookie);

After the server responds, it retrieves the kth key/value pair in the HTTP response header:

String key = conn.getHeaderFieldKey(k);
String value = conn.getHeaderField(k);

If the key equals the string set-cookie, then the value is a new cookie from the server and is stored for future use.

Automatic cookie handling

Since all communication methods in the MIDP application need to use cookies, we should encapsulate cookie-handling code in connection classes and reuse them throughout the application. Sun’s new J2ME/J2EE wireless showcase Smart Ticket Demo implements a simple cookie-aware wrapper class called SessionConnector. The usage is simple: we just call SessionConnector.open() instead of Connector.open() to get new HttpConnection objects, and cookies are handled automatically. We use the SessionConnector class to track sessions in the MIDP application in our example

The cookies are stored as a static data string in the SessionConnector class. That cookie string is automatically embedded in the request’s header when a connection opens.

Class SessionConnector also automatically retrieves cookies from server responses. It uses class HttpSessionConnection, which is a cookie-aware implementation of the interface HttpConnection, to represent HTTP connections. Class HttpSessionConnection‘s openInputStream() and openDataInputStream() methods retrieve cookies from an HTTP header and store them in the SessionConnector class when a server response is received.

Since cookies are stored as static data members, they are persistent within a VM. However, sometimes we might desire cross-VM persistence. We can achieve that by storing cookies in an RMS record store. Though this is not hard to implement based on the SessionConnector class, storing cookies in an RMS record store reaches beyond this article’s scope.

Put together cookie-handling code on both server and client sides

To summarize the discussion above, the MIDlet SendData manages the communication data as follows:

HttpConnection conn = SessionConnector.open( url );
conn.setRequestMethod(HttpConnection.POST);
// Prepare an XML document containing data to be sent to
// the server in string variable "post"
// ... some code ...
 // Send the XML data in the HTTP request through the output stream.
 DataOutputStream os = conn.openDataOutputStream();
os.write(post.getBytes());
os.flush();
os.close();
// Get response
DataInputStream is = conn.openDataInputStream();
//Parse the response XML document and extract useful information
// ... some code ...
// Important!
 is.close();
conn.close();

On the server side, the JSP updateDB.jsp processes data from SendData and forms a response:

HttpSession sess = request.getSession(true);
  // Session never expires. The expiration date/time can also
  // be set in servlet container's config file.
  sess.setMaxInactiveInterval(-1);
  String username = (String) sess.getAttribute("Username");
  if ( username == null ) {
    username = "anonymous";
  }
  // Get the JDBC driver
  Class.forName("org.gjt.mm.mysql.Driver");
  Connection conn = DriverManager.getConnection("jdbc:mysql://localhost/JWTest","JWTest", "JWPass");
  // Get data from MIDlet SendData
  Reader reader = request.getReader();
  // Parse incoming XML and store results into 
  // database through the JDBC driver.
  // And update string variable "status"
  //   ... some code ...
  conn.close()
  // The following part outside of scriptlet is written directly to
  // the output HTTP stream and can be retrieved by MIDlet "SendData". 
%>

  
    
      
    
  
 

Data communication using XML

In the previous section, we discussed how to establish persistent sessions between MIDlets and JSPs. Now we need to make the server and client sides understand each other. We decided to use XML as the communication data format. XML offers great advantages over flat byte streams, even though it increases bandwidth and processing overhead. XML expresses more complex data structures, tolerates minor imperfections, and is human friendly. As a result, XML ensures a more robust and maintainable system.

MIDP lacks built-in XML capabilities, so we must rely on third-party XML tools. Again, for resource-constrained mobile devices, the XML parser should not create too much overhead. Moreover, the parser must work on top of the limited string and I/O functionalities provided by CLDC. Several small, CLDC-compatible XML parsers are available under a variety of licenses. In this example, we use a stable, lightweight parser called kXML. Its source code is available under the Enhydra Public License and is free for commercial and noncommercial use.

kXML can act as either a tree-based or an event-based parser. Since our XML documents are data-centric, they do not need special event handling. Easy and robust data access to the XML hierarchical structure is our priority. Thus, we decided to use the tree-based parser. The standard Java API for tree-based parsers is JDOM. However, JDOM proves too complex for small applications, and its implementation requires core language functionalities not provided by CLDC. So, kXML comes with its own tree API, called kDOM.

We could write kDOM parsing and data-accessing code for each XML document exchanged between a MIDlet and a JSP. But that would prove inefficient and would produce much hard-to-maintain, duplicate code. We need a better design to handle XML communications. Our solution: use a custom XML format specially designed for relational database data.

Database-oriented XML

Since we are dealing with relational database tables, the communication data structure should match the database table structure. So in our example, each XML document has two levels of tags under the root element. The upper-level nodes map to table rows. The tag names are the table names. Inside each table row are lower-level nodes for each data field. The lower-level node tag names are field names, and the body strings are the contents in the corresponding table field. For example, we can represent two rows in our Notes table as follows:

<root>
  <Notes>
    <NoteID>1</NoteID>
    <UserName>Juntao</UserName>
    <Time>2001-12-07 18:00:15</Time>
    <Note>This is fun!</Note>
  </Notes>
  <Notes>
    <NoteID>2</NoteID>
    <UserName>Ju</UserName>
    <Time>2001-12-07 19:00:00</Time>
    <Note>This part needs more work!</Note>
  </Notes>
</root>

Our XML parser wrapper class DBXMLParser parses a database-format XML document into a tree object and provides methods to access the data:

public class DBXMLParser {
  private Document doc;
  public DBXMLParser (Reader reader) throws Exception {
    XmlParser parser = new XmlParser (reader);
    doc = new Document();
    doc.parse(parser);
  }
  // .. Data access methods ...
 }

The method getRowCount() gets the number of rows for a given table name:

/**
 *  The number of rows in table "tableName"
 */
public int getRowCount( String tableName ) {
  Element root = doc.getRootElement();
  int result = 0;
  // Iterate through the children of the root node.
  for ( int i = 0 ; i < root.getChildCount(); i++ ) {
   // Only the "ELEMENT" type nodes count.
   if ( root.getType(i) == Xml.ELEMENT ) {
      Element tr = (Element) root.getChild(i);
      if ( tableName.equals(tr.getName()) ) {
        result++;
      }
    }
  }
  return result;
 }

The method getFieldValue() retrieves data in a given field from a row specified by a table name and an index (starting from zero). For example, if we run the following code against the above XML segment, it returns the string Juntao:

String s = getFieldValue("Notes", "UserName", 0);

Please refer to the source code for that method’s detailed implementation.

The class DBXMLParser also has a helper method, rowString() (shown below), which assembles an XML node segment representing a data row from Java data objects. The input parameter tableName is the row node’s tag name; the parameter fieldNames contains field names in the row, and fieldValues contains field values. We can easily add more XML support methods to that class as needed:

/**
 *  Compose an XML string representing a row of data with
 *  a number of fields.
 */
public static String rowString (String tableName, String[] fieldNames,
                                String[] fieldValues) throws Exception {
  StringBuffer sb = new StringBuffer();
  // Start tag of the row.
  sb.append("<"+tableName+">");
  for (int i=0; i<fieldNames.length; i++) {
    // Start tag of a field.
    sb.append("<"+fieldNames[i]+">");
    // If the field contains text, write text as
    // the body of the field tag.
    if ( fieldValues[i] != null ) {
      sb.append(fieldValues[i]);
    }
    // Close the field tag.
    sb.append("</"+fieldNames[i]+">");
  }
  // Close the row tag.
  sb.append("</"+tableName+">");
  // Output the composed string.
  return sb.toString();
}

XML parsing on the server side

In the example, we use the same DBXMLParser class to process XML data on the server side for simplicity. In reality, since the server-side application has access to all of Java’s rich features, it can use more capable XML parsers. The server can validate XML documents and ensure that they conform to certain specifications. That could greatly reduce the hard-to-debug errors caused by malformed XML documents.

Build your own database-backed mobile apps

In this article, we demonstrated that J2ME and J2EE can work together seamlessly to provide a complete mobile commerce solution that includes wireless devices and backend servers. We discussed the overall architecture and many implementation issues, such as session management and communication protocols. Those techniques should help you design and implement your own database-backed wireless applications. Now it’s your turn to come up with some truly innovative mobile commerce ideas and start another revolution!

Michael Yuan and Ju Long are PhD candidates at
the University of Texas at Austin. They use J2ME and J2EE to
develop mobile research applications for projects in the Center for
Research in Electronic Commerce.

Source: www.infoworld.com