Collaborative challenges
Developing collaborative application platforms means keeping tabs on a host of evolving standards and data transformation issues
VENDORS FROM BOTH the integration world and the application development world are tackling the collaborative applications challenge, building new middleware layers they hope will become platforms for the development of these next-generation enterprise applications. Their efforts will ultimately move the enterprise beyond point-to-point integration using proprietary APIs and toward a standards-based, loosely coupled, business-rules-driven software layer that enables enterprises to stitch together existing systems components into collaborative applications.
But several thorny problems need to be solved to make collaborative applications work, including data transformation, business process coordination, and transactionality. Current efforts, based largely on existing open-system interfaces such as XML, are a transitional step toward a more services-oriented approach once Web services interfaces and protocols such as SOAP (Simple Object Access Protocol) and WSDL (Web Services Description Language) become widespread. Here’s a rundown of some of the key challenges.
Data transformation and synchronization
Although many legacy systems and packaged applications speak XML, their data models and schemas are frustratingly different. The definition of an “order” or a “customer” may vary greatly, for example, among an SAP system, a homegrown order management system, and an SCM (supply-chain management) system. Moreover, many enterprises don’t maintain comprehensive data dictionaries except in the heads of the people who originally wrote the systems’ code. So business vocabulary management — the mapping and transforming of data from multiple systems into a single consistent schema — is one of the tougher problems facing collaborative application platform developers.
New software approaches for rules-driven data cleansing and mapping, as well as dictionary definition, are under development. Vitria, for example, is developing a vocabulary-based transformation approach with a correlation engine that can scan through thousands of documents, each containing thousands of terms, and draw rules-based mapping inferences that can then be approved or rejected by a human analyst.
A related issue is the management of data scattered over multiple origin systems. How, for example, should a company consolidate data into one logical view with a unified architecture, thereby enabling data-source independence? Because application data continues to live and change in the origin systems, the new software layer must be able to retrieve origin data on the fly and also propagate changes back to the origin systems.
“Our intellectual property is all around dynamically aggregating data under different circumstances,” explains Rob Beauchamp, CTO of Austin, Texas-based Journee Software, which is developing collaborative middleware focused on CRM apps. Journee provides tools to allow enterprises to define the logical data model of their middle-tier software layer. “It’s acting as if it is a database, but it’s really not. … It’s really retrieving data from the underlying data stores at run time,” he says.
Other vendors, such as Asera and WRQ, are using structured XML schemas based on the World Wide Web Consortium (W3c) XML Schema standard to provide this unified view, exposing a common aggregate interface to the collaborative application and moving around XML documents with defined semantics, such as validation rules, rather than the underlying data itself.
Although raw XML lacks the more abstracted services-oriented interfaces that will come with SOAP, and thus requires more custom development, it reduces a company’s dependence on scarce Java programming talent. “It changes how people do development,” says Steve Benfield, CTO of Billerica, Mass.-based SilverStream. “It doesn’t require a severe technical skill set, and I don’t have to think about how I’m going to move data around. … The tools around XML do most of the work.”
Web services protocols would make collaborative application development even easier, providing the conventions and formats to define interfaces, messages, data types, and structures; to map messages onto different transports; and to format and store metadata. But in the meantime, enterprises and their vendors can roll their own using XML.
Managing process flows
Another obstacle to collaborative app development is the lack of standards for describing and orchestrating business process flows across multiple systems. “You’re basically stringing a whole bunch of applications together now instead of writing them in a more monolithic way,” so issues of modeling and process flow become even more important, Benfield explains. “How do I handle retries and time-outs, and what’s my quality-of-service agreement?” he asks, noting that in the absence of standards, most enterprises are just winging it, especially for collaborative applications that reach outside the firewall.
Furthermore, collaborative apps are likely to be long-running; that is, they have several steps that play out over time and potentially include human intervention as part of their workflow process. “Everybody has their own way of defining their own processes and interfaces,” says Anil Nori, CTO of Asera in Belmont, Calif. “There is really no workflow standard right now.”
Developing collaborative applications still involves a lot of application-specific human engineering: talking directly to the participating applications’ developers and finding out what the app can do, what kind of data it can act on, what its security and authorization model looks like, and so on. Only then can common rules and business processes be built atop participating systems.
A raft of proposed standards is in the works to solve this problem, including a host of vertical-market efforts. Microsoft’s XLANG, IBM’s WSFL (Web Services Flow Language), and BPMI from the Business Process Management Initiative are three closely watched contenders. Similarly, ebXML (e-business XML) has a business process component, but it is based on an object model and thus lacks abstractions for loosely coupled processes. “I’m just waiting for something very simple, like XLANG, that also makes sense conceptually, like WSFL,” says Eric Newcomer, CTO of Iona Technologies.
Until an overarching standard emerges, the tools for building collaborative apps will be targeted at developers rather than the businesses users who best understand the business processes. “Now we’re going from programming languages to markup languages to representing the way people think about the world,” explains Newcomer, saying that a key to collaborative application platforms will be their capability of allowing users to debug processes at the visual abstraction level rather than the XML/script level.
A third challenge will be providing support for transactions across multiple back-end systems. “The hard part is getting a transaction model wrapped around those back-end systems; so if it didn’t work in the third system down, it was able to roll back in those first two systems,” Journee’s Beauchamp explains.
Transaction definitions must also be consistent with process flow definitions so that transactions can be rolled back to the appropriate level of granularity in the process, which in turn must be tied to the level of granularity in the security model. One approach is XA, a proposed W3C standard for putting locks on systems while doing transactions across multiple systems to assure transaction integrity. But large software vendors such as Siebel don’t support XA yet, claiming that customers are not asking for it.
Another issue with XML-based collaborative applications is transaction performance. CORBA or Java RMI (Remote Method Invocation) may be more efficient than XML-based apps for high-volume transactions because all-text XML documents take up more space. “If you’re doing millions of these things, you may want to keep the data in a binary format,” SilverStream’s Benfield says. “But for most applications, we’ve found that XML works fine.”
Those vendors who would provide a platform for collaborative applications claim that their middleware architectures are open and that their competitors’ are ill-suited and proprietary. But all agree that some combination of technologies — XML-oriented middleware, traditional publish-and-subscribe messaging, and services-oriented Web services architectures — will be necessary to do the job. Enterprises want to leverage their existing application components, want the business logic to reside in middleware rather than at end points, want middleware that’s independent of any specific vendor’s control, and don’t want to write a lot of expensive custom code for each new app. Now it’s up to the vendor community to deliver — collaboratively.
