EDI*Key(tm) - Next Generation Electronic Commerce
This project started over four years ago. The vision was to provide a universal way of handling Electronic Data Interchange (EDI), independent of EDI syntax or backend repository databases. This initial scope expanded further through work contributed to the ANSI X12 Future Vision subcommittee on enabling Electronic Commerce across the Enterprise. A paper was written in April 1993, on distributed computer applications, predating current mainstream developments specifically with the Java language and the Java concept.
The three central goals are:
providing the next generation for EDI that is also fully backward compatible with the existing X12 and UN/EDIFACT standards, but provides key new business functions.
provide ability to seamlessly translate between X12 and UN/EDIFACT standards.
automate EDI exchanges using familiar code-free open spreadsheet and SQL syntax, with smart rule based tool mechanisms, thereby making EDI universally accessible.
Prolog provided the core enabling technology for this project. Without it this product would never have existed. It enabled the rapid prototyping of the concepts and demonstration of actual results against live complex data. Beyond that it also enabled brainstorming and what-if development for understanding and solving the underlining structural problems that would otherwise have proved intractable in traditional procedural programming languages.
This really only reaffirms the old adage that to innovate and gain new insight one must deviate from the well trodden path.
This paper discusses the concepts and the method itself, discusses its implementation in Prolog and then looks at where this product is expected to go next to gain widespread commercial use.
The following serves as a definition of the current state of the art in EDI, and therefore allows a substantive view of the shortcomings with regard to enabling the Electronic Enterprise. The initial discussion focuses primarily on the transaction sets in use, and translation methods, later discussions will highlight items more specific to Electronic Enterprises.
Regardless of the particular EDI standard, all are designed around an electronic representation of a paper business document. A unique identifier code is assigned for each type of business document. For example, in the ANSI X12 standard an invoice is referred to as document number X12.2, with a transaction set identification code of 810.
The basic unit of information on the electronic document is the "data element". For the invoice, each item being invoiced would be represented by a data element. Data elements can be grouped into compound data elements, and data elements and/or compound data elements may be grouped into data segments. Data segments can be grouped into loops; and loops and/or data segments form the business document.
The standards define whether data segments are mandatory, optional, or conditional and indicate whether, how many times, and in what order a particular data segment can be repeated. For each electronic document, a field definition table exists. For each data segment, the field definition table includes a key field identifier string to indicate the data elements to be included in the data segment, the sequence of the elements, whether each element is mandatory, optional, or conditional, and the form of each element in terms of the number of characters and whether the characters are numeric or alphabetic. Similarly, field definition tables include data element identifier strings to describe individual data elements. Element identifier strings define an element's name, a reference designator, a data dictionary reference number specifying the location in a data dictionary where information on the data element can be found, a requirement designator (either mandatory, optional, or conditional), a type (such as numeric, decimal, or alphanumeric), and a length (minimum and maximum number of characters). A data element dictionary gives the content and meaning for each data element.
In general, trading partners seldom employ the same database programs and computers, and accessing and storage formats differ significantly. While a structured query language (ANSI SQL) has evolved to unify the accessing method, accessing of data via SQL methods does not address remote transfers of data from one computer to another. More is needed before the data becomes useful in this context. The data must be translated from the format of one computer system to the other so that the latter can process it accurately.
Originally, translation software was developed to support a variety of private system formats. Most often, the sender and receiver were required to contract in advance for a tailored software program that would be dedicated to mapping between their two types of datasets. Each time a new sender or receiver was added to the client list, a new translation program would be needed by the new party to format their data to conform to the standards in use by the participants. Of course, this becomes expensive. Such static systems do not easily allow synchronization of business transactions in distributed business processes that involve global rules, but with participants and actions that are not predetermined. To solve these issues it is desirable to develop automated tools and techniques that are easy to use and allow decomposition of transactions in actions to be performed locally and mapping of local actions onto efficient protocol exchanges.
Previous attempts at improving the translation process have resulted in solutions that are still cumbersome, use inefficient computer programming language syntax and structures, and still require programming staff to implement. For the reasons explained previously, the fixed scripts can be cumbersome, especially for datasets with complex looping. Moreover, there is no flexibility to accommodate a new data structure. In the current state of the art, existing translation systems will abort when they encounter unfamiliar data conditions. Therefore, if a new trading partner comes along a new script must be custom written prior to EDI.
The concept of the Electronic Enterprise requires a transition away from paper form based EDI. Key concepts that are required are the encapsulation of business rules (in EDI parlance the Implementation Guidelines) and also mechanisms to handle state and flow control (such as provided by hyperlinks in Web based HTML). Also message sets must be able to handle partial information, where the complete information is not yet available, or simply is not required for the particular business process. This allows different parts of an enterprise to selectively contribute only the information that is germane to their business functions.
3 Overview of the method:
This tool is centered around a core module written in Prolog. The EDI*Key(tm) engine uses the power of rule based process definitions and backtracking built into Prolog to automatically analyze and manipulate the data structures embodied in the EDI messages. This system is both dynamic and robust, therefore the rule set is defined externally, while the method itself has embedded knowledge of EDI messaging that allows it to process messages even with partial or incomplete structural information.
This technology allows the user to express complex EDI interactions through an interface exactly analogous to using a spreadsheet template, with cells and functions controlling the process. This level of simplicity holds out the promise of providing integrated desktop EDI, with drag and drop translation on a MS Windows based PC system. It also allows the user to focus on the business rules and interactions, without having to use programming language level tools for EDI. Further to this users can exchange spreadsheet templates as a way to integrate their data interactions, and to build and access each others databases.
Existing EDI tools force rigid data structures on end user databases, where as this new translation system can extract and update data to and from heterogeneous data structures. What this means to the end user is that they merely have to tell the system where the data is, or where they want it, and it will access it directly.
The sum result is a contribution which will directly impact on the deployment of EDI in an Electronic Enterprise and to small and medium sized companies. This will include the following items:
Code free EDI data translation.
Direct data access to existing desktop database systems.
Implementation guideline rules expressed as simple spreadsheet templates that can be distributed electronically.
In memory single pass message translation that frees the user from constraints of database and message structure conditions traditionally imposed by EDI systems.
Ability to exchange data integration rules and database structures between end users on an ad hoc and rapid basis.
Provides a template mapping system which allows the details of the data manipulation required for translation to be expressed by a layperson, and without the requirement that they define the entire message structure in every detail.
Ability to easily propagate and maintain EDI business solutions to small businesses by merely exchanging spreadsheet templates. These can define automatically for the end user both remote databases, and or the data exchange between existing databases
Those companies that have existing business applications can easily exchange data by building their own spreadsheet templates. Those companies that are teaming with others can exchange templates to facilitate data exchange between team members. With existing EDI technology this is a time consuming and expensive prospect. The EDI*Key(tm) tool provides the bridge that can easily open and support such interactions.
4 Details of the system:
EDI*KEYtm provides a spreadsheet like metaphor that allows definition of all EDI transactions in a simple and familiar manner. This equates EDI transactions directly to a familiar table and field layout similar to that used for SQL processing, and associates the EDI elements directly to this structure. Such formal transaction descriptions can be intuitively assimulated by an end-user, unlike the formal EDI message structures used in defining EDI standards themselves.
A precis is presented here to give a high level overview of the template based techology. The full details of how the system can map directly between EDI transactions and database SQL formats or ODBC accesses is embodied in the currently pending patent application. However, some of this information is given here, along with a diagram illustrating the components of the template system, and how they interact together. By providing a intuitive GUI based access and control system, the whole set of tools can be packaged to allow an end-user the ability to define their own business processes without requiring in-depth EDI knowledge. Also the system can provide the required analytic capabilities as defined in the distributed processing requirements.
ANSI SQL is seen as the preferred method for database use with the EDI*KEYtm toolset. SQL provides a standard method of data accessing independent of specific database storage formats, but unlike EDI itself this does not address structure and data differences. SQL provides a powerful common point of reference and understanding for the Information Systems professionals who are already skilled in its use. It also provides a simple intuitive entry point for the new user with its English language like metaphor. It is presumed for purposes of the present discussion that the reader is familiar with the ANSI SQL programming standards and language, and its several popular commercial implementations. By linking this technology to EDI*KEYtm a familiar bridge is provided for the end-user to assimulate the concepts and use of EDI.
Further by reviewing the EDI transaction sets and standards currently in use a common repeating set of styles and formats can be deduced.
The EDI*KEYtm system uses the same consistent template metaphor to describe each of these different data structures, unlike the X12 EDI standard which uses combinations of up to six different syntax's for each of the structures.
The power of the template method is that these standards can be derived directly from the equivalent mapping template. This means that a recipient of a mapping template can deduce not only the EDI structural information, but also receives all the additional information that the template method brings. This is especially important for distributed processing, since it provides a means for the sender to communicate process related information independent of the receivers processing system and data formats.
Using the template method allows the end-user to build a high level abstracted picture of the message structure and also formal descriptions of each of their transaction sets.
It should also be noted that the template method is extremely compact and concise. Of course for clarity the template is shown here in a format suitable for review by a system operator. In actual transmission, the format can be compacted down by removing white space and other formatting and filler characters. This allows fulfilment of the objective of efficient protocol
EDI*Key(tm) already been proven to work for complex ANSI X12 transaction structures such as the EDI 856 with HL segments, and U.S Customs EDI 309 (which actually is not X12 standard EDI compliant and thus has many challenging aspects to the message structure). The ability to provide formal descriptions for the complete range of commercial transactions, modelling of transactions and their computational complexity, decomposition of transactions in actions to be performed locally; and mapping of local actions on to efficient protocol exchanges are all existing capabilities.
Figure 1. Schematic showing the components of the system
Figure 2. Example template derivable output structure and segments for ANSI X12 EDI transaction sets.
Example 323 structure:
Segment Name Req Max Max Loop
ID Des Use Occurs Reference
------ ------------------------------ --- --- ------ ---------
ST Transaction Set Header M 1 1 323 TSCN
V1 Vessel Identification M 1
LS Loop Header M 1
R4 Port M 1 999
V9 Event Detail M 999
LE Loop Trailer M 1
SE Transaction Set Trailer M 1
KEY FIELDS: V101
Associated data element segments.
ST,M Derived Structure
143,3,3,M,ID , SEGMENT ID
329,4,9,M,AN , TSCN
7,7,M,ID , vsl_id
10,10,M,AN , voy_no
4,4,M,ID , carr_code
R4,M Port Information
1,1,M,ID , port_indctr
10,10,M,AN , pc
V9,O Event Detail
3,3,M,ID , Literal
6,6,C,DT , port_arv_date / or dprt
4,4,C,DT , port_arv_time / or dprt
SE,M End of Set Marker
96,1,6,M,N0 , LINE COUNT, , , Calculated
329,4,9,M,AN , TSCN, , , Calculated
Figure 3. Example EDI X12 standard message input
V1*7039610*NEW ZEALAND QUEEN*D*104N*SCAC***L
R4*D*D*ORF*NORFOLK, VA**NORFOLK INTL TERMIN**
R4*L*K*WLG*WELLINGTON, NEW ZEALAND****
V9*EDD**920721**WELLINGTON, NEW ZEA***A40
This sample is a shipping port itinerary. It shows what ports the vessel will be visiting on a given voyage. It consists of a detail header, followed by one or more items lines.
Figure 4. Resulting output SQL statements
# Insert statement for Ship Schedule Header
INSERT INTO OMS_SS_HDR (
# Insert statement for Ship Schedule Details
INSERT INTO OMS_SS_DETAIL (
VALUES ('104N','D','JAX',1,'',920819,961127,140848,'E' );
INSERT INTO OMS_SS_DETAIL (
VALUES ('104N','D','ORF',2,'',920817,961127,140848,'E' );
INSERT INTO OMS_SS_DETAIL (
VALUES ('104N','L','MEB',3,920712,'',961127,140848,'E' );
INSERT INTO OMS_SS_DETAIL (
VALUES ('104N','L','SYD',4,920715,'',961127,140848,'E' );
INSERT INTO OMS_SS_DETAIL (
VALUES ('104N','L','WLG',5,920721,'',961127,140848,'E' );
Two of the most widely used EDI syntaxs are X12 and UN/EDIFACT. At first glance they appear to be significantly different in syntax and semantics. However, it is well known that UN/EDIFACT was designed from reviewing the X12 system, and this method proves that they are in fact highly similar in how they define components at all levels. The apparent differences are seen mainly by the human readers who attach more significance to the actual labelling of components than does a computer based interpretation.
4.1 Practical Example of using the Template Method:
5 Commercial use of Prolog:
When considering the use of Prolog on this project we must look at what Prolog offers compared to the 'C' class of languages and compilers. First it is clear that Prolog allowed the development of a complex and sophisticated tool in a very short time. This did include code reuse from modules that started out life parsing E-Mail messages. Also the project was centered around a domain that has traditionally been seen as a major strength of Prolog, parsing structured information. A significant factor here is the ability to run code interactively under the interpreter and examine program flow. Whenever the problem solution calls for using both backtracking and recursion traditional C style debugging methods fall short of allowing the programmer to review the complex program flows and data interactions that result. Prologs predicate based debugging and built-in trace facilities are faster and more intuitive.
Another clear advantage is the strong typing and deterministic checking. With an application like EDI a high degree of confidence is required in the results produced. Also dynamically handling multiple EDI syntax through external rules is a bold step that also requires confidence in the underlying completeness of the logic in the core engine. The Prolog compiler enforces coding that handles logical true/false conditions throughout the process.
A more modest factor is the built-in token generator within Prolog, and the ability to quickly create in memory token lists and then search them. These are at the heart of the traditional strength of Prolog in parser applications. Of course these tools can be added to C through extended library toolkits, and use of an enhanced memory allocation / de-allocation utility. Such tools though rarely integrate so seamlessly into the C environment as they are in Prolog.
Using Prolog for a commercial application requires careful analysis of the objectives and requirements of the project. One very clear indicator is Prologs ability to translate between internal and external representations of complex data structures. Building on from this intelligent features can then be added quickly and this is a real commercial strength of Prolog. The ability to innovate, explore and develop new ideas and approaches, and discover results.
6 Some Interesting Prolog Code Fragments:
To complete out this paper a couple of the more interesting segments of the actual code from within the parser engine are presented here, along with a brief description and discussion.
Transitioning a traditionally conservative industry such as the Electronic Data Interchange one, with entrenched standards and powerful bodies with vested interests is clearly a challenge. Business pressures provide the conduit for change and as such technologies that enable the Electronic Enterprise, while at the same time retaining backward compatibility with current EDI message sets are appealing. Also the X12 based USA community is under extreme pressure to migrate to UN/EDIFACT currently at significant cost. The technology outlined here provides an out for that dilemma by allowing the X12 community to leap-frog UN/EDIFACT. At the same time their existing X12 messages can be reconciled to EDIFACT by using template wrappers. Conversely EDIFACT based business partners can move building a complete Electronic Enterprise by simply extending their existing EDIFACT based messaging by adding template headers. No other technology currently available provides such an attractive combination. Future integration into World Wide Web standards of HTML and Java would be an added bonus that can also be realised.
8 Addendum June, 1997, on XML/Java:
An additional versioning scheme will be required, and this should ideally be server driven, where the server detects that the client requires either a later version of the XML template, or the Java components, and simply sends these to the client as an UPDATE message type.
Key for this system implementation however is that the client is smart and both data and form aware. So that the EDI*Key(tm) process running on the client can exchange data, update data and translate data as required as it processes the XML and thereby receives the templates and associated data. It will of course be able to reverse the process, and take outbound data, translate it, and generate the outbound XML as well. Clearly in this regard the XML can be implemented as nothing more than "SQL" from the point of view of the EDI*Key (tm) engine. An external set of rules defines how the data is output, SQL, XML, HTML(forms), or some other format. (This also ensures great flexiblity as the XML standard evolves and changes).
Some additional considerations also need to be taken into account, Process Control, and Object Oriented support. The latter obviously requires that the template and its data be encapsulated in a wrapper function. Several competing methods could be used for this, or again new XML tokens could provide part of this method. The Process Control is potentially easier to accommodate. The trend within the EDI community seems to be to use the IDEF process modelling language. Given this constraint XML tokens for IDEF entities can be assigned, and then process control lines added to the template format. These could then also be used in tandem with URLs to facilitate delivery and flow control.
David Webber, June 10th, 1997.
Paper orginally published at PAP'97, The Fifth International Conference on The Practical Application of PROLOG, Monday 21st April - Friday 25th April 1997, London, UK titled The Commercial Value of Prolog: An Exploration Through Practical Applications