RE: Future of FrameMaker: InDesign? [Long]

[HALL Bill <bill -dot- hall -at- tenix -dot- com>]

Dan Emory and a number of other contributors to the FM lists have been
disappointed by the limited improvements that the recent FrameMaker 6.0
release offers - after waiting 4 years from the 5.0 release. They think it
is quite possible that Adobe will abandon the product. In an earlier posting
to the Frame lists in response to these issues, I suggested that the problem
was not related to FrameMaker or Adobe specifically, but rather the
fundamental problem of keeping any kind of documentation requiring continued
maintenance over a long life time (say more than three years) in a
proprietary format. 

The questions were asked: When should companies begin considering shifting
out of FrameMaker format to SGML or XML? What impact does the volume of
legacy documents in FM proprietary format have on this decision? My short
answers, based on my own experiences detailed below, are as follows:

New documents: I am a firm believer that the time is past when any technical
author/publisher should rely on a proprietary text processing application is
past. FrameMaker is a good tool for long/complex documents, but its
fundamental failing is that, like MS Word, WordPerfect, Interleaf, etc. the
FrameMaker's document format is proprietary. Personally, I believe that all
new document projects should be developed in one of the international
standard markup languages, HTML, SGML or XML. Given that HTML is not well
controlled and is being replaced by XML, I would opt for the latter.
Assuming that you pay attention to the XML standards, SGML is only a way to
author in XML under DTD control. Whatever tool you are using for authoring
now, it is fairly certain that it will be obsolete in 3-4 years, so it is a
mistake to think of your authoring in a tool-specific way. It is much better
to focus on content and delivery.

Legacy documents: The essential consideration is to understand your
documentation maintenance cycle. How long are the documents in the authoring
cycle, and after they are initially delivered, how long do you have to
maintain the documents to keep them up to date? How frequent and extensive
is that maintenance? 

For documents with an anticipated life of more than four or five years, if
they are now in a proprietary format, you are faced with replacing your
authoring system anyway. Given the revolution in authoring and publishing
technologies fuelled by the explosive growth of the Web it is a fair
assumption that no product we are using today will be supported in four to
five years. Even if we are using the same brand of tool, it will have
changed radically by then. In this circumstance, your best bet is start
converting your long-lived data from proprietary formats to an XML/SGML
standard sooner rather than later. Yes, it will cost a lot now, but it will
cost even more to do it later.

And, if you are going to convert, it is worth investing in a good SGML/XML
content management database at the same time. Not only will this make the
conversion less painful, but it will give you the opportunity to implement a
range of process and single-sourcing improvements that may be in the
too-hard basket if you are dealing with proprietary authoring formats.

----------

By way of background, the company I work for produces documentation for the
ANZAC frigates, a class which will eventually include 10 ships (8 for
Australia and 2 for New Zealand). The project began in 1990. We have
recently delivered the fourth ship and launched the seventh. The planned
lifetime of a ship is 27 years from delivery. My particular responsibility
is the documentation systems for the maintenance documentation (some 2000
different equipment related maintenance routines per ship). Because of
engineering changes and the fact that the ships are produced for two
different navies, no two ships are identical - although more than 90% of the
equipment is identical across all ships.

We started authoring maintenance routines in 1993, using  WordPerfect merge
tables because we hadn't agreed on a final delivery format for the
documents. However, because the fields of the merge records provided a
parsable structure for much of the information (except procedural text), we
were able to "single source" more than thirty different extracts and
deliverables from the maintenance routines. However, as the number of ships
increased, the only way we could be reasonably certain that we had properly
managed the differences between ships, was to maintain a complete set of
routines for each ship - to the point we are now maintaining 8,000 separate
maintenance routine records in the increasingly obsolete and risky to
maintain WordPerfect environment. The company is totally dependent on my
skills to maintain the delivery system.

A related set of documents (specifications for overhaul of equipment by
external repair agents) where authoring did not begin until 1995, was
authored from the outset in SGML. The DTD was relatively easy to construct
(based on an existing Defence standard). These have been managed only at the
file level, because - unlike the maintenance routines - there has been no
need to deliver other products from the same data. These proved to be very
easy to maintain in SGML, and we have used a variety of products to do so
with no data problems or complaints from the authors.

By 1995 we recognised that it was increasingly risky to try to maintain the
growing volume of maintenance procedures in a proprietary format, however it
took until 1999 to convince management that we had no choice but to migrate
the data into SGML (or XML). In 1995 it probably would have cost us at least
$2 million to migrate our then data (2 ship-sets in development) to SGML.
Today it is costing us about half that to migrate 4 ship-sets). However, in
the intervening period it has cost us several times the $2 million to
implement client mandated changes to the documents, which could have been
done for a small fraction of what it did cost us if we had been able to do
the work in a controlled SGML environment.

We are currently converting the four ship-sets of data into SGML for
management in what we believe to be the world-wide state of the art SGML/XML
database (RMIT University's Structured Information Manager - "SIM" -
http://www.simdb.com/). We looked at everything available on the world
market in 1998, and despite dismissing SIM at the outset as being "too
academic", we finally concluded that the local product was technically
superior to any other product we reviewed. The decision was also not hurt by
the weak $A. As well as a very capable repository, indexing and delivery
environment, SIM includes a workflow management capability and a very
powerful object oriented scripting language called Ace. Ace has built in
libraries for RTF, HTML, SGML, XML, MARC (library markup), X39.50 (library
database communications protocol), Oracle API exchanges, etc. (If anyone in
North America is looking at XML databases, SIM already has some substantial
US customers, and my understanding is that RMIT is substantially beefing up
its US distribution network. For further information on the US distribution
arrangements, contact Phil Anderson - mailto:phil -at- mds -dot- rmit -dot- edu -dot- au -dot- )

Using SIM-based tools, the conversion process is comparatively straight
forward - but, unavoidably, involves labour intensive components: 

(1) A WordPerfect macro converts the merge table format into RTF,
substituting field and record delimiters with identifiable ascii strings as
tags.

(2) Ace script converts the RTF into SGML according to our strict authoring
DTD (if possible) or conforming to a "Loose" DTD including an error tags
which can be placed around structures which cannot be interpreted as
conforming to the Author DTD.

(3) Some WordPerfect records were so badly structured (there was no way to
control authors in the WordPerfect environment) that the Ace conversion
parser could make no sense out of them. These had to be re-worked in the
WordPerfect environment before they could be successfully imported. In our
initial conversions (performed at a time when the WordPerfect data was
reasonable stable), less than 3% of the records required manual intervention
in WordPerfect (as we got closer to going live, the rate of manual
intervention actually rose because more authors were making more changes -
and not getting them right in the uncontrolled environment).

(4) About 70% of the records were successfully converted all the way to the
Authoring DTD, the remaining ~30% require some intervention in FM+SGML
(under structural control by the Authoring DTD) to fix remaining problems.

(5) All of the records require author intervention to implement value adding
features allowed by the SGML, such as addition of system and effectivity
(engineering change ID vs Immediate) attributes at the record level, and
"language" (RAN vs RNZN) attributes at the element level.

Our live data has been converted through stage 4, and we are starting stage
5 this week.

The bottom line for our conversion from a "smart" single-sourcing word
processing application to SGML in a smart database, is that we are reducing
four ship sets of routines (~8,000 or >20,000 for the eventual 10 ships) to
less than 2000 routines for the class as a whole. Additionally, we will be
able to automate review, release and change management and link deliverable
elements to source documents used by authors to vastly improve our ability
to identify impacts of source document changes on our deliverables).
Previously, we replaced all ship-sets of documents each year, in the future
we will be delivering only changed routines as they occur. (The client also
benefits substantially from this, and have changed their maintenance
management system correspondingly). The net result reduces the documentation
maintenance requirement for our 5th delivery by 80% and 90% for the class of
10 ships.

The next release of SIM will include a comprehensive ability to manage and
reuse redundant document components down to the lowest element level. This
will allow us to manage all standard phrases, warnings, cautions, notes,
standard steps, etc. from single locations in the database. (SIM will have a
built-in capability to recognise texts that are identical, and will prompt
authors to determine if similar texts should actually be identical). These
capabilities will reduce the total volume of text requiring management by at
least another 50% - to reduce the overall text for 10 ships to less than 5%
of what would have been required had we tried to continue maintaining the
WordPerfect system.

Beyond this, now that we have the SIM technology implemented in-house, we
will be working to move most of our long-lived corporate documentation into
this environment. Two types of management are possible: strict control -
where authoring will be done under DTD control and they are stored as
"valid" SGML/XML, and relaxed control for other classes able to be managed
as "well formed" XML. The latter documents could be authored in well formed
XML or they may be automatically converted using SIM scripts. By storing
documents in well formed XML, SIM would be able to index them with
substantial intelligence, but it would not be feasible to re-use elements
with safety.

If anyone is seriously considering making this kind of shift, and has
specific questions on SGML/XML databases or conversion planning I might be
able to help with, I will try to answer them.  However, the response may not
be immediate, as I expect to be rather busy for the next month or so, since
I am one of the people who tasked to add value to our converted data before
we deliver our class set of data for Ship 5.

Bill Hall
Documentation Systems Specialist
Integrated Logistic Support
Naval Projects and Support
Tenix Defence Systems Pty Ltd
Williamstown, Vic. 3016 AUSTRALIA
Email: bill -dot- hall -at- tenix -dot- com <mailto:bill -dot- hall -at- tenix -dot- com>