RE: Quality

["Focus on 3 things: Quality, Quality, Quality" <raven -at- USABLE -dot- ENET -dot- DEC -dot- COM>]

Tom,
        Yes, the issue of user testing raises all sorts of interesting
question. And yes, NCR is heavily involved. No apologies necessary!
I also think Chris Velotta at NCR is also working on a book (to be
published by STC or Baywood or maybe MIT Press??)

        Now, however, I make an important distinction between user
testing and the kind of thing that Chris + Northern Telecomm (Bill
Hosier) + RPI (Phil Rubens, Bob Krull and grad students) when they talk about
using a document quality formula.

        (By the way, the citations for the prelimiary study that Tom
mentioned are:"Basing Document Quality Standard on Research" Bill
Hosier, Philip Rubens, Robert Krull, and Christopher Velotta,
Proceedings of the 1992 STC annual conference,  pp. 428-431 and
"Quality Measurement: Using a Document Quality Formula" Claudia M.
Hunter, Proceedings of the 1992 STC annual conference,  p. 432.)

        In the project described by the above articles, NOT ONE USER
WAS ACTUALLY TESTED! Rather, the researchers identified **over 140**
variables that might impact document quality. From those 140, they
decided that only 20 could actually be reliably measured (for example,
I think they decided that " elaboration of instructions" and "Logical
relationships" couldn't be reliably measured.)

        Then, as Hunter describes, the RPI writing Quality project
trained coders (NOTE!! They did not write computer programs--because
they depended on human holistic evaluations of concepts like
"paragraph cohesion", which, as far as I know, there is no algorythm
written for) who rated various books on the 20 variables--which means
they ignored 120 variables. Hunter does sum up that "the document
quality formula allows us to say that it is possible to identify
measureable variables in documents that have a real effect on
quality."

        In my (not-at-all humble) opinion, the document quality
formula project is well done, and it's a very noble endeavor; if we
could really create a reliable document quality formula, then we would
not have to involve users, we could just apply this forumla before we
ever sent a user guide to anybody, and we'd look wonderful and our
customers would be happier. But I don't think that will ever happen.
The quality formula CANNOT and SHOULD NOT replace user testing.
(an aside: I also don't think any of the people involved in the
project would say that the quality formula project is meant to
*replace* user testing; perhaps it might be done before user testing.
The issue remains, however, that training coders to evaluate 20
variables is a H**l of a lot of work--I know cause I was at RPI when
Claudia Hunter was a grad student working on this project. We've
talked about it, and all the code training and evaluating would
probably take just as much time as user testing.)

        Users (readers/customers) are really the only ones who can
identify "opportunities for misunderstanding" and the act of incuding
users in testing also creates relationships that effect how customers
perceive our products.

        So, that's my 2 cents for today. I would encourage anybody to
read the articles and to get the videotapes and workbooks from NCR,
because knowledge is power--the more we know about usablity
processes/quality processes, the better choices we can make.

Mary Beth Raven
Digital Equipment Corporation
Raven -at- usable -dot- dec -dot- com