Re. manual evaluation cards

Subject: Re. manual evaluation cards
From: Geoff Hart <geoff-h -at- MTL -dot- FERIC -dot- CA>
Date: Tue, 9 May 1995 11:50:23 LCL

Gwen Barnes wrote to express her pain at receiving an "F" in one of
the "evaluate this manual" cards returned by a reader. Cheer up, Gwen,
and anyone else who has been burned this way! Most of these cards
represent a poor excuse for usability testing, and statistically,
they're meaningless in most cases too.

Case in point: we include these cards in our technical notes series,
but they were designed by a manager who knows nothing about survey
design or usability testing. The results of the cards are analyzed
each year to check on our performance. Last year, our
"illustration/graphics" category slipped a bit, with a few more
"average" ratings than the previous year. When we went back to the
reports covered by the survey, the reports that scored the worst were
the ones with the clearest and best-produced graphics (judged by
objective and subjective standards); conversely, the reports with the
highest ratings had graphics that we produced under heavy work loads
and tight deadlines, and that we were a bit embarrassed to publish in
retrospect. The cards are poorly designed, and we didn't get a large
number of replies, yet the manager forced a review of the whole
graphics process, which provided few concrete results. More
importantly, my suggestion that we actually test the graphics on our
audience was ignored. So in effect, we conducted an ivory tower
exercise for no good reason and missed the point of usability
analysis. Go figure!

Yes, you can get useful information from these cards, but unless you
get hundreds of cards returned, or follow up with each respondent, you
probably won't get what you want to get.

--Geoff Hart #8^{)}
geoff-h -at- mtl -dot- feric -dot- ca

Disclaimer: These comments are my own and don't represent the opinions
of the Forest Engineering Research Institute of Canada.

Previous by Author: Re. Innate skills?
Next by Author: Re. Steve Jong's letter to congress
Previous by Thread: Re. Innate Skills?
Next by Thread: Re: Dinosaurs

What this post helpful? Share it with friends and colleagues:

Sponsored Ads