Rounding figures? (take II)

Subject: Rounding figures? (take II)
From: "Hart, Geoff" <Geoff-H -at- MTL -dot- FERIC -dot- CA>
To: "TECHWR-L" <techwr-l -at- lists -dot- raycomm -dot- com>
Date: Tue, 13 Mar 2001 15:01:36 -0500

Kevin McLauchlan, referring to my comment that "So instead, we "unround"
them and present numbers with more digits than the reader strictly needs,
but that produce the right calculation results.", observes: <<Um, always be
careful with this kind of thing. If you pad your numbers out to x-many
decimal places, to make all the columns line up and to make the arithmetic
appear neater, you may be suggesting a level of precision that is not valid.
That is, your measuring tools/methods may not have been capable of that
degree of precision.>>

Sorry, I was unclear, and the point's certainly worth clarifying. What I
meant was that if the spreadsheet data had three decimal places because
that's what we actually measured, and we rounded those numbers to a single
decimal place in the actual report (because that's what the reader expects
to see and will benefit from knowing), then we'll return to presenting those
original three decimal places if that's what's needed to match the result we
want to present (which was calculated using three decimal places, not using
the numbers rounded to 1 decimal place).

<<On the other hand, if you mix precision... where some numbers are taken to
three decimal places and others to only one place, you are implying a level
of precision in any calculated results that probably does not exist in the
data. Your results should always be shown to the coarsest resolution among
all the numbers that went into the calculations.>>

We mostly (see below) follow the standard mathematical rules for significant
figures, but since the original question politely requested a simple largely
nonmathematical explanation, I didn't figure that introducing a whole new
level of jargon was going to be all that helpful. <g> We're particularly
aware of the problem of significance because many of our research reports
are case studies, and thus present results that may vary significantly when
they're extrapolated to different operating conditions. That being the case,
we often present fewer significant figures than the pure rules of math would
justify using simply because the additional precision would be correct, but

--Geoff Hart, FERIC, Pointe-Claire, Quebec
geoff-h -at- mtl -dot- feric -dot- ca
"User's advocate" online monthly at

"Arthur C. Clarke had suggested that any sufficiently advanced technology
would be indistinguishable from magic--referring to a possible encounter
with an alien civilization--but if a science journalist had one
responsibility above all else, it was to keep Clarke's Law from applying to
human technology in human eyes."--Greg Egan, "Distress"


Develop HTML-Based Help with Macromedia Dreamweaver 4 ($100 STC Discount)
**New Dates!!** San Francisco (Apr 16-17), San Jose (Mar 29-30) or 800-646-9989.

IPCC 01, the IEEE International Professional Communication Conference,
October 24-27, 2001 at historic La Fonda in Santa Fe, New Mexico, USA.

You are currently subscribed to techwr-l as: archive -at- raycomm -dot- com
To unsubscribe send a blank email to leave-techwr-l-obscured -at- lists -dot- raycomm -dot- com
Send administrative questions to ejray -at- raycomm -dot- com -dot- Visit for more resources and info.

Previous by Author: Documenting a touch-screen interface
Next by Author: Saving Word to ASCII?
Previous by Thread: Help! Acrobat stopped working!
Next by Thread: Saving Word to ASCII?

What this post helpful? Share it with friends and colleagues:

Sponsored Ads