Significant digits

Subject: Significant digits
From: geoff-h -at- MTL -dot- FERIC -dot- CA
Date: Fri, 23 May 1997 12:51:26 -0500

Jennifer Kraus wondered about the use of "significant
digits" and gave the following example of her boss'

250 gallons = 940 L and not 946 L
500 gallons = 1890 L not 1893 L
30-125 psi = 2.1-8.62 bar not 2.06-8.62 bar

The basic notion is useful, but your boss's implementation
is wrong. You can see this quite easily by comparing your
first and second examples (gallons): your boss's version is
off by 10 L, versus only 1 L for your version (based on the
correct conversion factor). This tenfold difference is
likely significant in some or perhaps even many cases, even
though the magnitude of the difference (ca. 10/1000 = 1%)
may look small. The rule for conversions is to use as many
figures as you have available in the conversion factor, and
round only the end result, never the conversion factor. The
last example is also wrong because the boss is using
different numbers of significant figures for the two ends
of the range, a definite no-no. Worse yet, the difference
between the boss' value and the actual value is now closing
on 2%, which is beginning to get serious.

<<A lot of our literature expresses flow rates in gallons
per minute...commonly without using a decimal. So do I just
round up the liters?>>

Depends. The most important point in any conversion
(ignoring the pure mathematics for a moment) is the
magnitude of the smallest difference that is significant
for your users. If they won't notice differences of +/- 1
L, then you can safely round to the nearest liter. If, on
the other hand, a 0.01 L difference is important, than you
have to provide values accurate to the nearest 0.01 L. You
also have to consider the cumulative effect of the
difference. Think of a basement pump in flood-prone areas:
if the pump puts out 0.3 gpm less than the rate of incoming
water, that's 18 gallons per hour... so you certainly
couldn't round to "0 gpm", even though strict mathematics
would permit this. This is the sort of information you can
put in your style guide, but you've got to understand your
audience's needs.

<<I have two separate flow rates listed in two different
areas of a performance data sheet. One flow rate is 0.6
gpm, the other is 1 gpm.>>

This is _highly_ questionable if both describe the same
process. The difference between the two values would then
be 66% (1.0-0.6, all divided by 0.6), and unless your
equipment is very low precision, a difference this large is
unacceptable. But again, if nobody will notice a difference
of +/- 1 gpm, then you could probably round to the nearest
whole gallon. If not, don't round.

<<I may be making a mountain of a molehill here...>>

Quite on the contrary. Your boss is making a mountain into
a molehill, with potentially disastrous consequences for
anyone who tries to fly a plane over the ostensible
"molehill". If you need some ammunition for correct use of
significant digits, check any good statistics textbook
under "significant figures". But remember, there are always
two kinds of significant figure: the mathematical ones, and
the ones that are significant to your audience. Opt for the
latter if you have a choice.

--Geoff Hart @8^{)} geoff-h -at- mtl -dot- feric -dot- ca
Disclaimer: Speaking for myself, not FERIC.

TECHWR-L (Technical Communication) List Information: To send a message
to 2500+ readers, e-mail to TECHWR-L -at- LISTSERV -dot- OKSTATE -dot- EDU -dot- Send commands
Search the archives at or search and
browse the archives at

Previous by Author: Gantt charts?
Next by Author: Troubleshooting guide
Previous by Thread: Gantt charts?
Next by Thread: Re: Who has used FrameMaker or Frame+SGML to do single-source docs?

What this post helpful? Share it with friends and colleagues:

Sponsored Ads