Scale Resolution vs. Readability

How many times should the smallest division of an analog scale be subdivided?

How many times can the smallest division on an analog scale be subdivided? What are the factors that affect this decision? Many people say an operator should only subdivide into 4 equal parts. Some OEM’s have claimed they can subdivide into 10 equal parts (and that is completely unacceptable for Transcat). We’ll use an example of a force gauge with a range of 0 to 250 lbf and a resolution of 2.5 lbf (smallest division) to explain Transcat’s policy.

The reason most Metrologist and instrument operators are ok with subdividing 4 equal parts is that the instrument pointer is being viewed between a solid line (on the left or right of the smallest division) and the imaginary line that lies halfway between these two solid lines. In other words, the operator has at least one anchor or reference point. Given the smallest division of 2.5 lbf, if this is subdivided into 4 equal parts, then ¼ of one division equates to 0.625 lbf. That doesn’t seem feasible because it implies the resolution of the instrument with an analog scale is good to 1/1,000th of one pound of force. However, if the smallest division is subdivided into 5 equal parts, then 1/5 of one division equates to 0.5 lbf, which is a more believable resolution to 1/10th of one pound of force. Comparing the sub-increments demonstrates the point in selecting how many times one should subdivide the smallest division.

Subdividing into 4 equal parts yields values of 0.625, 1.25, 1.875, and 2.5 which makes decimal values that imply impractical resolutions.

Subdividing into 5 equal parts yields values of 0.5, 1.0, 1.5, 2.0, and 2.5 which keeps the resolution to half and whole number increments?

Now consider what the operator must do to achieve this subdivision into 5 equal parts. If the pointer is on the lower solid line of the smallest division, there is little error in the reading; in other words, the variability or standard deviation between operators taking such a reading is expected to be relatively small. The same goes with the pointer lying on the upper line of the smallest division. The next 1/5 of one division reading would be when the pointer is just above the lower solid line (or just below the upper solid line). This is also repeatable between operators because it lies close to, but not directly on, one of the marked scale lines. It’s the middle two pointer indications that become tricky and may have more variability between operators which can cause differences of opinion as to whether subdividing into 5 equal parts is feasible. Now imagine the midpoint between the two marked scale lines of the smallest division. If the pointer were to lie just to the left or right of this imaginary center line, but not directly in between, it would be evident to most operators that the pointer is not in the center and is also not just above the lower line or just below the upper line.

Therefore, it is possible with minimal training to obtain a readability of an analog scale to 1/5 of the smallest division, given that the pointer is proportionate to the marked scale lines and not significantly larger than the scale lines. Also, mirrored scales aid in reducing variability by minimizing parallax error (alignment of the pointer to a mirror image of itself so that the operator views it from approximately a 90° angle every time).

Let’s modify the example slightly to understand the tipping point in deciding whether to subdivide 4 time or 5 times. Let’s consider what happens with a force gauge that has a range of 200 lbf and a resolution (smallest division) of 2 lbf. Dividing into 5 equal parts is possible and makes sense from a believable resolution point of view, as the smallest subdivision equates to 0.4 lbf. However, dividing into 4 equal parts for this value of the instrument’s smallest division equates to 0.5 lbf. This makes more sense than 0.4 lbf because values of 0.5 lbf keep the sub-increments to half and whole number values (i.e., 0.5, 1.0, 1.5, and 2.0) as opposed to uncommon values (i.e., 0.4, 0.8, 1.2, 1.6, and 2.0).

Basing the decision to subdivide upon two common goals makes for a consistent policy. These goals are:

1. Believable resolution as the result of the subdivision process.
2. Maintaining half and whole increments as opposed to uncommon increments.