Measuring Quality in Technical Publications

by Robert DeFord

Any writer who has worked (or wants to work) on technical publishing projects should know that "quality" is always the big issue. Of course … quality is what counts, right? However, the frustrating truth of the matter is that everyone has a different way of judging quality. If you have read Lao-tzu’s Tao Te Ching, which attempts to explain quality, you may realize that defining true quality would be equivalent to discerning the nature of God.

Customers like to talk about the high quality of their goods and services, but when asked how they measure that quality, few can provide a satisfactory answer. Measuring the quality of even a simple object or tool is hard; getting agreement on validity of that measurement is even harder. Measuring the quality of something as complex and variable as a technical document is thought by many to be nearly impossible. They are wrong. It is possible to measure the quality of technical publications, but it requires a systematic approach based on a definition of quality that enables a method of measurement.

Webster’s Dictionary defines quality as the "degree of excellence that a thing possesses." This common definition is of little use for devising a method to measure quality. Any attempt to base a quality measurement process on this definition will soon flounder on the fact that measuring the degree of excellence something possesses is just as difficult as measuring how much quality it has.

The industrial quality control experts like to define quality as the degree to which a product matches its specification. At first glance, this definition seems to be superior to Webster’s. After all, many quality control systems rely on inspecting samples of the product to see how closely they meet specifications. However, everyone has purchased poor quality products that met the manufacturer’s specifications. Therefore, it is safe to assume that evaluating quality on the basis of this definition would not work well.

Rafael Aguayo, in a book examining the methods and practices of Dr. Deming, provides a definition of quality that readily leads to an effective process for its measurement:

Quality defined from the user’s point of view is anything that enhances satisfaction.1

Based on Aguayo’s definition of quality, a process for measuring quality in technical publications is straightforward. First, you identify the characteristics that lead to user satisfaction with technical publications and weight them according to their relative importance to the user. Second, you rate the publication on how well it embodies those characteristics. Of course, the validity of your quality measurement depends on how accurately you identify and weight the characteristics that lead to user satisfaction.

There are two good approaches to identifying and weighting these characteristics:

1. The profound knowledge approach
2. The user survey approach

The profound knowledge approach requires that the people identifying the characteristics be aware of what would satisfy the most demanding and knowledgeable users of the product. For example, suppose the task was to evaluate the quality of wine. If the people trying to identify the characteristics of fine wine drank very little wine, or knew little about good wine, they would not do very well at the task. They would either list the wrong characteristics, or assign the wrong weights to the characteristics.

At the end of this article is a technical publication scorecard. The Write Stuff developed this scorecard to measure the quality of its publications. We based the selection and weights of the characteristics on many years of experience and a great deal of customer feedback. You can use this scorecard to evaluate just about any document, or to establish customer criteria before you start writing (which will help avoid problems later). In addition, you can use it as a marketing tool and send it out to your end users or customers if you want them to evaluate your approach to technical publications.

While the profound knowledge approach works well for some people, you may not have the profound knowledge of technical publication required to create your own scorecard. In this case, you can employ the user survey approach to identify and weight the characteristics. While it requires access to a willing group of users who will take the time to complete a survey and return the information to you, it can give excellent results. It is best to start with a focus group of typical, knowledgeable users or customers. Ask this group to generate an initial set of characteristics that would enhance their satisfaction with technical documentation. Then ask them to weight the characteristics according to how important each characteristic is to them. You will find that working with a small group that can arrive at a consensus is faster than trying to obtain results from a survey of a large group.

After selecting an initial set of characteristics, you can send it to a larger survey group. Instruct this large group to evaluate the weighting of the characteristics and add any characteristics that do not appear in the initial set. Typically, you will not get a large number of additions and it will not take long to optimize your set.

Examine the final set of weighted characteristics carefully to determine which characteristics are the most important to users. It is easy to become trapped by the law of diminishing returns. For example, you could spend a lot of time working on design and layout issues when your users weighted these characteristics as far less important than clarity of writing and access to the information. In other words, to ensure a quick, useful measurement, it is best to eliminate characteristics that have a low weighting. Once the list has been pruned, it is a simple task to make a scorecard similar to the one below.

By using this scorecard to set the standards for quality, you will not only have an excellent tool by which to evaluate your technical publications work, but also a method of getting your customers and users on the same wavelength about the parameters of evaluation. This can minimize difficulties in customer relations, help the project run smoother, and make you look very attractive for future publications contracts.

Rafael Aguayo, Dr. Deming, The American Who Taught the Japanese About Quality (First Fireside Edition 1991)