The final countdown: evaluation and results

by Alicia Patterson

All PR practitioners need to produce results. In fact, some would say that PR practitioners especially need to produce results -- because many organisations discount the value of public relations and PR professionals.

In the past, this doubt was fostered by a profession that did not appreciate outcomes; it was focussed on process. There was a time when getting a media release out, writing a brochure and organising an event was enough. Now organisations want to see outcomes - they want to know that these activities are not done simply for the sake of them. Quite rightly, they want to see that these activities advance the organisational objectives or contribute to a broader agenda.

Public relations has matured in the sense that as a profession, we have become better at meeting this challenge and understanding that our programs need to be evaluated on TWO levels.

Firstly, and most obviously, a program must be evaluated against the original objectives. Did you achieve what you set out to achieve? Did you get the results you needed? Why or why not? Considering these questions makes it obvious that the ability to set realistic, timely and measurable objectives will give you direction in the evaluation of your program. Smart strategists match their evaluation mechanisms directly against their objectives.

The second level of evaluation of programs is to evaluate it against the broader or organisational objectives. You should use the opportunity, in the final analysis of your project, to consider the wider picture and setting in which it took place, and provide at least an informal evaluation of how it assisted the broader or organisational agenda.

Your first level of evaluation will demonstrate how well your program did on a technical and practical level, the second level demonstrates how it contributes on a ‘macro’ level -- outside the scope of just the public relations objectives or PR department’s professional performance.

This article concentrates on the first level of evaluation -- although hopefully the results on this level can clearly indicate how your program has been effective on the second level.

As with setting objectives, you are in control of how your program will be evaluated and judged. You can set the mechanisms in place -- just as you have set the benchmarks in the objectives. If it is not necessary (or reasonable) to have an objective that anticipates, for example, an increase in attendance numbers at a particular venue, then you will not select attendance numbers as an evaluation mechanism. You may instead select an evaluation method that concentrates on what people who attended thought about the venue, or how they have altered their use of the venue.

Essentially, there are two types of evaluation: quantitative and qualitative. Those of you who are scientists and accountants at heart will prefer quantitative methods which rely on the counting of numbers and production of figures -- percentage changes, mass and volume. An example of this would be a program that aimed to increase the numbers of children being immunised. Among the methods adopted in the strategy, one strategy may include a 1-800 phone number that is advertised in promotional literature and a television advertisement. It would be relevant and credible for you to build in the number of calls to the 1-800 number over the period of the campaign as an evaluation mechanism. Even more precisely, you may record (by asking callers) where they saw or heard the 1-800 number. In this way you will be able to evaluate which mechanism (the promotional literature or the television advertisement) was the most effective.

Quantitative methods include things like:

• telephone surveys
• door-to-door surveys
• counting incoming phone calls / attendance numbers
• sales figures
• column centimetres, or number of articles, published as a result of a media release.

They are methods that will provide you with a number. This is currency that most of us are used to seeing and it must be acknowledged that there is a certain bias to evaluation mechanisms that can produce these kinds of numbers. One word of caution: it is important that you investigate things like sample sizes when conducting this kind of evaluation. It is not credible, for example, to survey 25 people about their religious beliefs and extrapolate that to a whole city, state or nation. In might be credible if this sample of 25 was out of a population of 100 members of a particular social club.

Quantitative methods have their flaws. What if the telephone survey is done at the wrong time or place -- where your program has not been in effect? What if, because of the issue, respondents do not tell the whole truth? (A classic example of this is quantitative research done about sexual activity -- men overestimate and women underestimate because both are conscious of certain social expectations and taboos). How can you show how much of the increase in sales figures is related to your public relations activity and not something else (like a change in stock location in a store or advertising)?

One of the simplest and easiest ways to produce impressive quantitative results is to focus your evaluation not on program outcomes, but on evaluating the processes or program elements. You could, for example, say that your program has been successful because you printed and distributed 100,000 brochures to the public about immunisation. A smart PR professional would inspect this fairly closely -- and be ready for others to do so too. 100,000 brochures in circulation sounds great -- but were they dropped in letter boxes with other junk mail or were they distributed as a result of specific demand from the public?

Qualitative methods will appeal to those of you who feel that a number cannot tell you the ‘whole truth’ about the reach or effectiveness or QUALITY of your program. You may have generated 100 press clippings through two media releases, but what does this number tell you about how accurate those reports were -- or whether they positively reported your information at all? What about where the articles appeared -- was it in the back pages of the newspaper or on page one? A telephone survey may reveal that you have increased awareness about the importance of childhood immunisation against rubella from 30% to 80% but what depth is that awareness? Is it just an acknowledgement that the respondent saw the TV ad that was part of the program, or does it signify that they read a brochure and know about the issue in more detail?

Qualitative methods include things like:

• focus groups
• key respondent interviews
• analysis of media coverage audits.

Focus groups mean you can recruit people from your target audience and find out, through discussion whether your program reached them, and what impact it had, It gives you the chance to see the nuances in their reactions and closely investigate what they say and why they say it. Key respondents take you one step away from the bottom line and gathers the views from opinion leaders, in touch with your target audience, and finds out whether they believe it had an impact on their constituents.

The analysis of media activity and spots mean you can look at what the media did with your launch or media release. You can see if they printed your media release but totally reinterpreted all the facts in it to the disadvantage of your campaign; you can analyse whether the spot on the nightly television news bulletin was really any use if your target audience was 14-year-old boys (although it will certainly make your boss or client feel GREAT about your work).

It almost goes without saying that your evaluation mechanisms need to reflect the methods you adopted in your strategy. If you select media activity, then you need to build in some evaluation of this method against your objectives; you provide an analysis of the coverage and then you cross-reference this with (perhaps) four focus groups with your target audience. Through your target audience discussions, you find out that while they were aware of the media coverage the issue got it was the comments their doctor made to them that really made them take notice or change their attitude. This tells you that the strategy to target doctors with information about the issue (via articles in professional journals and a seminar series) really seemed to work!

You don’t need to go overboard in your evaluation but you will need to build in ways to find out ‘how you did’ in your program. You might ask yourself:

• How will I measure how I met my objectives?
• How will I tell if my program is successful?
• What will others in my organisation think is a fair way to evaluate how well the program worked?
• How well did each component of my program work? (This is process evaluation and more for your benefit than others.)
• What would I change if I did it again and why?

Being able to answer these questions will give you a great start to evaluating your program -- and future ones!

--AP
©1999 Alicia Patterson