What We Measure, Matters

Published on Wednesday, April 08, 2009
ARTICLES

Our actions are based on some theory of social change, whether we like it or not. If we publish research papers, it's because we have a theory about how insightful analysis influences policy. If we send emails urging recipients to rally against taxes, it's because we believe legislators fret over tax protestors. While we may boast about being practical-minded, at heart we are all theoreticians.

This is important when it comes to measurement, because just as we can neglect critical measures, we can obsess over minutia. In fact, precisely because the inconsequential is more easily measured, we risk drowning in data that don't bring us one whit closer to effective performance.

Take educational programs, for example. A measurement fetish might drive one to measure participants' feelings about every aspect of a seminar, their assessment of facilitators, immediate retention of key learning points, satisfaction with the venue, opinion of fellow participants, likelihood of using the ideas, enthusiasm for the audio-visual component versus written materials versus group discussion. You can measure the heck out of a seminar.

But absent sound theory, we'll measure everything in sight, and nothing that matters. Is our theory that participants possess qualities that make them likely to move into influential positions in academia and government? Do we believe what they've learned will affect their thinking years from now? Do we assume they'll be able to translate our ideas into practice? Measuring whether they liked the seminar, or even whether they immediately retained anything from it, won't get at the questions we really care about.

We might want to track program alumni to see if we've been selecting the right people. If we want to know whether our ideas have affected thinking and action, we need to survey not just immediately, but months later. It's messier, and more humbling, than a post-seminar feedback form, but more likely to provide a useful picture of program effectiveness.

So, back to theory. The longer the lag between action and outcome, the more precise must be our model of social change. Why, exactly, do we believe a newsletter or a website or a biker rally will yield good outcomes? In addition to sound theory, we should differentiate between three types of measures.

End goals. Outcomes like lower taxes, regulations whose benefits exceed costs and greater liberty are end goals. Since many efforts involve a lag between action and end goals, however, we have to rely on measures that correlate with end goal progress, namely proxy measures.

Proxy measures. By themselves these don't spell success, but they indicate whether our social change theory is operationally valid, and if we are making progress. Idea retention after an education program is not a valid end goal, but it can be one proxy. It's imperative that we not let proxies become our sole measure of success however. They are merely indicators that we are headed in the right direction.

Ancillary measures. These don't measure end-goal success, but they help us gauge organizational health, which is (we hope!) critical to end-goal success. Include here measures like fundraising, positive press mentions, retention of influential board members and so on. Again, we run the risk of being a stable but irrelevant organization if we let these indicators become our sole measures of success.

What we measure is what we'll do. This is why it's imperative to measure what matters, to measure as well as we can, and to remember that everything we are doing is based on some theory. Clarifying that theory can help us get a long way toward developing measures that matter.

Tony Woodlief is vice president of academic programs for the Market-Based Management Institute. Write him at tony.woodlief@mbminstitute.org. Woodlief will speak at the SPN Leadership Development Breakfast, April 23 in Los Angeles.