Measure for Treasure, Not Pleasure

Published on Wednesday, April 08, 2009
ARTICLES

As long as there have been public policy think tanks, there have been attempts to measure their performance. It's a lot easier said than done, in my experience. Because we are nonprofits, the usual business metrics don't quite apply. And as players in the political debate, we are trying to affect outcomes that depend not just on sound policy ideas, communicated well, but on factors such as candidate recruitment, legislative politics, media bias and national political trends over which we have little if any control.

During a recent strategic-planning process, my staff debated a wide variety of potential measures. We sought to identify a Core Score, an easy-to-track statistic or index to guide our organizational decisions. It was quite a challenge. Some wanted to measure the ultimate outcome: Was liberty advancing or retreating in North Carolina? Others thought we should measure things clearly under our control, such as research output or staff morale. Still others thought intermediate outcomes were the most important, measures such as audience size - circulation, viewership, web traffic - and audience perceptions of our credibility and influence.

In the end, we zeroed in on peer-to-peer measures. At least once a quarter, and more often in some cases, we use external news archives and web trackers to compare our audience impact to those of other left, right and center public-policy organizations in North Carolina, as well as to SPN members and other national groups. For media mentions, we primarily use

http://www.newslibrary.com/, along with data from Google News and BurrellesLuce, and focus mostly on total appearances and total audience (adjusted for the size of the market, in this case by state population). For external web tracking, we primarily use http://www.alexa.com/, along with supplemental data from http://www.trafficranking.com/ and Google Blogs, and focus mostly on page counts.

In addition, at least once a year, we conduct surveys of JLF's Primary Audience, the roughly 11,000 North Carolinians whose work brings them into regular contact with the legislative process, as well as donors, event attendees, and readers of our newspaper, Carolina Journal. Together, these make up a significant share of the 800,000 politically active North Carolinians, who form our Secondary Audience. These surveys are conducted by mail, email and online response. Among other things, we ask respondents to compare the reliability of the John Locke Foundation to that of other sources of information about state and local government.

Our goal is not to construct new sales pitches. We are measuring for treasure, not pleasure. We want the data to lead us to innovative, effective programs we can steal - I mean, copy respectfully - from other policy groups. And when the data show JLF at or near the top in a given category, that tells us we may have some valuable advice to share with our friends, and withhold from our ideological rivals.

We collect lots of other performance data, too, for the usual reasons: managing and evaluating employees, attracting new human and financial resources, and allocating our resources to programs with the highest rate of return. But we think the peer-group measures are critically important.

Some of the measurement tools we use are, admittedly, crude. That's why I would like to see SPN expand its efforts to collect internal performance data from its members. As long as the information is handled carefully and used properly, I believe it can be a powerful tool to inspire think tank managers, shape SPN programming, and build the professional competence and reputation of our movement.

John Hood is chairman and president of the John Locke Foundation. Write him at jhood@johnlocke.org.