When Not to Use Data Driven Decision Making

In 1993 two artists set out to make a data-driven painting. I’m not kidding. They contracted with a well respected research firm, ran a call center in the midwest, and used telephone interviews to gather opinions about the type of artistic content and composition people preferred.

The poll used a stratified (by state) random sample to snag 1,001 responses, which had nearly perfect distributions in terms of income, gender, race, education, and political beliefs. Evaluators would be hard pressed to obtain the same representativeness in most of our work.

The artists calculated simple descriptive statistics and used the analysis to create a painting which reflected the desired artwork of the majority. The result is the painting below, America’s Most Wanted. Imagine it dishwasher-sized. And yes, that’s George Washington and a random group of anonymous people in the picture.

The artists debuted the painting in Soho, published it in magazines and paraded it around the country in town hall meetings. The main sense of those in the crowd, upon digesting their data-driven art, was… embarrassment. The painting was quaint. Provincial. Even ugly. One of the artists, Komar, said, “It is my hope that people who come to see our Most Wanted paintings will become so horrified that their tastes will gradually change, and another poll would gather different results for creation of paintings that do not resemble blue landscapes.”

Now this poll was examining tastes, but the same problem occurs all over the place when we have the good intention to making data-driven decisions. Henry Ford said, “If I had asked people what they wanted, they would have said faster horses.” In evaluation we often run into this predicament with needs assessments. It isn’t as simple as asking people what they need. Some evaluators have wised up to this idea, so they observe behavior instead and surmise needs from their observations. But what would the artists have observed in this study? About 50 percent of the respondents said they visited an art museum never or less than once per year. Not much to observe. Sometimes the smartest decisions come from an innovator who knows the audience base well, not from the audience itself.

By the way, the painting below is a representation of all of the worst rated features, America’s Least Wanted.

The poll was replicated in several other countries around the globe, with nearly similar results. See the resulting paintings for other locations and explore the data at http://awp.diaart.org/km/


You Can’t Always Get What You Want

Reading a decent book, came across a great point:

The book is Start With Why and it really labors on the fact that great businesses lead with their WHY right out front, not their WHAT (or their HOW). Okay, but how does this relate to evaluation?

The author discusses how laundry detergent brands had forever been promoting how their formulas got clothes “whiter than white.” They had, smartly, conducted focus groups and asked people what they wanted out of a great laundry detergent and “whiter than white” was the answer. But when they rolled out that marketing campaign, competing with one another over which was whiter (that sounds weird), it didn’t have much of an effect on the consumer. Good thing they brought back in the scientists, anthropologists to be exact, who studied people washing their clothes. That’s when they discovered – a ha! – the first thing people do when they pull a load from the wash is to smell it. Yep. The fresh smell was more important than the level of whiteness. (Now you know why that aisle is so ridiculously scented at the supermarket, with dozens of fragrance variation.)

Back to evaluation: Focus groups are so often to go-to resource for needs assessments. Close seconds might be surveys or interviews, but these are other forms of self-report, where we are asking people directly about their needs. But those end up actually being their wants, more often than not. As Jane Davidson calls it, unconscious needs are really what we are after when we are designing programs and interventions. Those unconscious needs are the ones people are less likely to be able to articulate, simply because we humans are often lacking self-awareness. Perhaps, like the anthropologists witnessing laundry day, we should be observing a great deal more than we are asking.