The Year of Data Visualization and Reporting

On the plane, heading back from the American Evaluation Association’s annual conference in Anaheim. Long plane rides such a great opportunity for reflection. What’s on my mind? The overwhelming success of the Data Visualization and Reporting Topical Interest Group. We had so much support in getting this group launched and we were so embraced by the rest of the conference attendees. Highlights:

1. Slide Clinic – The night before the regular sessions began, we held an open clinic where attendees could bring their conference session slideshows for some quick diagnosis and triage. I recall a few years back, when I sat through a workshop given by someone who attended the clinic. Helping her choose legible fonts – improving her communication of her otherwise very insightful charts – was damned rewarding.

2. Ignite the Night – We held our required annual business meeting as an Ignite session. You know about Ignite, yes? Fast-paced, 5-minute talks where the slides auto-advance every 15 seconds, whether or not the speaker is ready. Never has that much fun been had without alcohol at a TIG business meeting, I’m sure of it. Video will be posted soon.

3. Data Visualization and Reporting-sponsored Conference Sessions – Audience size at our sponsored sessions was a clear indication that evaluators are becoming increasingly interested in good communication and reporting. We had the good problem of overcrowded rooms at all of our sessions – beyond standing room only, sitting on any open floor spot, spilling out into the hallway. For our first presence at the conference, we sure made ourselves known.

As founding chair of the TIG, I stepped aside after our business meeting, turning things over to the new chair, Amy Germuth. This year’s total rock star debut will keep feeding my soul until next year, people, in Minneapolis.

Advertisements

WTF HCZ?

We are big fans of the Harlem Children’s Zone around these parts. Like many others, I’m sure part of it is romantic – we aspire to see a social program work as well as HCZ appears because we love Harlem and we love magic bullets. If it works there, it can work here, too, right? Right??? If you’ve been following the developments lately, you’re as sad as me about the exposure of evaluation weaknesses and the implications of such nudity.

Not so long ago, HCZ was criticized for not truly achieving at the rate of some of its claims. The criticism came from the Brookings Institution (see the report) and was authored by Russ Whitehurst (who is well-known in evaluation as the head of the Institute of Education Sciences under Bush, where many believe the lunge toward randomized trials was re-institutionalized, but let’s not reopen that debate here). Suffice it to say, right or wrong, within a week of the Brookings Institution’s public criticism, we see that Congress is moving to cut the proposed Promise Neighborhoods, a proposal that had been based entirely on the success claimed by HCZ. We don’t know if the congressional move is a result of the influential Brookings report, but it seems highly coincidental and noteworthy, no?

Also highly coincidental and noteworthy is the recent job posting on the American Evaluation Association’s Career Center webpage for, you guessed it, evaluation help at HCZ. The job was posted exactly one week before the initial Brookings criticism. So what? So it is possible, if speculative, the evaluation weakness is at play here, folks. My impression, wherever I got it, was that HCZ had evaluation on its radar. But whether it is evaluation instability or a growing need for more evaluation staff, it is clear there is a gap around evaluation. At the very least, more public, independent publishing of evaluation reports from HCZ could have helped stave off the criticism. The website has a good deal about evaluation – but it is limited to a discussion of its commitment to evaluation and the evaluation of nonHCZ programs. Show me the iron-clad numbers! Or at least the well-documented success stories! If its there, it isn’t obvious.

Without published reports from credible sources, HCZ has little to stand on when defending itself. (And people ask me why external evaluators are needed…). It is put in the defensive position, instead of an offensive, reputable analysis of its own situation – good, bad, and ugly. Oh, HCZ. You have my admiration, but you need my field’s assistance.

Why Proposals Fail

Summer, as you  know, is proposal season. I’ve been up to my neck (literally – these proposals are huge) in stacks of papers, reviewing ideas seeking support from various federal agencies. Regardless of the agency, some proposals seem to fare less well for common reasons. Here’s my breakdown (and strictly mine – the weaknesses I identified were not always a shared concern with my other panelists) of why proposals fail, in no particular order:

1. The evaluation plans don’t clearly match the project’s goals and objectives. If the project is seeking to change the consumer experience but the evaluation is only looking at production of the consumer good, it will never be able to tell whether the project has met its goal. This could mean a review of the evaluation plans OR a revision to the project’s goals and objectives.

2. The evaluation is not evaluative. No targets or performance standards are set. The way the evaluation is structured will only enable them, in the end, to say descriptive things about what the project did – not how good or worthwhile it was.

3. Experimental designs typically, and surprisingly, lacked a power analysis to determine whether the project’s recruitment efforts are on track. In the era of accountability – and at a time when technology allows us to see ahead of time where we should focus our efforts – there is no excuse for a missing power analysis, at least in those designs where it is called for.

4. Letters of support were clearly written by project staff and cut-and-pasted by the supporters. Letter content was identical, save for the letterhead and signature line. I know it is unreasonable to request others to draft letters of support originally, in most cases. However, the letters I saw frequently left out key responsibilities of the supporting organizations. For  example, if the school district will need to commit to providing control condition classrooms, where no benefit to participation will be derived, that needs to be clearly agreed to up front. The danger is looking like your evaluation isn’t well-planned and hasn’t been throughly communicated to all parties.

5. The evaluation organization appears to have the collective experience necessary, but the specific individuals assigned in the proposal have no direct relevant experience in the tasks on the table. Too much narrative space is spent defending the established history of Evaluation Consultants, LLC, particularly when the actual evaluation staff bio- buried in the appendices – is weeeeeeeak.

6. It pains me to even have to write this one – but sometimes I saw proposals that did not yet have an evaluator identified. Sheesh! It is okay, Principal Investigators, to contact an evaluation team during proposal development and ask them to help you draft your evaluation plan. They will probably even help you write the evaluation section of  the proposal. You might want to draft up an memorandum of understanding that ensures they will be selected as your evaluator, should the award be granted. In the evaluation business, most of us are used to devoting a little free time to writing up plans that are (or sometimes aren’t) funded in the future. It is part of our work and it is okay for you to start talking to your evaluator the moment you start thinking about your program. In fact, it is highly encouraged. What? You don’t have one now? Go forth!

Okay, there is it – the top six reasons I saw proposals fail this summer. I’m hoping next year it will be a totally different bag. Did you see something different? Post it in the comments!

Oil In My Backyard

Right now, the midwest’s largest oil spill in history is flowing through my backyard. The pipeline, taking oil from Indiana to Canada, burst sometime Sunday or Monday this past week (3-4 days ago), sending 840,000 barrels of oil into a creek, that flows to the Kalamazoo River, that flows to Lake Michigan. As of this writing, the oil has been spotted just past at a nearby dam, right outside of Kalamazoo, where state workers are doing their best to clean it up before it fills my town and heads to the lake.

On the heels of the disaster in the Gulf, the community is hyper angry and action-oriented. Their questions are these:

1. Do we have the resources to stop the spill before it reaches the Great Lakes?

2. How much oil are we talking about here?

and

3. How in the hell did this happen?

Due to my disposition, I immediately saw these as the evaluation questions. In fact, these seem to be the most common evaluation questions of all time: What was the impact? To what extent? and What was the cause?

In the case here,

1. Yes, we have tons of resources. The oil spill hotline has turned down volunteers, saying they have had an overwhelming number of calls. We can stop this thing, if they’ll let us (outcome). But the leader of the outcome is the very company that owns the pipeline and, like BP, they are keeping others at bay (unexpected consequence).

2. New estimates from the EPA raise the total to 1 million gallons (output). Enough to fill a football field two feet deep and then a lot more. This illustrative description comes courtesy of the Freep and demonstrates another skill needed by evaluators – to describe the extent of the impact in such a way that it is understandable to a wide audience.

Freep photographer Andre J. Jackson also snapped this picture, a necessary visual of the impact at the riverside:

Canada Geese covered in oil sit along the Kalamazoo River after a pipeline ruptured in Marshall on Tuesday.   (ANDRE J. JACKSON/Detroit Free Press)

3. The cause? Enbridge Energy, whose PR-controlled Wikipedia page puts the spill at 19,500 barrels (or 20% 2% of the EPA’s estimate). Well, they are the guilty party, maybe not the cause per se. The cause is really their shoddy internal evaluation. According to the aforementioned wiki page, they have had 610 spills in the last 12 years. 610! If that sort of error rate was allowed in schools or social service organizations, they’d be run out of town. No good internal quality control can allow an average of 51 spills per year.

Look, I don’t pretend to know how to evaluate disaster response. That’s my friend, Liesel Ritchie. But what I do know is that cause-probing is clearly a natural phenomenon because there are a lot of Kalamazooans who want to hold Enbridge’s feet to the fire. And I know that there is a time when one should go native – I’ll see you at the river.

You Can’t Always Get What You Want

Reading a decent book, came across a great point:

The book is Start With Why and it really labors on the fact that great businesses lead with their WHY right out front, not their WHAT (or their HOW). Okay, but how does this relate to evaluation?

The author discusses how laundry detergent brands had forever been promoting how their formulas got clothes “whiter than white.” They had, smartly, conducted focus groups and asked people what they wanted out of a great laundry detergent and “whiter than white” was the answer. But when they rolled out that marketing campaign, competing with one another over which was whiter (that sounds weird), it didn’t have much of an effect on the consumer. Good thing they brought back in the scientists, anthropologists to be exact, who studied people washing their clothes. That’s when they discovered – a ha! – the first thing people do when they pull a load from the wash is to smell it. Yep. The fresh smell was more important than the level of whiteness. (Now you know why that aisle is so ridiculously scented at the supermarket, with dozens of fragrance variation.)

Back to evaluation: Focus groups are so often to go-to resource for needs assessments. Close seconds might be surveys or interviews, but these are other forms of self-report, where we are asking people directly about their needs. But those end up actually being their wants, more often than not. As Jane Davidson calls it, unconscious needs are really what we are after when we are designing programs and interventions. Those unconscious needs are the ones people are less likely to be able to articulate, simply because we humans are often lacking self-awareness. Perhaps, like the anthropologists witnessing laundry day, we should be observing a great deal more than we are asking.

If It Ain’t Broke

I found the cutest old-man optometrist. He puttered around the room, in cute old man fashion. He had a little cute old man mantra: “if it ain’t broke…”

Him: Are your contacts working okay for you?

Me: Sure, I guess.

Him: Well, if it ain’t broke…

Me: But aren’t you going to check my eyes???

He eventually did. But he must have repeated his mantra three or four more times during our appointment together.

It was while I was waiting for my eyes to dilate that I realized how “if it ain’t broke…” might be the worst phrase for an evaluator to hear. Why wait until things are broken to start fixing them? Waiting until things are broke means enduring a period of decline, a period of broken-ness, and a period of rebuilding to get things back up at the same operating level as previously. That sort of downtime impacts an organization’s productivity, effectiveness, and bottom line. When there are clear patterns and signposts established (especially in the eyecare industry), it would be much more efficient to watch out for those early warning signals and take action, rather than wait until it is broke. This is why evaluators are good at pattern recognition.

Now whenever I hear “if it ain’t broke…,” I cringe. Must be hard to examine my eyes that way.

Don’t Even Try

I love being on the other side. I am in the midst of reviewing evaluator letters of interest – miniproposals – to evaluate one of my work projects. Rarely am I in the position to need the evaluator. Usually I am the one submitting my ideas and credentials. The pile sitting in front of me holds an incredible range of quality. For some, I am honored that they would be interested in working with us. For others, I am reminded of a mistake I made early on in my professional evaluation career.

I was hired on to a grant, which had proposed to evaluate a community initiative, after the proposal was accepted and funding had landed. My team was geeked, particularly because the local community initiative had been so successful, other cities were adopting the model. We saw this rapid replication as an opportunity – perhaps even as a meat market. Hmmmm, which one of these pretties shall we go after? We, naturally, went for the largest, the richest, the most popular options and courted those community leaders around the country. We submitted evaluation proposals to them that were all basically the same, with selected search-and-replacing. At the time, I had never actually written an evaluation proposal and I use my naivete as an excuse, thankyouverymuch.

When the first rejection letter was returned to us, I was devastated (I mean, I cried. First rejection.) It was from Denver. And their chief complaint was that the proposal didn’t reflect an understanding of the Denver context. We had talked about this particular community initiative being so necessary because the larger community of Fill-In-The-Blank was a waning industrial center that needed revitalization. Hello? Been to Denver lately? That’s not them at all. They were right to reject us. We should have done more homework before submitting that proposal.

The same mistakes are sitting in front of me: boilerplate language that shows no evidence of even trying to understand who we are and what we do. While this might seem like an easy strategy (and who knows, one of the 400 letters sent out might actually land a job…), one shouldn’t be a surprised by rejection. Just like the guy who sidles up to me at the bar, I am thinking in my head, “don’t even try.”

Nix the Table of Contents

If the evaluation report is so long it needs a table of contents, you know you have gone too far.

I have been researching the communication of evaluation findings in preparation for an upcoming webinar on the topic and because I have a horse I’m currently riding called How Not to Annoy People with Evaluation. Experts in the field rarely mention much on communication of findings. Those who do give a decent turn to getting the right stakeholders at the table, even thinking about different ways to display findings. But invariably, the evaluation seems to produce a written report. Many evaluation budgets aren’t large enough to rework the written tome into brochures, newsletters, and interpretive dance routines to cater the findings to different audiences. We’re often stuck with the written report.

So then why do we torture the readers with dozens of pages of inane technical information before getting to the findings? (Rhetorical. I think I have an answer for another blog post.)

Reports 200 pages in length are not useful. Plain and simple. The narrative and graphics must be concise and to the point. I was sitting in a meeting at a local foundation about two weeks ago, with two foundation folks in the room, representing different institutions. They were lamenting, as we all do, about not having enough time to fully catch up on every activity of their grantees. They pinpointed annual reports, saying even executive summaries can be too long (and I read a recent “expert” in evaluation advise an executive summary 4 to 20 pages in length!) and then they begged to the ether, “bullet points! Please bullet points!”

To make evaluation useful, we must stop producing documents that better serve as doorstops. One good sign: if you have to create a table of contents, you have too many pages.