Beware the Case Study

I really really like case studies. In general, when I attend a conference or industry event, the presentations that I find most interesting and useful are the stories about "Here's what we were trying to do, here's what we did, here's why we chose this approach, and here's what happened."

But I've begun to get a bit more jaded about them lately.

At a recent Internet marketing event that I attended, for example, several case studies were presented. In every case, the brands saw amazing results. The solutions/campaigns/technology used were absolutely the best choices that the marketers could have selected.

Same thing at a recent advertising awards show. Of course, because it was a "creative" event, they didn't actually show any results generated by the winners, but the reels and screenshots and sample images, etc. certainly implied that the featured ads were the greatest pieces of design and copywriting ever beheld by mankind.

But some of us in the audiences at these events wondered aloud -- were these examples actually any good? Is there much we can learn from them?

A campaign by a major beer brand comes to mind. Their case study showed how they generated thousands of contest entries (and, we would assume, a whole pile of brand building value) for a particular promotion. Why was it successful? According to the case study, it had something to do with the amazingly-chosen media mix and the fancy new creative unit (ad size) used. But maybe, just maybe, it was successful because the brand spent millions of dollars promoting the contest. Maybe it had something to do with the prize being a dream vacation. Maybe the fact that the ads were basically just images of beautiful people in skimpy swimsuits had some influence on success. Or maybe the brand's history as the leader in market share for decades helped a little...

I would actually be more interested in a case study where a campaign like this failed miserably. If you have a multi-million dollar budget, few imagery restrictions, a product that's inherently linked to fun, and a well established brand, how on earth could you possibly screw up?

Unfortunately, case studies, like history, are written by the victors. There's not much reason for a marketing manager or ad agency to tell the world about their mistakes. Unless they're blaming someone else, of course.

One of my favourite marketing books is The End Of Marketing As We Know it by Sergio Zyman. Zyman is former CMO of Coca Cola and has been a consultant for numerous other huge consumer product brands. The main message of the book is pretty simple: marketing is meant to sell. One of things that makes the content so interesting, though, is that he spends more time talking about the failures than he does about the successes. It turns out, for example, that classic ad campaigns ("Mean Joe Greene", "Tastes Great, Less Filling", and many others) didn't actually do anything for the bottom line.

Case studies can be very dangerous, and I think it's all about context. If you're a marketer, you need to look very carefully at the similarities and differences between your brand and the examples. Don't assume that because X (seems to have) worked for those other guys it will automatically work for you too. Are their budgets similar? Is their background similar? Is their audience similar? Are the inherent product attributes even remotely similar?

It's pretty obvious to say "Your small brand can't count on the same success that a big brand saw" or "Your boring product can't count on the same success that a fun product saw", but it also works the other direction. Everybody loves the "Little brand that could" stories of some guy in his his mom's garage growing into a multi-national leader. But just as that little guy had to start with a different approach than his billion-dollar competitors, his billion-dollar competitors couldn't just look at this up-and-comer and follow his approach.

Case studies aren't useless, of course. Like I said, I really really like them. The key is to identify the similarities.
If I'm selling banking products, maybe I can relate to the fun-loving attributes of a beer campaign. Who doesn't love money, right?
If I'm selling high-end cosmetics, there's probably something to learn from the techniques of other ridiculously-high-margin products like pharmaceuticals...
A huge baby stroller producer should be able to use similar insights into the minds of new parents as a small, local daycare...