Last week, we released our Super Bowl advertisement findings on brand impact and awareness, using Civis Research, for three brands: Buick, TurboTax and Wix. Our research showed that established brands can build buzz and interest, while up-and-coming brands can swell their awareness. All in all, good news for Super Bowl advertisers.

While we found that advertising during the Super Bowl improves your brand awareness, we took this one step further by directly measuring the effectiveness of the ads. Which ads persuade consumers to try your brand? Do any ads deter people from wanting to use your products? It turns out — the message matters.

Civis clients use a scientific approach to survey research to find out which ads are best, and for which targets, before an ad is even in market. We gave a preview of this test with our Buick findings on Entrepreneur.com, where Falcons fans responded well to Atlanta-native Cam Newton’s ad, but Patriots fans did not. Hopefully Buick was targeting the Atlanta market and not Boston!

So what did we find for TurboTax and Wix?

TurboTax: Humpty Dumpty and The Realities of the American Health Care System
While our brand health analysis referenced this ad had a positive impact on buzz, this analysis adds that it likely had a negative impact on consumers’ interest in using TurboTax. Featuring a worse-for-wear Humpty Dumpty using the TurboTax app to check for medical bill deductions after his famous fall, people who were shown the ad expressed less interest in using TurboTax — a 63% chance of backlash, to be specific.

We also broke it down by Patriots and Falcons fans — while Patriots fans started out more interested in TurboTax overall, they experienced a far greater chance of backlash (97.8%!). While it’s not possible to know the exact cause from this testing, hopefully it’s not that the medical theme hit too close to home; one of their players, Dion Lewis, got injured during the big game, maybe a deja vu to last year’s season-ending ACL tear.

Wix: Food Trucks Are Less Flammable Than Restaurants
The Wix ad was one of the flashier of the year, complete with explosions, action movie stars, and a chandelier swing. Gal Gadot and Jason Statham laid waste to a restaurant, prompting the owner to start his own (presumably more defensible) food truck using a Wix website.

What’s most interesting are the big differences in how this ad performed across subgroups:

• It resonated best among young people, ages 18-34, while having a significant likelihood of negative backlash among older people.
• It performed best among the 38% of people who weren’t supporting either team in the Super Bowl — perhaps a group more likely to be “watching for the ads”.
• It also performed well among people who did not watch the Super Bowl at all, a group that has a strong overlap with young people.

So how did we come to these conclusions?

With data science, of course! More specifically, immediately following the game we exposed survey respondents on our weekly tracker to these ads in a controlled environment.

Civis’s experimental design enables unbiased and detailed results. We estimate people’s predicted responses, that is, how they would respond without having seen the ad. Then we measure the effect of the ad, based on the opinions of those who saw the ad. Given our approach, we’re able to test for backlash to ensure the ads don’t *hurt* a brand or have some other negative consequence. We can also simulate the effects of an ad on a particular subgroup based on demographics, such as whether an ad is more effective for different ages or geographies, as we saw for Wix.

While this analysis was done after the Super Bowl ads aired, testing messages in advance can help you find the most effective message for an appropriate audience. If your messaging doesn’t sit right with the audience you are intending to reach, the investment might not pay off.