As far as your wrong perception that I’ve somehow not alligned with my management, I’ll be writing on this blog in the near future about our fantastic new relationship with Compete.com and how our new offering fits in with our overall framework for ad effectiveness measurement. If you wish to reply to that blog posting, feel free. This one has run it’s course.

I believe I’ve stated my opinions very clearly. Your framework falls apart and, in fact, you’ve moved away from it in this blog. The framework I’ve stated stands up to the test of time and, in fact, works for any medium.

Ken

Sorry it’s taken me a while to post a reply. I’ve been engrossed with matters in the real world

Let’s stop pretending that a campaign can be reduced to a simple equation for every situation. You keep coming back to a single slide in our OPA presentation that said “search + site visitation + e-Commerce = a smart formula for measuring display ad effectiveness.” If one is measuring online impact, I contend that that’s a perfectly valid approach. I was not suggesting that one ignore offline sales and, in fact, we’ve been pioneers in measuring it.

It is interesting to me that you take issue with the use of trademark search and site visitation in measuring ad effectiveness. I suggest that your criticisms apply equally to brand perception, which you keep including as a critical evaluation metric. There is no justification for your claim that brand perception is the right thing to evaluate, but trademark search and site visitation are not. In fact, your own management team disagrees with you. Why, just the other day your company announced the following:

“On July 13 Compete announced the availability of its new online advertising effectiveness measurement capability which analyzes the impact of online advertising on consumers’ online behaviors such as engagement and conversion on brand sites, post-exposure search behavior and rival and third-party site visitation.”

http://www.dynamiclogic.com/na/pressroom/releases/?id=711

This is the same thing of which I’ve been trying to convince you for the better part of three weeks now, but still you insisted that : “post-view behavior shouldn’t be part of an effectiveness formula….search and visits to the advertiser site are not ultimate measures of ad effectiveness. In the Dynamic Logic framework – it’s all about perceptions and sales”.

Note especially the use of the term “ad effectiveness” in your own company’s release. Your management must have forgotten to let you know that they agree with me (and disagree with you) regarding what constitutes valid measures of ad effectiveness and that search and site visitation are indeed acceptable measures. I do notice that they didn’t bother to include a quote from you in their release. That should tell you something.

Your comments on cookie deletion are also rather uninformed. For example, it’s clear that you don’t understand comScore’s Brand Metrix methodology. We don’t – as you claim — survey our own panel in our methodology. And as far as cookie deletion is concerned, your methodology has no way of knowing that the person you define as a control respondent (who is supposed to have not seen a campaign ad at any time) isn’t just a computer whose ad cookies were deleted at some point in the past. That results in a contaminated control group and an understatement of ROI. We adjust for it. Dynamic Logic doesn’t.

I’m going to leave this discussion with a final suggestion. Before you publicly criticize your competitors (remember, you started this) and waste everyone’s time with your personal opinions, you should check-in with your superiors at DL and your compatriots at Compete to make sure you really understand your company’s official product strategy, positioning and philosophy. As Steve Martin once said to John Candy: “Next time you start a story, have a point.”

Gian

Assuming someone has bought into other measures of ad effectiveness, you’ve stated publicly that the best way to measure display = search + site visitation + e-commerce. I’ve said it’s about changes in perceptions + sales impact (online and offline). You eventually said that perceptions play a role diagnostically. I’ve said post-view behaviors are important diagnostically. I’ve made the point that if a campaign impacts sales but doesn’t affect online post-view behaviors, then it’s a success. Whereas the opposite is not true — campaigns can impact behaviors without being successful.

You are now saying we’ve failed to bring reach and frequency into the equation. If so, then we have both failed in that respect. It wasn’t part of your formula either. That being said, what you are talking about is an extension of ad effectiveness called impacted reach which is defined as reach times impact. Impact, as I’ve said before is changes in perceptions + sales. This is typically represented as a percentage — percent of people buying per million impressions, percent of people with changed perceptions per million impressions (etc.). So, impacted reach (reach times impact) is the number of people impacted by a campaign. We can calculate this for clients, not just in terms of perception changes, but also estimates for numbers of items sold (online and offline) due to the advertising.

I agree that on a campaign basis, one should examine impacted reach. We’ve had a tool built into our MarketNorms database (of over 5,000 campaigns) for the last several years, that allows one to calculate impacted reach and perform “what if” scenarios. Some times clients just take the ad effectiveness results and calculate the impacted reach themselves using their own reach and frequency data.

You are right that historically we’ve been the brand impact company. And, it’s true that historically comScore has been a traffic reporting company. You’ve moved into ad effectiveness where we’ve been for 10 years. We’ve come at this from different directions, but we now have competing solutions that can incorporate measures of perceptions, post-view behaviors and sales.

So, in the end, we are doing the same things — but we feel we are doing it just a bit better. And, we think your strong emphasis on online behavior (your entire formula for measurement included *only* online behavioral metrics), missed the mark.

* Perceptions. We track the entire campaign and survey people within the media footprint of the campaign. You take a needle-in-a-haystack approach, searching through the panel to find people exposed to the campaign, then seeing how many of those are willing to take a survey. The size of your panel limits the depth of insights you can provide and, in some cases, makes this measurement infeasible.

You’ve made a lot of the subject of cookie deletion, not just on this blog but elsewhere. What percentage of people do you think delete cookies between the time they see an ad and the time they take our survey? The percentage is small. We’ve done conservative simulations that show that the impact on our metrics is tiny and often smaller than the rounding error in the metrics (eg, an increase of 6.1 is measured to be 6.0 or 6.05). Since we are a media neutral company, it’s important for us to be conservative in these regards. We can provide clients with estimates that account for the very small bias that cookie deletion introduces but so far, after they are convinced how small it is, no one asks for it. They like the neutrality and are glad we don’t report misleading results.

* Post-view behaviors. Yes, you have more experience but we have partnered with Compete.com who is quite experienced and has a larger panel than yours. This means we can get more granular on certain more rare behaviors such as e-commerce. And, our sample sizes will be larger which measure greater statistical reliability.

* Sales. Your formula didn’t even include offline sales. You restricted things to online. We’ve developed a reliable and relatively inexpensive system for tracking through to offline sales and are about to announce extensions and enhancements to this.

A final word about media neutrality. Notice that our formula ad fx = changes in perceptions + sales, works for all media. People believed it this formula the internet became popular and it continues to be the best general way to assess advertising effectiveness.

Regarding a debate, I think a neutral environment rather than one of us being part of the others’ panel, would work best.

Best,
Ken

Let me summarize where we are.

In one of my previous posts, I have already made it clear that I agree that attitudes are important diagnostics. We have always included them in comScore’s Brand Metrix product, which also takes into account the deleterious impact of cookie deletion (which can lead to a serious understatement of ad impact). Clients tell me that Dynamic Logic doesn’t include any adjustments for cookie deletion in its products and so understates the impact of online ad campaigns.

I’m pleased to see that you now agree that sales, search and site visitation are also important diagnostics. DL has historically only provided attitudes, so it’s a step forward to hear you now say that actual behavior is also important. I do want to point out that these variables have always been included in comScore’s ad effectiveness measurement services.

However, there is one other extremely important dimension that comScore includes in its evaluation of the effectiveness of a media plan, and which you fail to mention. That is, what is the reach / frequency that was actually delivered in the media plan and what are the characteristics of the people who received the ad impressions? Cookie deletion is again the culprit here because it can cause the actual delivered media plan to be very different from what was planned to be delivered. comScore routinely provides clients with post-buy measurements and clients are telling me that DL doesn’t provide this capability. So in your example, I think you’re dead wrong. The buck doesn’t stop at A. Both B and C could have failed to cause a change in behavior because the correct media plan wasn’t delivered. You fail to even mention this frequent occurrence, so it looks to me like Dynamic Logic’s approach again misses the mark.

Gian

p.s. Happy to join you in a public debate. As you probably know, I am very active on the speaking circuit and I would be delighted to see you participate on one of my upcoming panels.

First, I’ll address and try to put to bed this issue of being misleading. As I’ve repeatedly written, I know that you did not intentionally mislead. But, I’ve seen evidence that people arrived at false conclusions (”lift” in headlines, tweets, direct inquiries from clients). I understand you have not experienced this. So, at this point, I think we should just agree to disagree and drop this issue.

But, on to the issue of ad effectiveness measurement framework, I think this is something of strong interest to the community and well worth continued debate. Your report with the OPA stated that the best way to measure ad effectiveness is by measuring impact on search, site visitation and e-commerce. We contend that the best approach is to measure changes in perceptions and sales (online and offline) supported diagnostically by post-view behavioral information.

I do not believe impact on search and site visitation is “irrelevant” as you have mis-quoted me. I wrote that these behaviors are “an important diagnostic tool” — pretty far from saying they’re irrelevant.

Furthermore, I believe advertisers should routinely measure all three: 1) impact on perceptions, 2) impact on sales (both online and offline) and 3) diagnostic behavioral information including changes in search, site visitation to the client site as well as related sites. And, I’m glad Dynamic Logic can offer this breadth of measurement.

Let me explain why perceptions and sales must take priority over post-view non e-commerce behaviors. Which of these scenarios do you think an advertiser would prefer?

The display ad campaign was shown to
A) Increase sales both offline and online but did not increase search volume or traffic to the site

B) Increased awareness, brand favorability and purchase intent, but did not increase search or site traffic

C) Increased search volume and traffic but did not change brand perceptions and did not increase sales.

I think 90% of advertisers would rank order the above, as written — A is first, then scenario B then C.

Here’s the bottom line. A campaign that increases sales is successful (with the caveat that perceptions weren’t harmed in the process). End of conversation. The buck, literally, stops there.

By contrast, a campaign that increases search and traffic may or may not be considered a success depending on what happens with brand perceptions and sales.

At the end of the day, if an ad campaign didn’t change anyone’s opinions and didn’t impact sales, it didn’t work. This is the same sentiment shared by Carrie Frolich, a person you quoted in your research.

I hope this helped clarify things for you and I look forward to a live debate.

Ken

Unfortunately, you seem unable or unwilling to grasp the fact that the research we recently conducted was not, as I have said in my previous posts on your blog — and as I made crystal clear in my presentation of the results — intended to establish a cause and effect relationship between display ads and sales. It simply described the characteristics of Internet users who are exposed to display ad campaigns. Many advertisers, publishers and media planners have told me that this type of analysis is of significant value to them and I reject your assertion that the approach is somehow flawed.

Based on the feedback that we have received from people who either attended the recent presentation or viewed the deck, we see no evidence that they have misunderstood the objectives and nature of the research.

In addition, as I explained in my previous posts and further elaborate below, comScore’s recent study was conducted as a supplement to previous comScore research (published in this month’s issue of the Journal of Advertising Research) that did firmly establish a causal relationship between ad exposure and lift in search, site visits, and sales. So, your claims about “misinterpretation” are really moot.

That said, your logic in claiming that trademark search and site visitation driven by display ads are irrelevant metrics escapes me. They are, after all, elements of many brands’ marketing plans and, as such, critical activities behind which marketers put substantial money in their efforts to persuade a consumer to purchase the advertised brand. In fact, research conducted by comScore (which I was invited to present at Professor Jerry Wind’s “Empirical Generalizations in Advertising” conference at the Wharton Business School and which was just published in the June issue of the Journal of Advertising Research) shows that display ads lift search queries and site visitation, both of which, in turn, help drive sales. It would be folly to ignore an understanding of the important role of these variables in helping a display campaign close a sale.

Let me be clear, I have no problem with your suggestion of including attitudinal metrics as diagnostics. They can certainly add value. But, I disagree with your assertion that search and site visitation not be used. It makes me wonder whether your understanding of how online advertising works isn’t based on the fact that Dynamic Logic has traditionally focused only on attitudinal metrics in its research and so you’ve never really had the opportunity to fully understand the synergy that exists between display and search ads and eventual sales? If so, you really need to get caught up. comScore’s Brand Metrix service provides not only attitudinal metrics but also a measurement of behavioral response, including lifts in search queries, site visitation, online and offline sales. Clients find this to be very valuable. Academics also, and I’m happy to forward you a copy of the Journal of Advertising Research so that you can grasp the significance of comScore’s research. You can also see a summary here:
http://www.comscore.com/Press_Events/Press_Releases/2008/11/Value_of_Online_Advertising

You should also read eMarketer’s current article “How Online Brand Advertising Can Influence Every Step Along the Purchase Funnel”:
http://www.emarketer.com/brandmeasurement/index.php/online-brand-advertising-influence-purchase/ This article does an excellent job of showing how display advertising can change consumer behavior and the benefits of measuring that behavior in all its manifestations. For example, because the conversion rates following a search query are so high, it’s valuable for an advertiser to understand if their display campaign has been effective in lifting the number of trademark search queries. You can’t gauge this by just measuring attitudes.

Over and out

Gian

p.s. Thanks for allowing me to promote comScore’s ad effectiveness measurement capabilities on the Dynamic Logic blog. It’s much appreciated.

As far as sales impact measurement, that’s been possible for a very long time in one form or another. And, neither I, nor Dynamic Logic make any claims to have been the first.

But, all this is a distraction from the main point of what I wrote. The main point was around our differences in terms of the framework for measurement. We agree with Carrie that perceptions + sales = the right way to measure display ad effectiveness. Whereas you focused on post-view online behaviors such as search, site visitation and e-commerce. E-commerce fits into the sales part of our formula but, we believe site visitation and search, althought very useful diagnostically, are typically not ad effectiveness metrics by themselves.

Finally, I am not criticizing the OPA. They support lots of great industry research. I disagree on the framework which I assume was developed by Comscore and not OPA. And, I think OPA would have been better served had more rigorous controls been used in the research. The results likely would have been positive either way, but the research community as a whole would have been more accepting of the findings had the research been better controlled.

Gian, we agree on many things and I think debate is very healthy for our industry. Perhaps we should arrange a public live debate. I’d welcome it.

Best,
Ken

While we’re discussing the subject of “misleading comments”, I’d like to address the claims you make in your article that DL is the first company to be able to measure the impact of display campaigns on sales (Yes, Ken, that’s what a reader would conclude from reading your article. Did you intend to create this perception?) In fact the 200+ studies included in comScore’s “Whither the Click” paper (which prove the positive impact of display ads on site visitation, trademark search queries, online and offline sales — all in the face of minimal clicks) were conducted over a period of several years — long before your parent company acquired Compete and you developed any capability of measuring sales and not just attitudes.

Sounds to me like your criticism of the OPA study is a lot like the pot calling the kettle black.

Regarding methods, I understand that you didn’t intend to prove causality. I pointed that out in my article and gave you credit for using the right terminology. However, that doesn’t mean it didn’t mislead. Based on what I have seen tweeted and based on comments we have received directly, many people have mis-interpreted the findings. Balancing with more controlled findings or including more caveats may have helped.