Article by Dynamic Logic's Ken Mallon: To GRP or not to GRP? August 26, 2009
Posted by steng in : Ken Mallon , add a comment
To GRP or not to GRP? Is online branding broken? These are questions facing Geoff Ramsey of eMarketer and the industry as a whole. Geoff recently commented on The Great GRP Debate, in his July 13, 2009 article. Also, in July, eMarketer published Online Brand Measurement: Connecting the Dots, based on industry interviews.
The continually debated issue is around metrics, metrics, metrics. What are the correct metrics? Are some better than others? Some claim online measurement is a mess. Is it?
GRPs are important … with a caveat
Why are GRPs important? Well, two reasons. First, many large companies use GRPs as inputs to media allocation and other models. They depend on these models to understand the impact advertising in each media is having. Although there are other and, sometimes more cost-effective, methods for evaluating the independent and synergistic effects of advertising across different media, these GRP modeling approaches have been in place for many years and change is difficult.
Second, GRPs are constructed based on reach and frequency. Reach is important because it’s an input to another important formula: impacted reach = reach times ad effectiveness. More on impacted reach in a moment.
So, in the GRP debate, I’m on the GRP side with a caveat. I think reach is critical to determining overall impact or impacted reach, as I call it, of a campaign, but GRPs may over-simply the reach concept.
GRPs over-simplify the concept of reach, especially online.
If you have reach and frequency, you can calculate GRPs, but GRPs are an over-simplification. Here is why. In the online world, there is a big difference in impact between delivering 10 million impressions to 5 million people versus delivering 10 million impressions to 2 million people. They can both represent the same number of GRPs, but the impacted reach of the latter example is far less.
The table below steps through the math as to why this is so. This concept can be illustrated for any measure of perception or with sales data. In this example, I chose brand favorability as the perception metric and the data come from the Dynamic Logic MarketNorms database. I looked at three different exposure frequency groups: those that saw ads from a given campaign exactly once, those who saw 2-3 ads and those who saw 4 or more ads. The data are aggregated across 71 campaigns. Those who saw 2-3, had an average exposure of 2.3 and those who saw 4+, had an average exposure of 10.2.
From the above example, one can clearly see that the total effectiveness or impacted reach, in the three scenarios, is vastly different even though the number of impressions, which are the closest things we have to GRPs, are the same. I went into more detail on this in an i-Media connection article two years ago. Note how dramatically reach decreases at different frequency levels. The reach among the 4+ group is less than 1/10th of the reach in the single frequency group. The impact, on a percentage basis, is higher in the high exposure group (2.7% became favorable to the tested brand who otherwise would not have been, versus <2% in the <4 exposure groups) but the loss in reach cannot be made up and shows itself in the impacted reach calculation.
The problem of having similar GRPs associated with vastly different impact is less of a problem offline. In the offline world, once you choose, for example, a magazine, TV show or other offline channel, the range of possible frequencies is somewhat limited. Online, someone can see your ad dozens or even hundreds of times.
More on impacted reach and ad effectiveness
Impacted reach is the number of people impacted by a particular ad campaign. And, I believe that in 90% of cases, impact boils down to one of two things. For an ad campaign to be considered effective, in any media, it has to either change people’s perceptions about your brand or product or it has to drive incremental sales or both.
So, ad effectiveness = changes in perceptions + sales impact. Ad effectiveness is not about ad interaction, clicking, driving traffic, etc. These post-view behaviors can be very important diagnostically, but I don’t think they belong in the ad effectiveness equation. I’d put them in the category of very important diagnostic information. They’d be in the same category as finding out if someone read the newspaper circular or finding out if someone noticed an end-aisle display. They are important and can help you understand what went wrong when perceptions and/or sales are not impacted, but they aren’t endpoints in themselves. This is why Dynamic Logic launched AdIndex Connects with Compete so that we can now provide enhanced post-view behavioral data in addition to the attitudinal and sales impact metrics. It’s a very valuable layer of insights.
Is brand impact measurement broken?
Now to the issue of online brand impact measurement. Is it broken? In the eMarketer Brand Measurement article, there is a nice section citing Dynamic Logic MarketNorms data. It shows that, on average, online advertising lifts brand metrics. But, it also shows that there is great variability in results and that the bottom 20% of online ad campaigns actually have negative impact on perception.
Based on our further research, the biggest driver of success versus failure is the quality of the ads. So, we believe that in-market optimization of ads should not be based on click rates or interaction rates but, rather, on creative quality. Conducting a copy test before or early on in the campaign, can have a huge positive impact on results.
So, we do not believe online branding is broken and neither is measurement. Advertisers conduct thousands of research projects per year that include brand impact measurement as part of the accountability of the campaign. It is not too difficult to also compute impacted reach and ROI metrics such as impacted reach per dollar spent. Advertisers who focus on good creative tend to be more successful and we support folks like the Online Publishers Association who are pushing the envelope by launching new ad formats that more closely mirror magazine advertising.
Thanks, eMarketer for the continued great articles. Look forward to commenting more in the future.
Ken Mallon
SVP Custom Solutions
Dynamic Logic
New Research: Position and ad shape may have more impact in online advertising than size August 20, 2009
Posted by dynamiclogicbeaty in : Dynamic Logic Press Release, Ken Mallon , add a commentNew research released today by Dynamic Logic reveals that online ad effectiveness depends less on size than it does on shape and placement, based on MarketNorms data.
Is Bigger Better When it Comes to Online Ad Size? Jury Still Out, According to Dynamic Logic
Position and Shape May Play a More Important Role than Size 
New York, August 20, 2009 -Research released today by Dynamic Logic, the leaders in measuring digital advertising effectiveness, reveals ads that are integrated into the content of the page, such as half banners and rectangles, are the most effective in driving online ad awareness and purchase intent.
The research, based on 2,390 online display campaigns that took place over the past three years, is from Dynamic Logic’s MarketNorms database, the largest in the industry. It found that half banners (234 x 60) and rectangles (180 x 150) were more effective than ads that frame the page such as leadersboards and skyscrapers.
Media Coverage:
Top story in Ad Age today! View Ad Age article:
Why Large Online Ad Formats Aren’t Industry’s Silver Bullet
Bigger Isn’t Necessarily Better When It Comes To Online Ad Formats
Dynamic Logic's Ken Mallon comments on recent OPA/comScore research June 30, 2009
Posted by Mark Blei in : Ken Mallon, Staff posts, industry news , comments closed Ken Mallon Dynamic Logic’s SVP Custom Solutions & Ad Effectiveness Consulting speaks out about recent research released by the Online Publishers Association in conjunction with comScore.
Ken is a 25-year research veteran having applied research methods, statistics and data mining expertise to a variety of fields including health, pharmaceuticals, marketing, and internet behavior. At Dynamic Logic, Ken heads the Custom Solutions team, providing specialized research and consulting services to top clients. Before joining Dynamic Logic, Ken was Director of Product Development, Director of Marketing Solutions, and Director of Data Mining at Yahoo!, where he helped to create Yahoo!’s data-mining group, as well as their behavioral targeting system, unique consumer insights, and passively collecting data and advertising products.
Ken holds a Master of Health Science in biostatistics from Johns Hopkins University, a Master of Science in statistics from Stanford University and a Bachelor of Science in secondary math education from University of Connecticut. Kens comments appear below.
Last week, OPA and comScore released some research titled “The Silent Click: Building Brands Online”. Dynamic Logic is a long-time friend of the OPA (we love you, Stu!), but I just had to comment on this. While we agree with one of the key messages of the research – to stop putting emphasis on CTR – we disagree with the overarching framework and believe some of the research methods are misleading.
FRAMEWORK
I won’t start with a methodological discussion (methods geeks, skip to the bottom of the article where we can have a nice statistical dialogue). Instead, let’s start higher with the overall framework. In the key findings (slide 52) this research states:
Search + Site Visitation + e-Commerce spending = a smart formula for measuring display ad effectiveness.
We fundamentally disagree with this concept. On page 5 of the deck, Carrie Frolich, Managing Director Digital, Mediaedge: cia, is quoted as saying:
“Remember why you’re advertising … You are not advertising for clicks … What you’re advertising for is to sell me stuff or change perception, and that’s what we need to be measuring against.”
Frolich is right-on. This is very similar to what I say on just about every panel and conference speaking engagement in which I participate:
“At the end of the day, you care about two things – did my advertising help sell something or did it change someone’s opinion. Everything else is a surrogate or noise.”
We believe that the right formula is:
Perception changes + sales (both online and offline) = the best formula for measuring display ad effectiveness.
We believe that post-view behaviors play a key role diagnostically and add color to the above.
The brand perceptions part can be handled via surveys (passively collected comments via blogs and otherwise can play a role as well) and the sales impact part must utilize one of several methods for capturing both online and offline sales which I’ll go into later.
I want to digress for a moment into a pharmaceutical industry parallel. During my years as a statistical scientist at UCSF, Genentech and Amgen, I was taught that at the end of the day, when talking about drug interventions, a medicine must do one of two things. It must either save lives or improve quality of life. All other things are surrogates or noise. Sound familiar? If a drug improved your cholesterol, but didn’t improve your quality of life or reduce the risk of death, would you take it? Probably not. If an ad format increased interaction but didn’t make people like your brand better and didn’t increase sales, was it effective?
I know a lot of this sounds like I’m discrediting post-view behavior. I don’t mean to do that. Post-view behaviors such as driving search, visits to the advertiser site, brochure downloads and trailer views can be important diagnostic tools for understanding lifts or lack thereof in brand perception and sales. But, by themselves, they are not ultimate measures of ad effectiveness. There is one post-view behavior worth clarifying – e-commerce. In the Dynamic Logic framework – it’s all about perceptions and sales – e-commerce is included in the latter. It’s part of the sales impact measurement. Online sales, although they are technically a post-view behavior, are not a surrogate. They are part of the real deal.
Now, let’s get back to Carrie Frolich. If she’s right and I believe she is, then why doesn’t the research presented include any brand perception results? And, why is the sales impact information only limited to online and not offline sales?
Dynamic Logic is known for measuring the branding impact of digital advertising and we do this via live web-intercept survey. But, what some of our clients are increasingly discovering is that we can do much more. We are currently advocating a broad-based way of measuring the impact of display ads – brand impact + offline sales estimation + e-commerce (where appropriate) + other post-view behaviors to add color and for diagnostic use.
Some nine months ago, we discussed internally the impact of the economy and wondered what we might be able to do for our clients who were feeling increasing pressure. What we heard again and again from our clients is “ROI, ROI – we want more ROI and … it can’t be expensive to measure”. Publishers, agencies, advertisers and technology providers are under increased pressure from management and clients to deliver results. This type of pressure has always existed in digital but now it’s heavier than ever.
As a result of this need, we have begun thinking very creatively about what we can do. We worked out a simple, straight-forward and relatively inexpensive way of estimating the offline and online sales impact of display advertising (and website exposure) applicable to just about any advertised product. This, combined with our patented approach to measuring changes in brand perception, completed the sales + perception = display ad impact formula we advocate and has been reinforced by Carrie Frolich. Also, since our tracking technology could be used to capture post-view behaviors such as e-commerce, we had that element as well.
What was missing were other post-view behaviors such as impact on search and impact on website visitation other than the advertiser site. An example of this would be measuring the traffic to kbb.com following display ad exposure to an automotive campaign.
Meanwhile, TNS became one of our cousin companies, opening the door to closer collaboration and data integration with Cymfony and Compete. Sometimes, when you ask for something, you get it! By collaborating with Compete, we can now estimate the impact of display ads on search volume and website traffic to any prominent site. And, what of Cymfony? What role can they play? Longer-term, we hope to use their technology to enhance the measure of perceptions (among other things). This is currently assessed largely via surveys, but could be enhanced with passively-collected data across the Web.
So, to sum up regarding the framework,
INCOMPLETE: Search + Site Visitation + e-Commerce spending = a smart formula for measuring display ad effectiveness.
CORRECT: Perception changes + sales (both online and offline) = the best formula for measuring display ad effectiveness.
And, by the way, although some of our capabilities are not fully productized in this regard, we can deliver today on this way of measuring digital advertising effectiveness and can do so in a relatively cost-effective way.
Thank-you, Carrie!
MISLEADING METHODS
And, now on the more tedious discussion of the methods used in this research. First, I give OPA and comScore credit for being responsible researchers and using the term correlation. Also, they didn’t do statistical testing to compare the control and exposed group. Since the groups were not scientifically equivalent, not testing is the right thing to do since statistical significance would have no meaning anyway.
That being said, I think more could have been done to make the comparison groups more comparable. This is important because, although they are careful to use the term correlation so that people should not conclude causality, we all know that the untrained reader will immediately conclude causality.
As an example, let’s look at slide 16. It shows that visitors to a brand’s site who had been exposed to that brand’s advertising ended up spending more time on the site and consumed more pages. Firstly, let me say this. I believe in display advertising. I believe online ad exposure is likely to cause someone to end up spending more time on a website. I’ve just seen too much controlled data to support it. So, I have to believe it. However,
1) This research doesn’t prove that ad exposure leads to greater site consumption
2) The authors don’t directly claim it to be true (they use the careful and correct term, correlation), but
3) Most people will arrive at the conclusion that online ad exposure drives increased brand consumption in terms of website time spent and pages visited (I’ve read dozens of tweets with that interpretation)
Although, again, I believe (3) is likely to be true, I’m just saying that it doesn’t follow from the data presented. Although the authors cover themselves statistically by not testing and by using the term correlation, I feel it’s a bit irresponsible given that much of the audience isn’t well-trained in statistical interpretation.
I have the same complaint for nearly all the comparisons that are made in this presentation. It’s a bit worse in cases where there is an unexposed versus exposed comparison. Let’s look at slide 17 for example. It shows that those who were exposed to advertising had 7% more e-commerce than unexposed (similar results on slide 18).
First, look at the x-axis. The range of x-axis is from 200 to 250. When I was a statistics student I learned that shortening the axis to make the difference seem larger, is a reporting no-no. But, that’s largely a pet-peeve – I can see why you’d want to do that. My main criticism is the selection of the unexposed as the control group. To avoid misleading results, one must select the comparison group very carefully and since it wasn’t done scientifically, great care should be taken to make the groups as comparable as possible either by weighting, multivariate regression, stratification or other techniques.
Let me explain just one possible reason for having a 7% difference that has nothing to do with ad exposure. Take the example of re-targeting. That’s when you show an ad to someone because they previously visited your site (and you smartly cookied them) or they previously performed a search in your category or for your brand. If you do a retargeting campaign, you are selecting a subset of the population who has a far greater chance of being interested in and therefore greater chance of purchasing your product. Same goes for good behavioral targeting (there’s good and there’s bad – but, I’ll save that topic for another blog). So, it very well could be that the exposure itself played no role in later e-commerce. It could just be a function of good targeting. And, if the unexposed group possibly came from other websites than the exposed, then in-context targeting could have been the reason for increased e-commerce.
Thus, again, although I believe in online ads and believe they increase e-commerce, the research presented doesn’t show it. The same methodological criticism applies to the rest of the results.
Again, we are supporters of the OPA. But, we think comScore did them somewhat of a disservice with this research. I believe that online advertising drives e-commerce, increases search and has many other positive brand benefits, I don’t think this research proves it.
If you want to measure the complete impact of display advertising – brand impact and sales, talk to us at Dynamic Logic. We can provide valuable and robust scientific data within our framework for measuring display ad effectiveness:
Perception changes + sales (both online and offline) = the best formula for measuring display ad effectiveness.
Best,
Ken Mallon
Senior Vice President
Custom Solutions & Ad Effectiveness Consulting
Follow us on twitter: Dynamiclogic Join our Facebook Page Join our group on LinkedIn
