Measuring Creative Effectiveness
with Marketing Mix Modeling
CMO: “Okay, you reported on the effectiveness of our channels, but what about the creative? What can you tell us about the performance of our creative?”
Almost all of our MMM clients ask us about this at some point during the project.
And of course they will, after all, creative is a big (if not the most important) part of media effectiveness.
The good news is, this is possible. In this article, we look at the different options available for MMM modelers to measure creative performance.
Approaches for Creative Measurement in MMM
As a modeler, you have a couple of options for measuring creative performance:
- Campaign-level analysis
- Leveraging ground truth proxies for creative impact
Let’s explore how they work.
Campaign-level analysis
Ideally, you can measure creative impact in a campaign level analysis. You should be able to split the media variable for a channel into the different campaigns that compose that variable.
For example, if you have 3 campaigns that ran during the year, then you will have 3 variables and their coefficients in the equation.
Those coefficients can then tell you the impact of the corresponding creative on your sales.
So if Campaign 2 has a bigger coefficient, that basically means that the same GRP/Impression is delivering more value with Campaign 2 than using Campaign 1.
The Problem: Too Many Campaigns
Obviously this is an ideal situation, because in real life things rarely work this way.
Most of the time you end up with way too many campaigns to measure, and it’s impossible to get them all in the same equation because:
- You either run out of degrees of freedom (have way more variables than data points that you need to estimate these)
- Multicollinearity between campaigns (i.e. some of the campaigns overlap) making it very difficult to get separate measurement for each campaign.
So what can you do? One way to remedy this is to have some ground truth around the quality of the campaigns.
Ground Truth Proxies for Creative Effectiveness
If possible, you can tap into research being done by the brand in terms of quantifying the impact of their campaigns.
Fortunately, this is becoming more common. For instance, ad pre-testing is making a comeback. Some brands also run post-tests to understand which ads were more impactful.
There’s also the potential of using AI for creative testing. In fact, a number of attention measurement companies us this technology. This can provide the necessary ground truth for creative in a scalable way.
So with reliable information on which campaigns stood out, you can have them as separate variables and test whether or not they have a bigger or smaller impact compared to the average variable impact.
Conclusion
Measuring creative with MMM is not only possible but essential for understanding media impact. Afterall, the quality of the content is a big driver of a channel’s effectiveness. Providing insight into that enhances the credibility and usefulness of your MMM results.
We explored two ways to do this: campaign-level analysis and leveraging ground truth proxies. For clear-cut cases, campaign analysis works well. If the data cannot accommodate that, consider integrating ground truth sources where available.
By using these techniques, you can recommend what specific creative your client needs to run to be more impactful, improve their media performance and their ROI.
