7 Stats to Evaluate Your Marketing Mix Model

In the previous part of this MMM Guide series, we explored one of the cornerstones of Marketing Mix Modeling: Regression Analysis. In this article, we cover how to evaluate your marketing mix model and ensure the robustness and reliability of your results. This step is crucial and should be performed at the modeling stage of the MMM workflow. Now that you have built a marketing mix model that makes sense, how do you make sure it’s statistically sound?

There are a number of tests you need to conduct to be sure. Here are the key statistics that should be computed and analyzed to evaluate the robustness of your marketing mix model.

1. R Squared: How Much Variance Is Explained by the Model

The coefficient of determination, or R Squared (R²), measures the percentage of total variance in the dependent variable that is explained by the model. For example, if R² equals 85%, it means 85% of the variance in your sales data is explained by the variables in the model — media, promotions, distribution, seasonality and so on.

A modified version of R² is the adjusted R². One drawback of the standard R² is that it increases as more variables are added to the model, regardless of whether those variables actually improve explanatory power. Adjusted R², on the other hand, only increases when a new variable improves the model more than would be expected by chance — and decreases when a variable contributes less than expected. Adjusted R² is therefore the preferred metric when comparing two models.

R Squared

$R^2=1-\frac{\mathrm{RSS}}{\mathrm{TSS}}$

OR

$R^2=\frac{\sum \left(\widehat{Y}-\overline{Y}\right)2^{\mathrm{}}}{\sum \left(Y-\overline{Y}\right)2^{\mathrm{}}}=\frac{\mathrm{ESS}}{\mathrm{TSS}}$

Adjusted R Squared

$\overline{R}{}^2=1-\frac{(1-R^2)(N-1)}{N-p-1}$

OR

$\overline{R}{}^2=1-\frac{\mathrm{RSS}\times \frac{1}{(n-k)}}{\mathrm{TSS}\times \frac{1}{(n-1)}}$

k denotes the number of independent variables (k=p-1)

2. F Statistic: Overall Model Relevance and Significance

The F test measures the overall significance of the model. The null hypothesis (H₀) states that all the coefficients in the equation are equal to zero at the same time.

Using the same example — if R² equals 85%, meaning 85% of the variance in sales is explained by the model variables — the F test checks whether all the coefficients of media, promotions, seasonality and distribution take zero simultaneously. If they do, the model is not significant overall and none of the variables appear to impact the dependent variable.

If the F test is significant, you then proceed to test each variable individually. Be aware: rejecting H₀ of the F statistic does not mean every single variable in the equation is significant on its own.

F Statistic

$F=\frac{{{\sigma }_1}^2}{{{\sigma }_2}^2}$

where

${\sigma }^2=\frac{\sum (x-\mu {)}^2}{n}$

Testing the Overall Significance of the Regression (F-Test)

$H_0:{\beta }_1={\beta }_2=\cdots ={\beta }_k=0$

H₀: not all the parameters are 0

$\frac{\sum {(\widehat{y}-\overline{y})}^2\times \frac{1}{(k-1)}}{\sum {\widehat{\mu }}^2\times \frac{1}{(n-k)}}=\frac{\mathrm{ESS}\times \frac{1}{(k-1)}}{\mathrm{RSS}\times \frac{1}{(n-k)}}\sim F_{k-1,n-k}$

So if F_cal > F_table reject H₀

We can calculate an F_cal value from the value of R Squared

$F_{\text{cal}}=\frac{R^2\times \frac{1}{(k-1)}}{(1-R^2)\times \frac{1}{(n-k)}}\sim F_{k-1,n-k}$

3. T-Statistic: Relevance and Significance of One Predictor

The t-stat is a measure of coefficient reliability. It is the ratio between the computed (or estimated) coefficient and the standard error of that coefficient.

The bigger the t-stat the better — it means the coefficient is statistically different from 0 (where 0 indicates no relationship between the dependent and independent variable). In other words, if another sample were drawn, you would likely obtain a similar coefficient value, and that value is reliably different from zero.

The t-stat is computed for each independent variable in the equation. As a rule of thumb, if the t-value of a variable — search spend, for example — is greater than 2, then that variable is statistically significant and has a meaningful impact on sales.

T-Statistic

$t_{\widehat{\beta }}=\frac{\widehat{\beta }-{\beta }_0}{\mathrm{SE}\left(\widehat{\beta }\right)}$

4. The Standard Error of the Estimate (SEE) and the MAPE: Model Accuracy

The Standard Error of the Estimate (SEE) is the average error across all residuals produced by the model. This metric should be as small as possible — a small SEE means the difference between the real values and the model’s estimates is consistently controlled across all data points.

The MAPE, or Mean Absolute Percentage Error, is also a measure of prediction accuracy — and like SEE, the smaller the better. MAPE is computed as the average of the individual absolute forecast errors divided by the actual values for each period. The closer MAPE is to zero, the more accurate the predictions. Its advantage over SEE is that it expresses error in percentage terms, making it easier to interpret regardless of the scale of the dependent variable.

The Standard Error of the Estimate

$\mathrm{SEE}=\sqrt{\frac{\sum {(x_i-\overline{x})}^2}{n-1}}$

Simple Regression Model

$Y={\beta }_1+{\beta }_{2}x$

The Standard Error of the Slope

$\sigma \left(\widehat{{\beta }_2}\right)=\frac{\widehat{\sigma }}{\sum (x_{2i}-\overline{x_2})}$

The Standard Error of the Regression

$\sigma =\sqrt{\frac{\mathrm{RSS}}{n-2}}=\sqrt{\frac{\mathrm{TSS}-\mathrm{ESS}}{n-2}}$

Multiple Regression Model

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$

The Standard Error of the Regression (Multiple)

$\widehat{{{\sigma }_{\mu }}^2}=\frac{\sum {\widehat{\mu }}^2}{(n-k)}=\frac{\mathrm{RSS}}{(n-k)}$

The Standard Error of the Slope (Multiple)

$\widehat{\mathrm{Var}}\left(\widehat{\beta }\right)=\widehat{{\sigma }_{\mu }}(X^{T}X{)}^{-1}$ $\mathrm{s.e.}\left(\widehat{{\beta }_j}\right)=\widehat{{\sigma }_{\mu }}\sqrt{a_{\mathrm{jj}}}$ $\widehat{\mathrm{Var}}\left(\widehat{\beta }\right)=\left[\begin{array}{ccc}\mathrm{var}\left(\widehat{{\beta }_1}\right)& \mathrm{cov}(\widehat{{\beta }_1},\widehat{{\beta }_2})& \mathrm{cov}(\widehat{{\beta }_1},\widehat{{\beta }_3})\\ \mathrm{cov}(\widehat{{\beta }_1},\widehat{{\beta }_2})& \mathrm{var}\left(\widehat{{\beta }_2}\right)& \mathrm{cov}(\widehat{{\beta }_2},\widehat{{\beta }_3})\\ \mathrm{cov}(\widehat{{\beta }_1},\widehat{{\beta }_3})& \mathrm{cov}(\widehat{{\beta }_2},\widehat{{\beta }_3})& \mathrm{var}\left(\widehat{{\beta }_3}\right)\end{array}\right]$

MAPE

$M=\frac{1}{n}\sum {}_{t=1}^n\left|\frac{A_t-F_t}{A_t}\right|$

5. Durbin Watson: Checking for Autocorrelation

Durbin-Watson (DW) is used to test whether the residuals of the model are independent of one another.

Autocorrelation — the assumption that residuals are not correlated across time periods — is a core requirement of the OLS estimation method. The Durbin-Watson statistic varies between 0 and 4. A value of 2.0 indicates the absence of autocorrelation.

Durbin Watson

$d=\frac{\sum {}_{t=2}^T{(e_t-e_{t-1})}^2}{\sum {}_{t=1}^T{e_t}^2}$

6. The Variance Inflation Factor: Checking for Multicollinearity

The Variance Inflation Factor (VIF) indicates whether the explanatory (independent) variables are correlated with one another.

In regression analysis, the VIF statistic should be under five for every variable. If it is not, this signals a multicollinearity problem.

Multicollinearity is one of the most challenging issues in MMM projects. Brands tend to deploy several channels simultaneously — especially digital ones such as search, display and YouTube — which results in highly correlated variables and makes it difficult to disentangle the individual impact of each channel.

The Variance Inflation Factor

$\mathrm{VIF}=\frac{1}{1-{R_i}^2}$

7. Jarque Bera, Kurtosis, Skewness: Checking for Normality

Jarque-Bera, kurtosis, skewness and related tests are different ways to test the normality of the residuals. In OLS regression, residuals should be normally distributed — this is a core assumption of the method. Violating it can undermine the reliability of your coefficient estimates and significance tests.

Jarque Bera

$\mathrm{JB}=\frac{(n-k)}{6}(S^2+\frac{{(K-3)}^2}{4})$

Evaluate Your Marketing Mix Model for Business Logic, Not Just Statistics

Establishing that a model makes sense statistically is crucial. But that is only one half of the story. MMM is about modeling consumer behavior, so it is equally crucial that the model makes commercial and business sense. For example:

• When modeling a CPG brand, it is not possible to accept that price has a positive impact on sales. One cannot present results that say: the higher the price, the higher the sales. That is counter-intuitive.
• If a business is spending $20 million on advertising, and the model says that the advertising coefficient is negative, then they need to re-evaluate their model to see if they made the right transformations.

Not only should you take the time to evaluate the equation from a statistical standpoint, but you should also assess whether the level of elasticity, seasonality, ROI and the different parameters you are estimating make sense business-wise. You should be able to seamlessly defend them when presenting results back to the client or your internal team.

“Modeling is an iterative process. You will often find yourself going through multiple models before you are finally satisfied with the results. With rigorous model evaluation, you can be confident you are on the right track in your marketing mix modeling journey.”

Model Statistics in MassTer

With MassTer, you can evaluate your model using all the statistics above. When building and tweaking your model, these statistics are instantaneously updated and displayed in a table above your model — giving you a real-time view of model quality at every iteration.

Frequently Asked Questions

What is a good R-squared value for a marketing mix model?

There is no universal benchmark, but adjusted R² above 0.8 is generally considered strong for MMM. More importantly, always report both R² and adjusted R² together. A large gap between the two suggests overfitting — the model fits historical data well but may not generalize to new periods.

What does the Durbin-Watson statistic tell you in MMM?

It tests whether the residuals are correlated across time periods (autocorrelation). A value close to 2.0 means residuals are independent — which is what you want. Values significantly below 1.5 or above 2.5 indicate a problem with the model’s structure that should be addressed before interpreting results.

What is MAPE and what is an acceptable threshold in marketing mix modeling?

MAPE (Mean Absolute Percentage Error) measures the average percentage difference between your model’s predictions and actual values. A MAPE below 10% is generally considered good in MMM. Above 20% suggests the model’s predictions are unreliable and the specification should be revisited.

How do you spot multicollinearity in an MMM model?

Check the Variance Inflation Factor (VIF) for each variable. Any variable with a VIF above 5 signals multicollinearity. This is particularly common in digital-heavy models where search, display and social spend tend to move together. Solutions include variable transformations, dimensionality reduction, or rethinking the channel groupings.

Key Takeaways: How to Evaluate Your Marketing Mix Model

• ✓Use adjusted R², not just R². R² always increases as you add variables. Adjusted R² only increases when a new variable genuinely improves the model — making it the reliable choice for model comparison.
• ✓The F-statistic tests overall significance before you read individual results. A significant F confirms the model explains meaningful variance — but rejecting H₀ does not mean every variable is significant on its own.
• ✓T-stat above 2 is your threshold for variable significance. Below 2, question whether the variable belongs in the model at all.
• ✓MAPE below 10% signals reliable predictions. Report it alongside SEE for a complete picture of model accuracy.
• ✓Durbin-Watson near 2.0 means no autocorrelation. Values far from 2 signal a structural problem in the model.
• ✓VIF above 5 is a multicollinearity warning. Common in digital-heavy MMM projects where channels are deployed simultaneously.
• ✓Statistical soundness is necessary but not sufficient. The model must also make commercial sense — counterintuitive results (e.g. price positively impacting sales) should always be challenged before presenting.