A model that fits the data is not the same as a model that tells the truth about the business. Before any result reaches a budget conversation, it has to pass two tests: is it statistically sound, and does it match commercial reality. This piece covers the first test.
Seven statistics tell you whether your model is measuring signal or memorizing noise. None of them asks you to read a formula. All of them decide whether you can defend the numbers when a CFO pushes back. A result you understand is a result you can defend.
1. R-squared: how much of the story the model explains
R-squared tells you how much of the movement in your KPI the model accounts for. An R-squared of 0.88 means the model explains 88% of the variation in sales across the period. On weekly data, a well-specified model usually clears 0.80.
Here is the trap. R-squared rises every time you add a variable, whether or not that variable means anything. Add a dummy for every odd week and you can manufacture a near-perfect fit that has learned nothing. That is why adjusted R-squared matters: it only rises when a new variable earns its place. Before you trust a high R-squared, ask one question of every variable in the equation, does it have a documented commercial reason to be there. If not, the fit is inflated and the conclusions are unreliable.
2. F-statistic: does the model explain anything at all
The F-statistic tests the whole model at once. Taken together, do your variables explain more than random noise would. A significant result, with a p-value below 0.05, confirms the model has real explanatory power and clears you to examine the individual variables inside it. A non-significant F is a stop sign. It means the model explains nothing beyond chance, and nothing downstream is worth reading.
3. t-statistic: the honesty check on every variable
The F-statistic judges the model. The t-statistic judges each variable inside it. It tells you whether a channel’s measured effect can be reliably separated from zero. The rule of thumb: a t-statistic above 2 means the effect is real at a conventional confidence level. Below 1.5, treat it with caution. Below 1, the variable has shown no measurable independent impact in this dataset.
This is the statistic I watch most closely in client work, because it is the honesty check. If the uncertainty around a channel’s effect is wider than the effect itself, that channel could plausibly have zero impact, or even the opposite of what the estimate suggests. Reporting a number like that as if it were solid is how measurement loses credibility. Every effect you put in front of a CMO should have a t-statistic that justifies the confidence you are placing in it.
4. Standard error and MAPE: accuracy a boardroom can read
Two statistics tell you how close the model’s predictions sit to reality. The Standard Error of the Estimate is the average size of the model’s miss, expressed in the units of your KPI. MAPE, the Mean Absolute Percentage Error, says the same thing as a percentage: on average, how far each period’s prediction lands from actual sales.
MAPE is the accuracy metric clients ask for most, because it needs no statistical training to read. Below 10% is excellent. Between 10% and 20% is acceptable. Above 20%, find the periods driving the error before anyone uses the model to move budget. A number a CMO can state plainly in a board meeting is worth more than one only the analyst understands.
5. Durbin-Watson: the test that catches a missing season
Durbin-Watson checks whether the model’s errors are independent from one period to the next, or whether they follow a pattern the model failed to capture. The value runs from 0 to 4, and you want it close to 2.0. Below 1.5 almost always means a structural variable is missing, usually a seasonal one. If the Christmas peak is not in the model, the error will spike every December, and Durbin-Watson will fall. The fix is not statistical gymnastics. It is finding the pattern in the residuals and adding the calendar variable that explains it. The value should recover toward 2.0 once you do.
6. VIF: the reason your digital channels fight each other
The Variance Inflation Factor measures how much each variable overlaps with the others. When Paid Search, Display, and Paid Social all run always-on and move together, the model cannot tell whose effect is whose. Their estimates become unstable and their reliability collapses. Keep VIF below 5 for every variable. Above that, the channels cannot be cleanly separated. The remedy is structural: group the correlated channels, apply a splitter, or move to a pooled model that gives the data more room to distinguish them. This is the most common failure I see in digital-heavy models, and the one most often missed.
7. Jarque-Bera: the test that flags what the model forgot
Jarque-Bera checks whether the model’s errors are normally distributed, reading their skewness and how heavy their tails are. A non-significant result means the errors look normal, which is what you want. A significant result is a signal, not a verdict. It usually means an outlier or a one-off event the model has not accounted for: a promotion, a stockout, a competitor shock. Rather than dismiss it, go and find the event and give it a variable. Normality is rarely the real problem. A missing piece of the story usually is.
Statistical soundness is necessary, but it is not enough
A model can pass all seven of these and still be wrong. Statistics confirm the model fits the data. They say nothing about whether it reflects the business. Three commercial checks close that gap.
Spend-to-contribution alignment. A channel taking a large share of the budget should show a contribution to match. When it does not, there has to be a business reason. Sometimes there is, branded paid search converts existing demand cheaply and efficiently. When there is not, the model needs another look.
Year-on-year consistency. The contribution story should track what actually changed in your activity. A channel whose contribution climbs while its investment falls needs an explanation before it reaches a client.
ROI and cost-per-acquisition benchmarking. Every channel’s efficiency should sit within range of comparable work. Outliers in either direction are a prompt to investigate, not a result to report.
A model is statistically sound when these seven diagnostics pass. It is commercially sound when it tells a story that matches what you know happened. You need both before the numbers leave the room.
The bottom line
Frequently Asked Questions
It has to pass two tests. Statistically, the seven diagnostics in this article should clear their thresholds. Commercially, the contribution story has to match what you know happened in the business. A model that passes one test but fails the other is not ready to guide budget.
On weekly data, a well-specified model usually clears 0.80. Treat a very high R-squared with suspicion if it was produced by unsupported dummy variables, because that is memorization, not explanation. Adjusted R-squared and MAPE together give a more honest read than R-squared alone.
Because they run always-on and move together, so the model struggles to separate their individual effects. Keep VIF below 5. Above that, group the correlated channels, apply a splitter, or move to a pooled model that gives the data more room to tell them apart.