Are Linear Models *Actually* “Easily Interpretable”?
I originally published this blog post on LinkedIn.
An oft-repeated meme in data science circles is that linear models can sometimes be preferable to other machine learning methods because they’re “easily interpretable”; but how true is that statement? Depending on your background, you probably first encountered linear regression in either a machine learning or statistics course. If you’re coming from a machine learning background, you probably think of linear regression as the following:
That is, the model is predicting some outcome variable based on a linear combination of input variables. The βs are selected by minimizing the sum of the squared errors for a training set of data, i.e.,
But if you come from a statistics background, you probably think of linear regression (AKA, ordinary least squares/OLS) as the following:
At a glance, it might seem like the two approaches are similar, but they actually diverge considerably in their interpretability. When people encounter linear regression for the first time, the model is typically presented as a data generating process in which there is a linear relationship between the input variables and the output variable. That is, the model is saying that, ceteris paribus (all else equal), a one unit increase in x_i will literally cause a β_i change in the output. Further, the parameters/coefficients/βs that maximize the likelihood of the stats equation are the same parameters that minimize the sum of the squared errors in the machine learning equation (if you’ve never heard that before, I recommend working it out for yourself; you’ll want to take a look at the PDF of a normal distribution and incorporate some logarithms and calculus). So there’s a temptation for data scientists who use linear regression for prediction to interpret the coefficients causally, but the coefficients only have a causal interpretation under a set of very strict assumptions. Specifically, the model must have (from the Wikipedia link):
- Correct specification
- Strict exogeneity
- No linear dependence
- Spherical errors
- Normality
If any of these assumptions aren’t satisfied, then the coefficients will not be an accurate reflection of the causal relationships between the input variables and the outcome variable (note: as pointed out by Matthew Drury in the comments of this article on LinkedIn, the normality assumption [actually, the spherical errors assumption too] is not necessary to obtain consistent estimates of the coefficients; however, it’s used in the vast majority of OLS applications because knowing the sampling distribution of the parameter estimates is necessary for hypothesis testing, i.e., determining whether or not a result is “significant”; Matt also brings up the concept of “interventions” — if you’re interested in learning more about the philosophy behind using mathematical models to investigate causality, check out this Stanford Encyclopedia of Philosophy article on the subject). To fully appreciate the nuances of these assumptions, let’s consider a few of them in the context of a typical machine learning workflow.
Most data scientists start a modeling project by first collecting some data and doing some feature engineering. Right off the bat, we’re in danger of violating assumptions 1–3. Oftentimes, some (or all) of the actual causal variables are not available to the data scientist, which means the model cannot be correctly specified (violating 1) and will likely not exhibit strict exogeneity (violating 2). Exogeneity is a concept that (in my experience) many data scientists aren’t familiar with because it’s not given a lot of attention in non-stats/economics curricula, but it’s extremely important to understand when interpreting linear models causally, so I’ll briefly review it here.
Exogeneity concerns the ϵ term in the stats equation. ϵ is often referred to as a “noise” or “error” term. However, in my opinion, these names for ϵ are confusing because there is a second equation associated with the statistical interpretation of linear regression:
This equation shows the estimated parameters for the model specified in the stats equation given a sample of data. Because ϵ-hat is literally the difference between the true value for the outcome variable (y) and the predicted value given the estimated model parameters (y-hat), ϵ-hat is the “error” of the model in a colloquial sense, but a better description for ϵ-hat is that it is the sample estimate of ϵ.
So what, exactly, does ϵ represent, if not the model’s “noise” or “error”? Well, in the data generating process specified in the stats equation, ϵ represents the impact on the outcome variable from all of the causal factors that are not explicitly included in the model. And, as outlined in the assumptions, the combined impact of these additional causal factors are normally distributed with a mean of zero.
OK, back to exogeneity. Strict exogeneity refers to a situation in which a model doesn’t suffer from endogeneity, which is when at least one of the variables included in the model is correlated with ϵ. But how can one of the input variables be correlated with ϵ if it’s normally distributed with a mean of zero? The answer is that there must be a disconnect between the actual data generating process and your guess for the data generating process. To be more explicit, let’s say some true data generating process has the following form:
and that x_1 and x_2 are correlated. Further, let’s say you guessed that the data generating process looks like the following:
ζ represents your assumption that the impact on the outcome variable from all of the causal factors not explicitly included in the model is normally distributed and has a mean of zero. But if the model suffers from endogeneity, then one of the input variables will be correlated with ζ. Based on what we know about the true data generating process, can we say whether or not our guess for the data generating process suffers from endogeneity? Using simple algebra, we can see that:
so
Because x_1 and x_2 are correlated, a change in x_1 will yield a predictable change in ζ and thus our model suffers from endogeneity. When a model suffers from endogeneity, the estimates for the coefficients will not reflect the true causal relationships between the inputs and output variable. Note, this example demonstrates a case where endogeneity was the result of an omitted variable, but endogeneity can also occur in situations with reverse causality.
Clearly, the lack of access to causal variables complicates the causal interpretation of linear regression models built for predictive purposes. As an aside, it’s worth spending some time thinking about why a model can have high predictive accuracy without incorporating any of the actual causal factors.
An additional complication arises from multicollinearity (assumption #3). Multicollinearity describes a situation where the input variables are highly correlated, which makes the estimated coefficients for the model extremely unstable. When attempting to improve the accuracy of the predictive model, it’s quite common for data scientists to throw in many correlated features, exacerbating the challenges associated with multicollinearity in the process.
So, are linear models actually easier to interpret than other machine learning methods? I personally don’t think so. In fact, I feel the opposite. With black box models, the focus is on making sure the model will generalize well. The way you do that is by testing the model’s performance on a held out data set. Evaluating the accuracy of a model is pretty straightforward — the lower the error, the better. Because of the strong resemblance between the machine learning model and the data generating process specified in the stats equation, it’s tempting to overinterpret the coefficients in a linear regression model that was built for prediction, so data scientists must exercise restraint to avoid making inaccurate claims regarding the relationship between the input variables and the outcome variable. While linear models can be extremely powerful tools for investigating causality, using them effectively requires clear thinking with regards to the data generating process and a degree of nuance when interpreting the results, both subtle skills that are harder to acquire than is sometimes suggested.
