Is it acceptable to run two linear models on the same data set?

For a linear regression with multiple groups (natural groups defined a priori) is it acceptable to run two different models on the same data set to answer the following two questions?

1. Does each group have a non-zero slope and non-zero intercept and what are the parameters for each within group regression?

2. Is there, regardless of group membership, a non-zero trend and non-zero intercept and what are the parameters for this across groups regression?

In R, the first model would be `lm(y ~ group + x:group - 1)`, so that the estimated coefficients could be directly interpreted as the intercept and slope for each group.The second model would be `lm(y ~ x + 1)`.

The alternative would be `lm(y ~ x + group + x:group + 1)`, which results in a complicated summary table of coefficients, with within group slopes and intercepts having to be calculated from the differences in slopes and intercepts from some reference. Also you have to reorder the groups and run the model a second time anyway in order to get a p-value for the last group difference (sometimes).

Does this using two separate models negatively affect inference in any way or this this standard practice?

To put this into context, consider x to be a drug dosage and the groups to be different races. It may be interesting to know the dose-response relationship for a particular race for a doctor, or which races the drug works for at all, but it may also be interesting sometimes to know the dose-response relationship for the entire (human) population regardless of race for a public health official. This is just an example of how one might be interested in both within group and across group regressions separately. Whether a dose-response relationship should be linear isn't important.

-

Are you sure that you want to use linear regressions? Dose-response relationships are almost never linear over a substantial dose range. – Michael Lew Jun 8 '12 at 5:24
@Michael, sorry, that was a bad choice of example, I guess. I am wondering about this in general. The particulars of dose-response relationships shouldn't get in the way. I edited the question to note this. – Jdub Jun 8 '12 at 12:04
Have you considered a random intercept, random slope model? – Max Jun 8 '12 at 15:37

1 Answer

Let me start by saying that I think your first question and first R model are incompatible with each other. In R, when we write a formula with either `-1` or `+0`, we are suppressing the intercept. Thus, `lm(y ~ group + x:group - 1)` prevents you from being able to tell if the intercepts significantly differ from 0. In the same vein, in your following two models, th `+1` is superfluous, the intercept is automatically estimated in R. I would advise you to use reference cell coding (also called 'dummy coding') to represent your groups. That is, with $g$ groups, create $g-1$ new variables, pick one group as the default and assign 0's to the units of that group in each of the new variables. Then each new variable is used to represent membership in one of the other groups; units that fall within a given group are indicated with a 1 in the corresponding variable and 0's elsewhere. When your coefficients are returned, if the intercept is 'significant', then your default group has a non-zero intercept. Unfortunately, the standard significance tests for the other groups will not tell you if they differ from 0, but rather if they differ from the default group. To determine if they differ from 0, add their coefficients to the intercept and divide the sum by their standard errors to get their t-values. The situation with the slopes will be similar: That is, the test of $X$ will tell you if the default group's slope differs significantly from 0, and the interaction terms tell you if those groups' slopes differ from the default groups. Tests for the slopes of the other groups against 0 can be constructed just as for the intercepts. Even better would be to just fit a 'restricted' model without any of the group indicator variables or the interaction terms, and test this model against the full model with `anova()`, which will tell you if your groups differ meaningfully at all.

These things having been said, your main question is whether doing all of this is acceptable. The underlying issue here is the problem of multiple comparisons. This is a long-standing and thorny issue, with many opinions. (You can find more information on this topic on CV by perusing the questions tagged with this keyword.) While opinions have certainly varied on this topic, I think no one would fault you for running many analyses over the same dataset provided the analyses were orthogonal. Generally, orthogonal contrasts are thought about in the context of figuring out how to compare a set of $g$ groups to each other, however, that is not the case here; your question is unusual (and, I think, interesting). So far as I can see, if you simply wanted to partition your dataset into $g$ separate subsets and run a simple regression model on each that should be OK. The more interesting question is whether the 'collapsed' analysis can be considered orthogonal to the set of individual analyses; I don't think so, because you should be able to recreate the collapsed analysis with a linear combination of the group analyses.

A slightly different question is whether doing this is really meaningful. Image that you run an initial analysis and discover that the groups differ from each other in a substantively meaningful way; what sense does it make to put these divergent groups together into a discombobulated whole? For example, imagine that the groups differ (somehow) on their intercepts, then, at least some group does not have a 0 intercept. If there is only one such group, then the intercept for the whole will only be 0 if that group has $n_g=0$ in the relevant population. Alternatively, lets say that there are exactly 2 groups with non-zero intercepts with one positive and one negative, then the whole will have a 0 intercept only if the $n$'s of these groups are in inverse proportion to the magnitudes of the intercepts' divergences. I could go on here (there are many more possibilities), but the point is you are asking questions about how the groups sizes relate to the differences in parameter values. Frankly, these are weird questions to me.

I would suggest you follow the protocol I outline above. Namely, dummy code your groups. Then fit a full model with all the dummies and interaction terms included. Fit a reduced model without these terms, and perform a nested model test. If the groups do differ somehow, follow up with (hopefully) a-priori (theoretically driven) orthogonal contrasts to better understand how the groups differ. (And plot--always, always plot.)

-
default
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 7, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9395275712013245, "perplexity_flag": "middle"}