When linear combinations of independent random variables are still independent?

We have a random vector $X=(X_j)_{j=1,...,n}$, whose components $X_j$ are mutually independent. We build a new random vector $Y=A X+b$, with $Y=(Y_k)_{k=1,...,m}$, $A=(a_{k,j})_{k=1,...,m;j=1,...,n}$, $b=(b_k)_{k=1,...,m}$.

When the components of $Y$ are still mutually independent random variables? How to prove the result? I suppose this can be related with the rank of $A$ but I'm not sure and I cannot find a way to prove if I don't make some special assumption about the law of $X$.

I know $U$ and $V$ are independent iff $f(U)$ and $g(V)$ are independent for each and every pair of measurable function $f$ and $g$. This can help, I think, when, for example, $Y_1=X_1+X_2$ and $Y_2=X_3+X_4$.

But when does $Y_1 = X_1 + X_2$ and $Y_2 = X_1 - X_2$? Or when does $Y_1=X_1 + X_2$ and $Y_2=X_5$?

None of the texts I searched discuss seriously this topic, but leave it in a certain way to "intuition"?

Thanks

-

I don't think there's a general result here unless the $X_i$ are Gaussian. – Qiaochu Yuan May 30 '11 at 13:14
As you've noted, the general result is true if $A = a_1 \oplus a_2 \oplus \cdots \oplus a_p$ for some $p$ where the $a_i$ are row vectors. – cardinal May 30 '11 at 13:48

3 Answers

If $A = a_1 \oplus a_2 \oplus \cdots \oplus a_m$, for $m \leq n$, where $a_i$ are row vectors of dimension $n_i$ such that $\sum_{i=1}^m n_i = n$ and $\oplus$ denotes the direct sum, then the random vector $Y$ has independent coordinates.

This is not hard to see since $Y_1$ is measurable with respect to $\sigma(X_1, \ldots X_{n_1})$, $Y_2$ is measurable with respect to $\sigma(X_{n_1+1}, \ldots, X_{n_1+n_2})$, etc., and these $\sigma$-algebras are independent since the $X_i$ are independent (essentially, by definition).

Obviously, this result still holds if we consider matrices that are column permutations of the matrix $A$ described above. Indeed, as we see below, in the case where the distribution of each $X_i$ is non-normal (though perhaps depending on the index $i$), this is essentially the only form that $A$ can take for the desired result to hold.

In the normal-distribution case, as long as $A A^T = D$ for some diagonal matrix $D$, then the coordinates of $Y$ are independent. This is easily checked with the moment-generating function.

Suppose $X_1$ and $X_2$ are iid with finite variance. If $X_1 + X_2$ is independent of $X_1 - X_2$, then $X_1$ and $X_2$ are normal distributed random variables. See here. This result is known as Bernstein's theorem and can be generalized (see below). A proof can be found in Feller or here (Chapter 5).

In the case where $A$ cannot be written as a direct sum of row vectors, you can always cook up a distribution for $X$ such that $Y$ does not have independent coordinates. Indeed, we have

Theorem (Lukacs and King, 1954): Let $X_1, X_2, \cdots, X_n$ be $n$ independently (but not necessarily identically) distributed random variables with variances $\sigma_i^2$, and assume that the $n$th moment of each $X_i(i = 1, 2, \cdots, n)$ exists. The necessary and sufficient conditions for the existence of two statistically independent linear forms $Y_1 = \sum^n_{i=1} a_i X_i$ and $Y_2 = \sum^n_{i=1} b_i X_i$ are

1. Each random variable which has a nonzero coefficient in both forms is normally distributed, and
2. $\sum^n_{i=1} a_i b_i \sigma^2_i = 0$.
-

Great answer. (And thanks for the Lukacs and King result.) – Did Jun 5 '11 at 8:21

Did you find this question interesting? Try our newsletter

email address

I think little can be said in general, apart from the rather trivial case in which each of the resulting $y_i$ depends on non-intersecting subssets of $X$ (what would correspond to each column of A having no more than one non-null value).

Some weaker result can be obtained by considering non-correlation instead of independence (or, what would equivalent, restricting to gaussian variables), because the covariances matrices are simply related

$C_Y = A \; C_X A^t$

By assumption, $C_X$ is diagonal, and we want fo find out for which $A$ matrices $C_Y$ is also diagonal. Again, little can be said in general, more than just that. If we restrict further the assumptions, and assume that $x_i$ are iid (or just equal variances), then we get that orthogonality of $A$ in sufficient (but not necessary).

In particular, for the case $Y_1 = X_1 + X_2$, $Y_2 = X_1 - X_2$, $Y$ is not guaranteed to be uncorrelated, unless we assume $x_i$ are iid (or have same variance).

-

Something relatively strong can be said. See my updated answer. – cardinal May 30 '11 at 14:46
@cardinal: At first sight, that seems practically equivalent to what I stated. The normal assumption is equivalent to use correlation in place of independence. And the summation seems as the non-diagonal terms in my $C_y$ to me. – leonbloy May 30 '11 at 16:38
It appears much stronger to me. – cardinal May 30 '11 at 16:42
Ah, yes, I see it now. – leonbloy May 30 '11 at 17:23

The Skitovich-Darmois theorem A (Skitovich(1953), Darmois(1953), see also A. Kagan, Yu. Linnik, and C.R. Rao (1973, Ch.3)). Let $\xi_j$, where $j=1, 2,\dots, n,$ and $n\geq 2$, be independent random variables. Let $\alpha_j, \beta_j$ be nonzero constants. If the linear statistics $L_1=\alpha_1\xi_1+\cdots+\alpha_n\xi_n$ and $L_2=\beta_1\xi_1+\cdots+\beta_n\xi_n$ are independent, then all random variables $\xi_j$ are Gaussian.

-
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 82, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9429833292961121, "perplexity_flag": "head"}