# What is the covariance of mixture Bernoulli distribution?

For mixture of multivariate Bernoulli distribution we have that,

$$p(x|\mu,\pi) =\Sigma_{k=1}^{K}\pi_kp(x|\mu_k)$$ where $$p(x|\mu_k) = \prod_{i=1}^{D}\mu_{ki}^{x_i}(1-\mu_{ki})^{1-x_i}$$

I read it from the book that

$$E[x] = \Sigma_{i=1}^{K}\pi_k\mu_k$$ $$cov[x] = \Sigma_{k=1}^{K}\pi_k(\Sigma_k+\mu_k\mu_k^T) - E[x]E[x]^T$$

The mean is trivial to prove, however I can't find proof for the covariance and I don't know how to prove it.

Can anyone help?

-

## 1 Answer

You know that, in general, $cov[x] = E[x x^T] - E[x]E[x]^T$ and you want to compute the first term. We can use the property: $E[g(X)] = E [ E(g(X)|k]]$

But $E(x x^T | k)$ (i.e., fixing the population 'k' index of the mixing), is $\Sigma_k+\mu_k\mu_k^T$ (don't let the $k$ subindexes confuse you: they are the first and seconds moments of $x$, for a fixed population index $k$).

Now, the above is a function ok $k$, and its expectation is $\sum_{k=1}^K \pi_k (\Sigma_k+\mu_k\mu_k^T)$

Notice that this result is not really connected to the Bernoulli distribution, it's valid in general.

-

Hi, can I know why $$E(xx^T|k) = \Sigma_k+\mu_k\mu_k^T$$ I think that's the thing that confuses me – Jing Nov 30 '12 at 4:55
@Jing: It's just the same first equation of my answer, rearranged, and applied not to $x$ (the mixed variable) but to $x_k$ (one of the $K$ components of the mixture) – leonbloy Nov 30 '12 at 11:50
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 12, "mathjax_display_tex": 5, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": false, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9498900771141052, "perplexity_flag": "head"}