## Is there a relationship between Entropy of a fininte distrete probability distribution and the squre sum of the values of probability mass function of that distribution?

### Remember to vote up questions/answers you find interesting or helpful (requires 15 reputation points)

Sorry for the long title. What I mean is that for two vectors (a_1,...,a_n) and (b_1,...,b_n) with the property $a_i,b_i \geq 0$ and $\sum a_i =\sum b_i =1$.

If $-\sum a_ilog(a_i) > -\sum b_ilog(b_i)$ implies $\sum a_i^2 < \sum b_i^2$ or something similar?

-

Certainly not as you ask it. If h(a)>h(b) implied |a|_2<|b|_2, then h(b)>h(a) would imply |b|_2<|a|_2 and you would get h(a)>h(b) if and only if |a|_2<|b|_2.   This would mean that the entropy and the L2 norm are measuring the same thing (they're not). – Anthony Quas Jan 6 at 4:07
@Anthony I am not sure, if it can be ture under this specific condiction. – gstar2002 Jan 6 at 10:42
and I think for the case n = 2, it is true. – gstar2002 Jan 6 at 21:51

## 1 Answer

Of course, in general it is not true that inequality between entropies implies the same inequality between $\ell^2$ norms. Although this is true for $n=2$, it fails already for $n=3$ (as the entropy $H(p_1,p_2,p_3)$ is obviously not constant on the level curve determined by conditions $\sum p_i=1$ and $\sum p_i^2=Const$).

Nonetheless, there is a deep link between these two quantities. In order to explain it, it is better to somewhat change the viewpoint. Namely, given two probability distributions $P=(p_1,\dots,p_n)$ and $Q=(q_1,\dots,q_n)$ (for simplicity I assume that all $p_i,q_i$ are strictly positive), the corresponding Kullback-Leibler deviation is defined as $$I(Q|P) = \sum p_i \log \frac{p_i}{q_i} \;,$$ and the so-called information energy as $$\chi^2(Q,P) = \sum \biggl(\frac{p_i}{q_i}-1\biggr)^2 q_i \;.$$ Both these quantities measure "closedness" of $P$ and $Q$ (although they are not distances) and are monotone invariants of the pair $(Q,P)$ (they do not increase under quotient maps). If $Q=U_n$ is the uniform distribution, then these quantities are, up to linear rescaling, precisely the entropy and the $\ell^2$-norm of $P$, respectively, as $$I(U_n|P) = \log n - H(P)$$ and $$\chi^2(U_n,P) = n \sum p_i^2 - 1 \;.$$

Now, the link between $I$ and $\chi^2$ is provided by the fact that in naturally defined infinitesimal limits the Kullback-Leibler deviation $I$ and the information energy $\chi^2$ coincide and produce the Fisher information metric. One can read more about this in the corresponding wiki articles, in the old book Statistical decision rules and optimal inference by Čencov (AMS, 1982), or in more recent publications on information geometry.

-

@ R W, first off,  thanks a lot for the very informative reply! But I still have one more question, what do you mean by "naturally defined infinitesimal limits "? do you mean n go to infinite? – gstar2002 Jan 15 at 22:11
No - infinitesimal here means that time goes to 0. The number of points remains fixed, but instead of a single distribution on $n$ points one looks at a family of distributions parametrized, say, by the interval $[0,1]$ (in other words, about a path in the space of distributions).   – R W Jan 16 at 0:32
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 25, "mathjax_display_tex": 4, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9148876070976257, "perplexity_flag": "head"}