## Entropy of a general prob. measure [closed]

### Remember to vote up questions/answers you find interesting or helpful (requires 15 reputation points)

How is entropy of a general probability measure defined?

-
I can suggest en.wikipedia.org/wiki/Differential_entropy – falagar Jul 23 2010 at 13:10

2
Did you try Google and/or Wikipedia? – Qiaochu Yuan Jul 23 2010 at 15:14

1
I am casting the final vote to close, because even though the original question has generated some interesting details, it wasn't asked with enough background context., nor with indication of what the questioner already knows or has read. – Yemon Choi Jul 23 2010 at 20:31

## 3 Answers

The Entropy of a function $f$ with respect to a measure $\mu$ is

$$Ent_{\mu}(f)=\int f \log f d\mu - \int f d\mu \log(\int f d\mu )$$

The entropy of a probability distribution $P$ with respect to $\mu$ is given by $Ent(\frac{dP }{d\mu })$. I a not aware of a general definition that would not implie a reference (here it is $\mu$) measure ...

-

### You can accept an answer to one of your own questions by clicking the check mark next to it. This awards 15 reputation points to the person who answered and 2 reputation points to you.

It is not. If a probability measure on $\mathbb{R}$ is absolutely continuous and has density $f$, then "entropy" usually refers to the differential entropy, defined in the Wikipedia page falagar linked to. If the probability measure has discrete support, entropy is defined by an analogous formula, given in this Wikipedia page. In the most classical treatments, these are the only situations covered at all.

However, both of these are special cases of the more general notion of relative entropy that Helge and robin girard pointed out: in the continuous case the reference measure ($\nu$ in Helge's notation, $\mu$ in robin's notation) is Lebesgue measure, and in the discrete case the reference measure is counting measure on the support.

-

Entropy is not defined for a single probability measure!

Entropy is a relative thing, you define between two measure $\mu$ and $\nu$. Then the entropy is defined by $$\mathcal{E}(\mu,\nu) = \begin{cases} - \int w(x) \log(w(x)) d\nu(x),& d \mu = w d\nu; \\ -\infty, & otherwise.\end{cases},$$ I might have messed up the signs. $w$ is the Radon-Nikodym derivate of $\mu$ with respect to $\nu$.

-
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 15, "mathjax_display_tex": 2, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9403489828109741, "perplexity_flag": "middle"}