# Tagged Questions

The kullback-leibler tag has no wiki summary.

0answers
22 views

### Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?

I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...
1answer
82 views

### Is the square root of the symmetric Kullback-Leibler divergence a metric?

It is well known that the square root of the Jensen-Shannon divergence is a true metric, but how about the square root of symmetric KL: D(P||Q)+D(Q||P)? I have reasons to believe that it also is a ...
0answers
13 views

### Is there any meta-approach for variable selection based of measures of similarity between each two variables?

Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
0answers
29 views

### Comparing divergence from uniform distributions with differing supports (discrete)

Imagine we have a potentially biased coin and a potentially biased six-sided die and we want to know which is more biased than the other. Firstly, is this a reasonable goal? Could it make sense to ...
0answers
16 views

### Comparison of second-order probability matrices

I've searched over the site for matrix comparison questions, and it seems that the biggest response is 'what do you need it for'. I have two second-order probability matrices, in which the rows and ...
1answer
229 views

### Bhattacharyya distance for histograms

One of the ways to measure the similarity of two discrete probability distributions is the Bhattacharyya distance. In computer vision, for example, it is used to evaluate the degree of similarity ...
0answers
54 views

### Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where ...
1answer
86 views

### Orthogonal intersection of linear family and exponential family

I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question. In statistical maniolds ...
2answers
124 views

### Connection between Fisher metric and the relative entropy

Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way? D( p(\cdot , a+da) \| p(.,a) ) ...
1answer
49 views

### Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab

Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values. How do I ...
0answers
100 views

### Kullback-Leibler vs Hellinger Distance

I am working on this problem in which I have a dataset of n-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find k examples from the dataset that ...
0answers
75 views

### Estimating parameters using Kullback-Leibler and Nelder Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters ...
0answers
152 views

### What's the correct way to calculate information gain ratio?

I'm trying to implement information gain ratio[1] to find how much a variable affects contributes to class membership in a naive bayesian classifier. I hope to use this for both weighting and to find ...
0answers
57 views

### Kullback-Leibler Divergence in laymans language

Can some one please explain me in lay mans term what is KL divergence method for divergence and what advantage it has. Kindly help. Thank you
0answers
104 views

### Logistic regression, loss function and KL divergence

In decision theory, a loss function signature is supposed to be output space * output space -> error There seems to be many different definition of 'the ...
0answers
72 views

### On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
0answers
140 views

### Kullback-Leibler divergence for multivariate binomial distributions

I understand KL divergence abstractly, but I'm not exactly sure how you would calculate it for a multivariate binomial distribution (such as an Ising model on a random graph). If I am sampling 100 ...
1answer
284 views

### Kullback-Leibler divergence: negative values?

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc ...
0answers
151 views

### KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$. Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
0answers
96 views

### Sample distribution for Kullback-Leibler distance

For two $n$ dimensional multivariate normal distributions $X_{1}\sim N\left(\mu_{1},\Sigma_{1}\right)$ and $X_{2}\sim N\left(\mu_{2},\Sigma_{2}\right)$, the Kullback-Leibler distance is given by ...
0answers
38 views

### Lomax distributions - Kullback Leibler divergence

Does anyone know of a reference for an expression for the Kullback-Leibler divergence between two Lomax (Pareto II) distributions? Not really worried which way the Lomax is parameterized.
0answers
150 views

### Comparing symmetric KL divergence values

I am trying to understand how to compare KL divergence values. I have 3 sets of data, Good (G), Bad (B) and Unknown (U). I compute the symmetric form of KL for: Sym_GU = KL(G||B) + KL(B||G) Sym_BU ...
1answer
177 views

### Multivariate time series model evaluation with conditional moments

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula ...
2answers
173 views

### Maximum entropy sampler

I want to sample from a distribution which has fixed to a given values mean(=0), standard deviation(=1), skewness(=0) and kurtosis. I also want this distribution to be as general as possible, i.e. to ...
1answer
213 views

### Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are ...
0answers
134 views

### Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]],$$ where $L$ is some loss function and $\hat x$ is the ...
1answer
540 views

### Kullback-Leibler divergence

Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler ...
1answer
149 views

### Property of KL-divergence

Let $p_1$ and $p_2$ be two distinct probability distributions. Define $$L(q)=D(q||p_1)-D(q||p_2)$$ where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in ...
1answer
286 views

### Kullback–Leibler divergence between two wishart distributions

The result is shown in: [1] W.D. Penny, KL-Divergences of Normal, Gamma, Dirichlet, and Wishart densities, Available at: www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps But could anyone help ...
1answer
74 views

### How does one express the decrease in minimal type II error bound for each observation added?

Problem: I have a "classifier" that uses some arbitrary hypothesis test on observations from one of two known probability distributions: $P_0$ (null hypothesis $H_0$) is a zero-mean Gaussian ...
2answers
931 views

### Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
0answers
115 views

### Shall I use sqrt of J-S divergence or K-L divergence as a measure of deviation from uniform pdf?

Some colleagues are using 'normalized' K-L divergence to measure deviation of unit-area histogram (pseudo discrete pdf) from corresponding uniform distribution ($N$ equal-length bins). Maybe it is ...
2answers
229 views

### Similarity / dissimilarity of two large bimodal datasets

I am interested in assessing the divergence, or similarity or dissimilarity of 2 datasets that are the results of 2 different lidar instrument measurements. Each dataset has over 90,000 values and ...
3answers
2k views

### Measure of similarity or distance between two symmetric covariance matrices

are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimension)? I am thinking here of analogues to KL divergence of two probability ...
3answers
962 views

### How to compute the Kullback-Leibler divergence when the PMF contains 0s?

I have the following timeseries obtained using the data posted below. For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...
1answer
201 views

### How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
0answers
122 views

### How to to calculate the topic distribution of a document

I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
3answers
678 views

### Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and ...
1answer
1k views

### Kullback–Leibler vs Kolmogorov-Smirnov distance

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions. Is there ...
2answers
3k views

### KL divergence between two univariate Gaussians

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p). ...
4answers
2k views

### An adaptation of the Kullback-Leibler distance?

Look at this picture: If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
1answer
734 views

### Estimate the Kullback-Leibler divergence

I would like to be sure I am able to compute the KL divergence based on a sample. Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85. set.seed(937) theta <- ...
2answers
2k views

### Kullback-Leibler divergence - interpretation

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
1answer
244 views

### Properties of Battacharyya distance vs Kullback-Leibler divergence

What properties do these measures have and how can I determine which one is better for a given purpose? What are extreme cases where they differ a lot?
3answers
3k views

### Questions about KL divergence?

I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 31, "mathjax_display_tex": 2, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.8921208381652832, "perplexity_flag": "middle"}