text:
\section{Introduction}

Uncertainty quantification is a fundamental desiderata of statistical inference and data science.  In supervised learning settings it is common to quantify uncertainty with either conditional entropy or mutual information (MI).
Suppose we are given a pair of random variables $(X, Y)$, where $X$ is $d$-dimensional vector-valued and $Y$ is a categorical variable of interest. Conditional entropy $H(Y|X)$ measures the uncertainty in $Y$ on average given the value of $X$. On the other hand, mutual information quantifies the shared information between $X$ and $Y$.
Although both parameters are readily estimated when $X$ and $Y$ are low-dimensional and ``nicely'' distributed, an important problem arises in measuring these quantities from higher-dimensional data in a nonparametric fashion \citep{mines}.

We address the problem of high-dimensionality by proposing a decision forest method for estimating conditional entropy under the framework that $X$ is any  $d$-dimensional random vector and $Y$ is  categorical. Because we restrict $Y$ to be categorical, we can easily compute the maximum-likelihood estimate of $H(Y)$ and hence use our estimator to compute mutual information.

We present Uncertainty Forests (UF) to estimate these information-theoretic quantities. UF combines quantile regression forests~\citep{Meinshausen06} with honest sampling~\citep{denil14}, and introduces a finite sample correction to improve performance while preserving consistency.
We prove that our estimation technique is consistent (meaning that it converges in probability to the true estimates of conditional entropy and mutual information, under mild conditions on the joint distribution of $X$ and $Y$), and demonstrate via simulations that UF performs well in both low- and high-dimensional settings when estimating conditional distributions, conditional entropy, and mutual information.

Finally, we provide a real-world application of our estimator by measuring information about the various properties of \textit{Drosophila} neurons that are contained in neuron cell types. Scientific prior knowledge poses various relationships between particular features and the cell type, which are supported by the mutual information and conditional mutual information estimates of UF.

\section{Background}
\subsection{Problem Formulation}

Suppose we are given two random variables $X$ and $Y$ with support sets $\mathcal{X}$ and $\mathcal{Y}$, respectively. Let $x$, $y$ denote specific values that the random variables take on and $p(x), p(y)$ be the probabilities of $X=x$ and $Y=y$. The unconditioned Shannon entropy of $Y$ is $H(Y) = -\sum_{y \in \mathcal{Y}} p(y) \log p(y)$.
Analogously, conditional entropy is $ H(Y|X) = \sum_{x \in \mathcal{X}}{p(x)H(Y|X = x)} = -\sum_{x \in \mathcal{X}}{p(x)\sum_{y \in \mathcal{Y}}{p(y|x)\log{p(y|x)}}}$, where $p(y|x)$ is the conditional probability that $Y=y$ given $X=x$ and $H(Y|X = x)$ is the entropy of $Y$ conditioned on $X$ equaling particular value $x$. This quantity represents the average uncertainty in $Y$ having observed $X$. In the case of a continuous random variable, the sum over the corresponding support is replaced with an integral, and probability mass functions are replaced by densities. For the remainder of this work, we assume that $\mathcal{X} = \mathbb{R}^d$ and $\mathcal{Y} = [K] = \{1, ..., K\}$ for positive integers $d$ and $K$.
Mutual information, $I(X; Y)$, can be computed from conditional entropy $ I(X; Y) = H(Y) - H(Y|X) = H(X) - H(X|Y)$.
Mutual information has many appealing properties, such as symmetry, and is widely used in data science applications \citep{mixedksg}.

\subsection{Related Work}

A common approach to estimating mutual information relies on the \textit{3-H} principle \citep{mixedksg},
$I(X; Y) = H(Y) + H(X) - H(X, Y)$,
where $H(X, Y) = -\sum_{x \in \mathcal{X}} \sum_{y \in \mathcal{Y}} p(x,y) \log p(x,y)$ is the Shannon entropy of the pair $(X, Y)$. Procedures typically estimate unconditional entropy for the three separate random variables. Examples of these include kernel density estimates and ensembles of $k$-NN estimators \citep{Beirlant1997, Leonenko08estimationof, berrett2019, sricharan}. One method in particular, the KSG estimator, popular because of its excellent empirical performance, improves $k$-NN estimates via heuristics \citep{ksg}. Other approaches include binning~\citep{binning} and von Mises estimators~\citep{vonmises}. Unfortunately, many modern datasets contain mixtures of continuous inputs $X$ and discrete outputs $Y$. In these cases, few of the above methods work well, as while individual entropies $H(X)$ and $H(Y)$ are well-defined, $H(Y, X)$ is either ill-defined or not easily estimated, thus rendering the \textit{3-H} approach intractable~\citep{mixedksg}.

A recent approach, referred to as Mixed KSG, modifies the KSG estimator to improve its performance in various settings, including mixed continuous and categorical inputs \citep{mixedksg}. Computing both mutual information and conditional entropy becomes difficult in higher dimensional data. Numerical summations or integration become computationally intractable, and nonparametric methods (for example, $k$-nearest neighbor, kernel density estimates, binning, Edgeworth approximation, likelihood ratio estimators) typically do not scale well with increasing dimensions \citep{mines, gaoetal}. While a recently proposed neural network approach, MINE \citep{mines}, addresses high-dimensional data, the method does not return an estimate of the posterior $p(y \mid x)$, which is useful in uncertainty quantification. Additionally, MINE requires that the network of choice be expressive enough so that the Donsker-Varadhan representation can be used to approximate the mutual information, an assumption that is difficult to inspect. Other theoretically analyzed deep approaches to mutual information estimation can make stringent assumptions on the distribution of learned weights \citep{deep1}. Finally, in the interest of an ``out-of-the-box" method, another common pitfall of deep approaches is sensitivity to hyperparameter choice.

Another popular ad-hoc approach relies on estimation of the posterior probability $p(y \mid x)$. Calibration methods, such as isotonic regression, map classification scores, whether they are scalar projections of the data in linear discriminant analysis or empirical posteriors from random forests, to ``calibrated" posteriors using a held-out set. A thorough theoretical analysis of such methods has yet to be addressed \citep{isotonic}.

\subsection{CART Random Forest} \label{sec:cart} Classification and Regression Tree (CART) Random Forest is a robust, powerful algorithm that leverages ensembles of decision trees for classification and regression tasks \citep{Breiman2001}. In a study of over 100 classification problems, F\'ernandez-Delgado et al. \citep{Delgado14} showed that random forests have the overall best performance when compared 178 other classifiers. Furthermore, random forests are highly scalable; efficient implementations can build a forest of 100 trees from 110 Gigabyte data ($n$ = 10,000,000, $d$ = 1000) in little more than an hour \citep{scalablerf}.

Random forest classifiers are instances of a bagging classifiers, in which the base classifiers are decision trees. A decision tree first learns a partition of feature space, then learns constant functions within each part to perform classification or regression. Precisely, $ \mc{L} $ is called a partition of feature space $ \mc{X} $, if for every $ L, L' \in \mc{L}$ with $L \neq L'$, $L \cap L' = \varnothing$, and $ \bigcup_{L \in \mc{L}} L = \mc{X} $. This $ \mc{L} $ is learned on data $\{X_i, Y_i\}_{i=1}^n$ by recursively splitting a randomly selected subsample along a single dimension of the input data based on an impurity measure \citep{Breiman2001}, such as Gini impurity or entropy. The trees are grown until nodes reach a certain criterion (for example, a minimum number of samples). The bottom-most nodes are called ``leaf nodes''. Additionally, only a random subset of dimensions of $X$ are considered for the split at each node.

Given a partition $ \mc{L} $, let $L(x)$ be the part of $ \mc{L} $ to which $x \in \mc{X}$ belongs. Letting $\ensuremath{\mathbbm{1}}[\cdot]$ be the indicator function, a possible predictor function $\hat{g}$ for classification is $\hat{g}(x) = \operatornamewithlimits{argmax}_{y \in \mc{Y}} \sum_{i=1}^n \ensuremath{\mathbbm{1}}[Y_i = y \text{ and } X_i \in L(x)]$. This is the plurality vote among the training data in the leaf node of $x$. For a regression task, a corresponding predictor $\hat{\mu}$ is $\hat{\mu}(x) = \frac{\sum_{i = 1}^n Y_i \cdot \ensuremath{\mathbbm{1}}[X_i \in L(x)]}{\sum_{i = 1}^n \ensuremath{\mathbbm{1}}[X_i \in L(x)]}$, which is the average $Y$ value for the training data in $L(x)$. For a forest, $B$ trees are learned on randomly subsampled points. These points used in tree construction are called the `in-bag' samples for that tree, while those that are left out are called `out-of-bag' samples.

\vspace{-6pt}

\subsection{Honest Sampling, Balanced Trees, and Random-Split Trees} In decision forest algorithms, because data are typically split to maximize purity within child nodes, the posteriors estimated in each cell tend to be biased toward certainty. Honesty, as shown in \citet{cart}, helps in bounding the bias of tree-based estimates of posterior class probabilities \citep{Athey2019-uw,Wager2018}. \citet{Wager2018} describes honest trees as those for which any particular training example $(X_i, Y_i)$ is used to partition feature space, or to estimate the quantity of interest, but not both. This property can be achieved in (at least) two ways. The first method is splitting the observed sample into two sets, one set  for learning the partitions of feature space $\mathcal{X}$, and one set to make vote on the plurality or average within each leaf node.
We refer to them as the  ``partition" set $\mathcal{D}^{\text{P}}$ and ``voting" set $\mathcal{D}^{\text{V}}$, respectively.
(\citet{denil14} called them ``structure'' and ``estimation'', which we avoid because both sets are used to estimate different quantities).

For example, say we wish to estimate the conditional mean function $\mu(x) = \mathbb{E}[Y \mid X = x]$. Let $\mathcal{L}$ be a partition of feature space $\mc{X}$, as described in Section \ref{sec:cart}. Letting $m < n$, such an $\mc{L}$ can be learned via a decision tree with $\mathcal{D}^{\text{P}} = \{(X_1,Y_1),...,(X_m,Y_m)\}$. This leaves $\mathcal{D}^{\text{V}} = \{(X_{m+1},Y_{m+1}),...,(X_n,Y_n)\}$. Letting $L(x)$ be the part of $\mathcal{L}$ to which $x$ belongs, the conditional mean estimate can be $\hat{\mu}(x) = \frac{\sum_{\mathcal{D}^{\text{V}}} Y_i \cdot \ensuremath{\mathbbm{1}}[X_i \in L(x)]}{\sum_{\mathcal{D}^{\text{V}}} \ensuremath{\mathbbm{1}}[X_i \in L(x)]} = \frac{\sum_{i = m + 1}^n Y_i \cdot \ensuremath{\mathbbm{1}}[X_i \in L(x)]}{\sum_{i = m + 1}^n \ensuremath{\mathbbm{1}}[X_i \in L(x)]}$.
This method can be applied at the forest level, that is, using the same partition points to learn every decision tree, or at the tree level, in which the data is randomly partitioned into partition and voting sets in each tree.
\citet{denil14} has shown that tree level splitting increases performance for low sample sizes, while for higher sample sizes the performance difference is indistinguishable. A second way of achieving honesty is by ignoring the responses of interest $Y_i$ and only using $X_i$ and any auxiliary variables $W_i$ to place splits in each decision tree. \citet{Wager2018} proposes an example of this method in estimating heterogeneous treatment effects via random forest.

Balanced trees are described by \citet{Wager2018} and \citet{Athey2019-uw} as $\alpha$-regular, meaning that at each split in a tree, at least an $\alpha$ fraction of the data are placed in each child node. Finally, trees for which each dimension has a nonzero chance of being used for the split are called random-split. This can be achieved, for example, by randomly choosing one candidate dimension uniformly at each split. For honest, balanced, random-split trees, consistency can be shown for a very general class of random forest estimates \citep{Athey2019-uw}.

\section{Uncertainty Forests}
\label{sec:uf}

Given observations $\mathcal{D}_n = \{(X_1, Y_1), ..., (X_n, Y_n)\}$, the goal is to estimate conditional entropy $H(Y|X) = \mathbb{E}_{X'} [H(Y|X = X')$ and use that to estimate mutual information. UF provides a consistent estimate of the conditional entropy, and an empirically non-biased estimate with sufficient sample size.

UF partitions the data into three sets: a partition set $\mathcal{D}^{\text{P}}$, a voting set $\mathcal{D}^{\text{V}}$, and evaluation set $\mathcal{D}^{\text{E}}$. $\mathcal{D}^{\text{P}}$ and $\mathcal{D}^{\text{V}}$ are used to learn low-bias posteriors $\hat{p}(y \mid x)$, and consequently the function $\hat{H}(Y \mid X = x)$. $\mathcal{D}^{\text{E}}$ will be used to estimate its expectation $\ensuremath{\mathbb{E}}_{X'}[H(Y \mid X = X')]$. Note that for forest-level evaluation, $\mathcal{D}^{\text{E}}$ {\bf need not be labeled}, as only the $x$ values are necessary to evaluate the function $H(Y \mid X = x)$. If data is difficult to label, UF can effectively leverage unlabelled data in a semisupervised fashion.

\subsection{Algorithm}

\begin{enumerate}[start=0]
    \item Set hyper-parameters including tree convergence criteria, number of trees $B$, and finite sample correction coefficient $\kappa$, and (possibly unlabelled) evaluation set $\mathcal{D}^{\text{E}}$.
    \item \textbf{Build Trees}\\ For each tree $b$ from 1 to $B$ (the maximum number of trees):
    \begin{enumerate}
        \item Randomly subsample $s \leq n$ data points from $\mathcal{D}_n - \mathcal{D}^{\text{E}}$.
        \item Randomly split the $s$ data points into a partition set $\mathcal{D}^{\text{P}}_b$ and voting set $\mathcal{D}^{\text{V}}_b$.
        \item Using $\mathcal{D}^{\text{P}}_b$, learn a decision tree partition $ \mc{L}_b $. $L_b(x)$ is the leaf node of $ \mc{L}_b $ that $x$ ``falls" into.
        \item Using the $\mathcal{D}^{\text{V}}_b$, letting $N_b(x) = \sum_{i \in \mathcal{D}^{\text{V}}}\ensuremath{\mathbbm{1}}[X_i \in L_b(x)]$ be the leaf size of $L_b(x)$, estimate the conditional probability of $Y = y$ given $X = x$ by $\tilde{p}_b(y \mid x) = \frac{1}{N_b(x)} \sum_{i \in \mathcal{D}^{\text{V}}} \ensuremath{\mathbbm{1}}[Y_i = y \text{ and } X_i \in L_b(x)]$.
        This is the empirical frequency of $y$ (given by the voting data) in the leaf node of $x$.
        \item When $Y$ is categorical, all samples in a leaf estimator may belong to one class even though the probabilities for other classes are nonzero, which biases finite-sample estimates of the conditional distribution. UF remedies this by adapting a robust finite sampling technique described in Vogelstein et al. \citep{Vogelstein13}. We first replace all zero probabilities with $1/ (\kappa N_b(x))$ (where $\kappa > 0$ is some suitably chosen constant), and renormalize the probabilities. The finite-sample corrected estimate is thus $\hat{p}_b(y \mid x) = \frac{\tilde{p}_b(y \mid x)}{\sum_{k=1}^K \tilde{p}_b(k \mid x)}$.
        If $N_b(x) \overset{P}{\rightarrow} \infty$, then $\hat{p}_b(y \mid x)$  approaches $\tilde{p}_b(y \mid x)$.
    \end{enumerate}
    \item \textbf{Estimate conditional entropy function}
        \begin{enumerate}
            \item Average all of the posterior estimates from each tree: $\hat{p}(y \mid x) = \frac{1}{B} \sum_{b = 1}^B \hat{p}_b(y \mid x)$.
    \item Set the conditional entropy function estimator $\hat{H}(Y\mid X = x) = -\sum_{y \in [K]} \hat{p}(y \mid x) \log \hat{p}(y \mid x)$.
    \end{enumerate}
    \item \textbf{Compute conditional entropy and mutual information}
    \begin{enumerate}
        \item Evaluate at every point in $\mathcal{D}^{\text{E}}$ and average to yield $\hat{H}(Y \mid X) = \frac{1}{|\mathcal{D}^{\text{E}}|} \sum_{i \in \mathcal{D}^{\text{E}}} \hat{H}(Y\mid X =
        X_i)$.
    \item Letting $\hat{p}(y) = \frac{1}{n} \sum_{i \in \mathcal{D}_n} \ensuremath{\mathbbm{1}}[Y_i = y]$, estimate $H(Y)$ with $\hat{H}(Y) = -\sum_{y \in [K]} \hat{p}(y) \log \hat{p}(y)$.
    \item Let $\hat{I}(X; Y) = \hat{H}(Y) - \hat{H}(Y \mid X)$.
    \end{enumerate}
\end{enumerate}

UF uses basic CART for tree construction, which has a number of hyper-parameters. These include which objective function to optimize for each split, how many samples to use to generate each tree ($s$),
minimum samples in a leaf before splitting ($k$),
maximum tree depth, and number of trees ($B$). Random forest has been shown in practice to be very robust to hyperparameters \citep{probstTunability}. For UF, we do not impose a maximum depth and use general rules-of-thumb for the other choices (minimize Gini impurity, $k=1$, $B = 300$). We use $|\mc{D}^{\text{P}}| = 0.4 \cdot n$, $|\mc{D}^{\text{V}}| = |\mc{D}^{\text{E}}| = 0.3 \cdot n$ in experiments ($s = |\mc{D}^{\text{P}}| + |\mc{D}^{\text{V}}|$). The UF pseudocode is described in detail in the supplementary material. In experiments, we find that letting the evaluation set $\mathcal{D}^{\text{E}}$ be randomly sampled at the tree-level performs better than holding out a set that is common for all trees. We also use $\kappa = 3$, after sweeping over choices in $[0.1, 10]$. Letting $T$ be the number of threads, the learning time complexity is $O(Bdn \log^2(n))$, the memory complexity is $O(B(nd + K))$, and the storage complexity is $O(Bn)$. \cite{sporf}

\subsection{Consistency of Uncertainty Forest Estimates}

Under mild conditions, UF provides a consistent estimate of conditional probability, conditional entropy, and mutual information. All proofs are collected in the supplementary material. Assume that $\mathcal{Y}$ is discrete ($\mathcal{Y} = [K]$), and that $\mathcal{X} = \mathbb{R}^d$. Let $\hat{H}_n(Y \mid X)$ be the conditional entropy estimate and $\hat{I}_n(X; Y)$ be the mutual information estimate, now indexed by $n$ for explicitness. Suppose that $|\mathcal{D}^{\text{E}}| = \gamma n$ for $0 < \gamma < 1$. Suppose that the subsample size $s_n$ is chosen such that $s_n \rightarrow \infty$ and $\frac{s_n}{n} \rightarrow 0$. Consider the following assumptions.
\begin{assumption}
    \label{thm:support}
    Suppose that $X$ is supported on $\mathbb{R}^d$ and has a density which non-zero almost everywhere. This is equivalent to have $X$ be uniformly distributed in $[0, 1]^d$ due to the monotone invariance of trees \citep{denil14}.
\end{assumption}
\begin{assumption}
    \label{thm:lipshitz}
    Suppose that for each $y \in [K]$ the conditional probability $p(y \mid x)$ is Lipshitz continuous on $\mathbb{R}^d$.
\end{assumption}
\begin{assumption}
    \label{asm:bigcells}
    Each tree $b$ is constructed such that for all $x \in \mathcal{X}, \epsilon > 0$, $P[N_b(x) < \epsilon] \rightarrow 0$ as $n \rightarrow \infty$.
\end{assumption}
\begin{theorem}[Consistency of the conditional entropy estimate] Given Assumptions  \ref{thm:support}-\ref{asm:bigcells}, the conditional entropy estimate is consistent as $n \rightarrow \infty$, that is, $\hat{H}_n(Y \mid X) \overset{P}{\rightarrow} H(Y \mid X)$.
    \label{thm:cond_ent}
\end{theorem}
This result states that the estimate $\hat{H}(Y \mid X)$ is arbitrarily close in probability to the true $H(Y \mid X)$ for sufficiently large $n$. As a simple consequence, due to the consistency of the maximum-likelihood estimate $\hat{H}(Y)$ for $H(Y)$ based on empirical frequencies, we have Theorem \ref{thm:mutual_info}. Both results rely heavily on recent theoretical work by \citet{Athey2019-uw}.
\begin{theorem}[Consistency of the mutual information estimate] With the same assumptions as in Theorem~\ref{thm:cond_ent}, $\hat{I}_n(X;Y)\overset{P}{\rightarrow} I(X; Y)$ as $n \rightarrow \infty$.
\label{thm:mutual_info}
\end{theorem}

\section{Simulation Results}

\subsection{Posterior Probability Experiments}

Consider the following setting: let each $Y_i$ be Bernoulli with 50\% probability to be either $+1$ or $-1$; let each $X_i$ be normally distributed with mean $Y_i \times \mu$ and variance one, where $\mu$ is a parameter controlling effect size. A CART random forest, a CART forest with isotonic regression calibration \citep{isotonic}, and UF (using both honest sampling and finite sample correction) are trained on data drawn from the above distribution with $\mu = 1$. We estimate posterior distributions and plot the posterior for class $1$ in Figure \ref{posterior}.
\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{fig1.pdf}
  \caption{Comparison of estimated posterior distributions using random forest algorithms. Left plots show posterior distribution of $Y=1$ given $x$ from CART, honest, and UF. Ten trials are plotted for each algorithm, the mean is highlighted.
  Right-most plot shows variance, over 100 trials, of posterior estimates vs $x$. $\mu = 1, n = 6000$ for all plots. }
  \label{posterior}
\end{figure}
As $x$ increases, the true probability that $Y$ is one givne $X = x$ increases. Thus, unsurprisingly, all random forest algorithms have $\hat{P}(Y = 1 \mid X = x)$ decrease to $0$ as $x$ becomes more negative, and increase to $1$ as $x$ becomes more positive.
However, the posterior estimated from UF has significantly lower variance than both normal CART forests and isotonic regression forests (Figure \ref{posterior}, right).

\subsection{Conditional Entropy Experiments}

These better posterior estimates from UF carry over to better estimates of conditional entropy. Figure \ref{fig:convergence}A shows that UF estimates converge to the truth as sample size increases, whereas isotonic regression forest estimates and CART forest estimates are biased even for large sample sizes in the low-dimensional setting. All three algorithms behave as expected when the effect size ($\mu$) increases: they all approach zero conditional entropy, as seen in Figure \ref{fig:convergence}B. CART forests and isotonic regression forest however, exhibit a bias for small effect size.
\begin{figure}
  \centering
  \includegraphics[width=0.75\linewidth]{fig2.pdf}
  \caption{Behavior of random forest estimates for conditional entropy. Top plots are for $d=1$; bottom plots are for $d=20$. The left plot shows estimates vs. increasing sample size ($\mu = 1$). Twenty trials are plotted with high transparency to show variance. Right plot shows estimates vs. increasing $\mu$ ($n = 3000$ for $d=1$ and $n = 6000$ for $d=20$). UF is consistent, and other approaches remain biased for very large sample sizes. UF is also closest to the truth for low effect sizes.
  }
  \label{fig:convergence}
\end{figure}
For the high-dimensional experiment,  $X_i$'s are multivariate Gaussians, where the mean of the first dimension is still $Y_i \times \mu$ but each additional dimension has mean $0$ and identity covariance: $X_i \sim \mathcal{N}(\underbrace{(Y_i \mu, 0, \dots, 0)\T}_{d}, \underbrace{I_d}_{\text{Identity matrix}})$,
where $\mathcal{N}(\theta, \Sigma)$ refers to the Gaussian distribution with mean $\theta \in \mathbb{R}^d$ and covariance matrix $\Sigma \in \mathbb{R}^{d \times d}$.
Because each added dimension is noise, the conditional entropy does not change. This allows us to compare behavior of our forest estimates to truth \citep{Ramdas15}.
Figure \ref{fig:convergence}C show that when $d=20$, the UF estimate still converges to truth as sample size increases. Interestingly, the bias of isotonic regression forests decreases in the high-dimensional setting, suggesting that in this setting the true posteriors might be (approximately) monotonic in the learned posteriors. When $d = 1$, the bias of IRF becomes worse as sample size increases.
Figure \ref{fig:convergence}D demonstrates that CART forests remain highly biased, whereas the other approaches lack bias in these high-dimensional settings.

\subsection{Mutual Information Experiments}
We compare UF to the KSG, mixed KSG, and isotonic regression estimators of mutual information~\citep{ksg, mixedksg, isotonic}. Consider three simulation settings, all based on mixtures of Gaussians with various parameters. We compute normalized mutual information, $I(X; Y)/\min\{H(X), H(Y)\}$. For each setting, we consider both dimensionality $d = 1$ and increasing dimension up to $d=20$. Only the first dimension depends on the class label, each additional dimension is an independent, standard Gaussian. Because only the first dimension contains any signal, mutual information does not change with increasing dimensionality \citep{Ramdas15}. In the case of two classes, $Y$ is drawn Bernoulli with probability $\pi$. For the three class case, the classes are drawn multinomial from the vector $(\pi, \frac{1-\pi}{2}, \frac{1-\pi}{2})$. We explore the robustness of the estimators both with $d = 2$ and changing class prior $\pi$, as well as increasing dimensionality when $\pi = \frac{1}{2}$.
\begin{description}
\item[Spherical Gaussians] A mixture of two Gaussians, the same distribution as in Figure \ref{fig:convergence}. The $\mu$ parameter controls the effect size while $y \in \{-1, +1\}$ controls the class label: $X \mid Y=y \sim \mathcal{N}(y(\mu , 0)\T, I)$.
\item[Elliptical Gaussians] To quantify performance of MI estimators when the Bayes optimal decision boundary is not axis-aligned, consider elliptical Gaussians: $X \mid Y=y \sim \mathcal{N}\left(y (\mu , 0)\T, \Sigma_y\right)$,
where $\Sigma_{-1} =
\begin{bmatrix}
3 & 0 \\
0 & 1
\end{bmatrix}$, and $\Sigma_{+1} = I$.
\item[Three Classes] To quantify performance for greater than two classes, let $y \in \{0,1,2\}$, and $X \mid Y = y \sim \mathcal{N}(\mb{\mu}_y, I)$
where $\mb{\mu}_0 = (0,\mu)\T, \enskip \mb{\mu}_1=  (\mu, 0)\T, I), \enskip \mb{\mu}_2 = (-\mu, 0)\T$.

\end{description}

Figure \ref{mi} shows the performance of each estimator. When $d=2$, UF, KSG, mixed KSG, and IRF all do reasonably well, though IRF is heavily biased for various level of class balance. As dimension increases, the KSG and Mixed KSG estimators suffer a significant performance degradation.
In the three class case, the KSG suffers a worse bias in high-dimensional settings. On the other hand, the UF estimator maintains performance as dimensionality gets high, but incurs more bias amid severe imbalance. For these Gaussian settings, IRF actually improves its performance from low-to-high dimensional data. In comparison to other methods, UF shows strong performance in high-dimensional and moderately imbalanced settings.

\begin{figure}
  \centering
  \includegraphics[width=0.75\linewidth]{fig3.pdf}
  \caption{Mutual information estimates in three different (``near'') Gaussian settings. Left: an example sample for each setting. \emph{Center}. Normalized mutual information for $d=2$ and $n = 6000$. \emph{Right}. Normalized mutual information estimates at $n = 6000$ for various dimensions. The mean of twenty trials is plotted. Both KSG and Mixed KSG break down in the high-dimensional setting, past $d = 8$. Similar to \ref{fig:convergence}, isotonic regression is biased in the low-dimensional setting, but improves for these distributions for high dimensions. For the two-class settings, UF successfully recovers the mutual information, while is suffers slight bias in the three-class setting. See main text for equations.
  }
  \label{mi}
\end{figure}

\section{Mutual Information in Drosophila Neural Network (Connectome)}

An immediate application of our random forest estimate of conditional entropy is measuring information contained in neuron types for the larval \textit{Drosophila} mushroom body (MB) connectome \citep{mb}. This dataset, obtained via serial section transmission electron microscopy, provides a real and important opportunity for investigating synapse-level structural connectome modeling \citep{Vogelstein2019-om}. This connectome consists of 213 different neurons ($n = 213$) in four distinct cell types: Kenyon Cells, Input Neurons, Output Neurons, and Projection Neurons. The connectome adjacency matrix is visualized in Figure \ref{fig:app} (left).
\begin{figure}
  \centering
  \raisebox{-0.425\totalheight}{\includegraphics[width=0.35\linewidth]{adj_matrix.pdf}}
  \begin{tabular}{|c | c c c|}
    \hline
    $X_{\text{in}}$      & $\hat{I}(Y, X_{\text{in}})$           & $\hat{I}(Y, X_{\text{out}} \mid X_{\text{in}})$     & $\hat{I}(Y,X)$            \\
    \hline
    claw              & {\bf 0.294}      & 0.622  & 0.917 \\
    dist              & {\bf 0.474}      & 0.442  & 0.917 \\
    age               & {\bf 0.595}      & 0.321  & 0.917 \\
    cluster           & {\bf 0.800}      & 0.116  & 0.917 \\ \hline
    cluster, claw      & 0.802   & 0.114  & 0.917 \\
    cluster, dist      & 0.845   & 0.071  & 0.917 \\
    cluster, age       & 0.913   & 0.003  & 0.917 \\
    cluster, claw, dist & 0.847  & 0.069  & 0.917 \\
    cluster, claw, age  & 0.917  & 0.000  & 0.917 \\
    cluster, dist, age  & 0.587  & 0.329  & 0.917 \\
    \hline
  \end{tabular}
  \caption{\emph{Left} \textit{Drosophila} larva right hemisphere connectome. Groups of neurons are labelled $K$ for Kenyon Cells, $I$ for Input Neurons, $O$ for Output Neurons, and $P$ for Projection Neurons (PN). Black cells represent the presence of an edge between the two corresponding nodes.
  \emph{Right} Adjacency spectral embedding applied to the MB connectome shows clear cluster groups for each neuron type. This suggests a strong dependency between neuron type and neural features.}
  \label{fig:app}
\end{figure}
Each neuron comes with a mixture of categorical and continuous features: ``claw'' refers to the integer number of dendritic claws for Kenyon cells, ``dist'' refers to real distance from the neuron to the neuropil, ``age" refers to neuron (normalized) age as a real number between -1 and 1, and ``cluster" refers to the community detected by the latent structure model as in \citet{lsm}.
We compute mutual information with $Y$ as the neuron type and $X$ as various subsets of the features. Because neuron type has been a subjective categorical assignment based on gross morphological features, we expect mutual information to be high for the entire feature vector. Running a permutation test with 1000 replicates, the test statistic $\hat{I}_n(Y, X) = 0.913$ was found to be statistically greater than zero with a $p$-value of $0.001$. Regarding the individual features, scientific prior knowledge posits a few relationships that are confirmed by the mutual information estimates. Letting $X_{\text{in}}$ be the subset of features under consideration, we compute $\hat{I}(Y, X_{\text{in}})$, the mutual information between $Y$ and the ``in" features, and $\hat{I}(Y, X_{\text{out}} \mid X_{\text{in}})$, the additional information given by the ``out" features (Figure \ref{fig:app} (right)). We confirm that the estimates $\hat{I}(Y, X_{\text{in}}) + \hat{I}(Y, X_{\text{out}} \mid X_{\text{in}}) = \hat{I}(X,Y) $
adhering to the chain rule of mutual information.
Because only Kenyan cells have dendritic claws, this feature is unable to discriminate between the other three classes, explaining why it has the lowest mutual information estimate among the features.
Age is typically computed using distance from the neuropil, which explains why both of these features yielded similar information regarding $Y$.
Figure 4 of \citet{lsm} presents compelling evidence of latent structure model clusters being closely related to cell type, which is corroborated by its highest mutual information estimate among  neuron features.
Moreover, adding one or two additional features increases our estimated MI with respect to neuron class.

\section{Conclusion}
We present Uncertainty Forest (UF), a nonparametric method of consistently estimating conditional entropy through randomized decision trees. Empirically, UF performs well in low- and high-dimensional settings. Furthermore, when extending our estimator to estimate mutual information, UF performs better than the mixed KSG and KSG estimators in a variety of settings. UF has strong theoretical justification in comparison to calibration methods such as isotonic regression forests. In machine learning and statistics, this tool could be adapted for feature selection, while in biology and neuroscience, scientists can find quantify uncertainty between various properties and labels.

The main limitation of this work is that it is only able to estimate uncertainty quantities for categorical $Y$; that said, the UF algorithm can be modified for continuous $Y$ as well.
Computing the posterior distribution $\hat P(Y|X = x)$ when $Y$ is continuous can be accomplished with a kernel density estimate instead of simply binning the probabilities. A theoretical and empirical analysis of this extension is of interest. When $Y$ is multivariate, a heuristic approach such as subsampling $Y$ dimensions or using multivariate random forests can be explored.

On the theoretical side, important next steps include rigorous proofs for convergence rates. Studying the behavior of UF estimates in more complicated nonlinear, high-dimensional settings should be explored as well. Practical applications such as dependence testing and $k$-sample testing for high-dimensional, nonlinear data will be natural applications for these information theoretic estimates.

\paragraph{Data and Code Availability Statement}
\label{sec:repo}

The implementation of UF, the simulated and real data experiments, and their visualization can be reproduced completely with the instructions in code available on \href{https://github.com/neurodata/uncertainty-forest}{GitHub}. The experiments were run in parallel on a 1TB RAM machine with 96-cores.

\nocite{*}

\section*{Acknowledgements}

\input{acknowledgements}

\vspace{5mm}

\section{Pseudocode}
\label{sec:code}
The \textproc{FitRandomForest}, \textproc{GetInBagSamples}, and \textproc{ApplyTree} operations, as well as the \textproc{NumClasses} and \textproc{NumLeaves} fields are all standard functions in the \texttt{scikit-learn} decision tree and bagging classifier modules \citep{scikit-learn}.
\begin{algorithm}
   \caption{Uncertainty Forest (Forest-level)}
   \label{alg:cerfestimate}
\begin{algorithmic}
\Require Training set $\mathcal{D}_n$ and evaluation set $\mathcal{D}^{\text{E}}$, $\theta= \{$number of trees $B$, minimum leaf size $k$, subsample size $s$ $\}$, finite correction constant $\kappa$.
\Ensure Conditional entropy estimate $\hat{H}(Y \mid X)$.
\Function{UncertaintyForest}{$\mathcal{D}_n$, $\mathcal{D}^{\text{E}}$, $\theta$, $\kappa$}
    \State $\mc{T} = \mathcal{D}_n - \mathcal{D}^{\text{E}}$
    \State model = \textproc{FitRandomForest}$(\mc{T}, \theta)$.
    \State posteriors = [].
    \For{tree $b$ \textbf{in} model}
        \State posterior = \textproc{EstimatePosterior}($\mc{T}$, model, $\kappa$, $b$)
        \State posteriors.append(posterior)
    \EndFor
    \State $\hat{H}(Y \mid X)$ = \textproc{EvaluatePosteriors}($\mathcal{D}^{\text{E}}$, model, posteriors).
    \State \Return $\hat{H}(Y \mid X)$.
\EndFunction
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
   \caption{Uncertainty Forest (Tree-level)}
   \label{alg:cerfestimate2}
\begin{algorithmic}
\Require Training set $\mathcal{D}_n$ and evaluation set $\mathcal{D}^{\text{E}}$, $\theta= \{$number of trees $B$, minimum leaf size $k$, subsample size $s$ $\}$, finite correction constant $\kappa$.
\Ensure Conditional entropy estimate $\hat{H}(Y \mid X)$.
\Function{UncertaintyForest}{$\mathcal{D}_n$, $\mathcal{D}^{\text{E}}$, $\theta$, $\kappa$}
    \State $\mc{T} = \mathcal{D}_n - \mathcal{D}^{\text{E}}$
    \State model = \textproc{FitRandomForest}$(\mc{T}, \theta)$.
    \State conditional\_entropies = []
    \For{tree $b$ \textbf{in} model}
        \State posterior = \textproc{EstimatePosterior}($\mc{T}$, model, $\kappa$, $b$)
        \State conditional\_entropies.append(\textproc{EvaluatePosteriors}($\mathcal{D}^{\text{E}}$, [model[$b$]], [posterior])).
    \EndFor
    \State $\hat{H}(Y \mid X)$ = conditional\_entropies.\textproc{Mean}().
    \State \Return $\hat{H}(Y \mid X)$.
\EndFunction
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
   \caption{Posterior Estimation}
   \label{alg:est_posterior}
\begin{algorithmic}
\Require training set $\mc{T}$ (full dataset without evaluation), fitted random forest model, finite correction constant $\kappa$, tree index $b$
\Ensure Posterior probability estimate tree $b$.
\Function{EstimatePosterior}{$\mc{T}$, model, $\kappa$, $b$}
    \State $K$ = model.\textproc{NumClasses}.
    \State $\mathcal{D}_\text{P}$ = \textproc{GetInBagSamples}(model, $b$)
    \State $\mathcal{D}_\text{V} = \mc{T} - \mathcal{D}_\text{P}$.
    \State $L$ = model[$b$].\textproc{NumLeaves}.
    \State vote\_counts = $[0]^{L \times K}$.
    \For{observation $(x, y)$ \textbf{in} $\mathcal{D}_\text{V}$}
        \State $l$ = model[$b$].\textproc{ApplyTree}($x$)
        \State vote\_counts[$l$, $y$] = vote\_counts[$l$, $y$] + 1.
    \EndFor
    \State leaf\_sizes = \textproc{RowSum}(vote\_counts).
    \State posterior = $[0]^{L \times K}$.
    \For{leaf index $l$ \textbf{in} $[L]$}
        \For{class $y$ \textbf{in} $[K]$}
            \State posterior[$l$, $y$] = vote\_counts[$l$, $y$] / leaf\_sizes[$l$].
        \EndFor
    \EndFor
    \State posterior = \textproc{FiniteSampleCorrect}(posterior, $\kappa$, leaf\_sizes)
    \State \Return posterior
\EndFunction
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
   \caption{Posterior Evaluation}
   \label{alg:eval_posterior}
\begin{algorithmic}
\Require evaluation set $\mathcal{D}^{\text{E}}$, fitted random forest model, posteriors for each tree
\Ensure Conditional entropy estimate $\hat{H}(Y \mid X)$.
\Function{EvaluatePosteriors}{$\mathcal{D}^{\text{E}}$, model, posteriors}
    \State $B$ = model.\textproc{NumTrees}.
    \For{observation $x_i$ \textbf{in} $\mathcal{D}^{\text{E}}$}
        \For{tree $b$ \textbf{in} model}
            \State $l$ = model[$b$].\textproc{ApplyTree}($x$).
            \For{class $y$ \textbf{in} $[K]$}
                \State posterior = posteriors[$b$].
                \State $p_b(y \mid x_i)$ = posterior[$l$, $y$].
            \EndFor
        \EndFor
        \State $p_i(y \mid x)  = \frac{1}{B} \sum_{b=1}^B p_b(y \mid x_i)$.
        \State $\hat{H}_i(Y \mid X) = -\sum_{y \in [K]}p_i(y \mid x) \log p_i(y \mid x)$.
    \EndFor
    \State $\hat{H}(Y \mid X) = \frac{1}{|\mathcal{D}^{\text{E}}|}\sum_{i \in \mathcal{D}^{\text{E}}} \hat{H}_i(Y \mid X)$.
    \State \Return $\hat{H}(Y \mid X)$.
\EndFunction
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
   \caption{Finite Sample Correction}
   \label{alg:finite}
\begin{algorithmic}
\Require posterior, finite correction constant $\kappa$, leaf\_sizes
\Ensure Corrected posterior.
\Function{FiniteSampleCorrect}{posterior, $\kappa$, leaf\_sizes}
    \State $(L, K)$ = posterior.\textproc{Shape}.
    \For{node index $l$ \textbf{in} $[L]$}
        \For{class $y$ \textbf{in} $[K]$}
            \If{posterior[$l$, $y$] == 0}
                \State posterior[$l$, $y$] = $\frac{1}{\kappa \cdot \text{leaf\_sizes}[l]}$
            \EndIf
        \EndFor
    \EndFor
    \State normalizing\_factor = \textproc{RowSum}(posterior).
    \For{node index $l$ \textbf{in} $[L]$}
        \For{class $y$ \textbf{in} $[K]$}
            \State posterior[$l$, $y$] = posterior[$l$, $y$] / normalizing\_factor[$l$]
        \EndFor
    \EndFor
    \State \Return posterior.
\EndFunction
\end{algorithmic}
\end{algorithm}

Note that $ \mathcal{D}^{\text{E}}$ can contain unlabelled data, as only the $x_i$ values are used.
\section{Proofs}
\label{sec:proof}
\setcounter{theorem}{0}
\setcounter{assumption}{0}

In this section, we present consistency results regarding estimation of conditional entropy and mutual information via Uncertainty Forest. The argument follows as a nearly direct consequence of \citet{Athey2019-uw}, in which random forests that are grown according to some specifications, and solve locally weighted estimating equations are consistent and asymptotically Gaussian. We review the assumptions.
\begin{assumption}
    Suppose that $X$ is supported on $\mathbb{R}^d$ and has a density which non-zero almost everywhere. This is equivalent to have $X$ be uniformly distributed in $[0, 1]^d$ due to the monotone invariance of trees \citep{denil14}.
\end{assumption}
\begin{assumption}
    Suppose that for each $y \in [K]$ the conditional probability $p(y \mid x)$ is Lipshitz continuous on $\mathbb{R}^d$.
\end{assumption}
\begin{assumption}
    Each tree $b$ is constructed such that for all $x \in \mathcal{X}, \epsilon > 0$, $P[N_b(x) < \epsilon] \rightarrow 0$ as $n \rightarrow \infty$.

\end{assumption}

Next, we summarize the notation and main result of \citet{Athey2019-uw}. Let $\theta(x)$ be a parameter that is implicitly defined as the solution to some equation $M_\theta(x) = 0$. For example, in estimating the conditional mean $\theta(x) = \ensuremath{\mathbb{E}}[Y \mid X = x]$, we can use
\begin{align*}
    M_\theta(x) = \ensuremath{\mathbb{E}}[Y \mid X = x] - \theta(x) = 0.
\end{align*}
To simplify notation, we suppress the dependency of $x$ in $\theta$ and simply write $\theta=\theta(x)$ wherever this dependency can be deduced from the context. To develop a sample estimate for $\theta$, we start with a sample score function $\psi_\theta$ such that
\begin{align*}
    \ensuremath{\mathbb{E}}[\psi_\theta(Y) \mid X = x] = M_\theta(x).
\end{align*}
Now, let us be given a dataset $\{(X_1, Y_1), ..., (X_n, Y_n)\}$. The sample estimate $\hat{\theta}$ solves a locally weighted estimating equation
\begin{align*}
    \sum_{i=1}^n \alpha_i(x) \psi_\theta(Y_i) = 0.
\end{align*}
In the case of conditional mean estimation, we will have $\psi_\theta(Y) = Y - \theta$. The $\alpha_i(x)$'s  weigh highly observations with $X_i$ close to $x$, as to approximate the conditional expectation of $\psi_\theta(Y)$, or $M_\theta(x)$. In a random forest, these weights amount to being the empirical probability that the test point $x$ shares a leaf with each training point $X_i$. Under mild assumptions, \citet{Athey2019-uw} have shown such a method to be consistent for the estimation of $\theta(x)$. For such a result to be used for our purpose of estimating conditional entropy, we define $M_\theta(x)$ as
\begin{align*}
    M_\theta(x) &= p(y \mid x) - \theta
\end{align*}
Using this definition, we can derive consistency of $\hat{H}(Y \mid X = x)$. Given this function of $x$, we have the conditional entropy written as
\begin{align*}
    H(Y \mid X) &= \ensuremath{\mathbb{E}}_{X'}[H(Y \mid X = X')]
\end{align*}

We first confirm that the finite-sample corrected posterior $\hat{p}_b(y \mid x)$ approaches the uncorrected $\tilde{p}_b(y \mid x)$ for large $n$.
\begin{lemma}
\label{thm:finite}
As $n \rightarrow \infty$
\begin{align*}
    |\hat{p}(y \mid x) - \tilde{p}(y \mid x)| & \overset{P}{\rightarrow} 0
\end{align*}
\end{lemma}
\begin{proof}
    We index the estimates with $n$ to make explicit the dependence on sample size. For tree $b$, in the cases where $0 < \tilde{p}_{b,n}(y \mid x) < 1$ for all $y$, we have |$\hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)| = 0$. Otherwise, for $\mathcal{N} = \{y : \tilde{p}_{b,n}(y \mid x) = 0\}$ and $y  \in \mathcal{N}$, then
    \begin{align*}
        |\hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)| &= \frac{1}{\kappa N_{b,n}(x)}.
    \end{align*}
    By assumption \ref{asm:bigcells}, we have that $N_{b,n}(x) \overset{n}{\rightarrow} \infty$ for all $x$. Thus,
    \begin{align*}
        \frac{1}{\kappa N_{b,n}(x)} \overset{P}{\rightarrow} 0
    \end{align*}
    Similarly, the constant $c = \frac{|\mathcal{N}|}{\kappa N_{b,n}(x)} \leq \frac{K}{\kappa N_{b,n}(x)} \rightarrow 0$ in probability. Thus, for $y \not\in$ $\mathcal{N}$,
    \begin{align*}
        |\hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)|
        &= |(1-c)\tilde{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)|\\
        &= |c \cdot \tilde{p}_{b,n}(y \mid x)| \leq c \rightarrow 0
    \end{align*}
    as $n \rightarrow \infty$. The estimate
    $\hat{p}_n(y \mid x) = \frac{1}{B} \sum_{b=1}^B \hat{p}_{b,n}(y \mid x)$ is a finite average of the $\hat{p}_{b,n}(y \mid x)$. Given $\epsilon > 0$,
    \begin{align*}
        P[|\hat{p}_n(y \mid x) - \tilde{p}_n(y \mid x)| > \epsilon]
        &= P[|\tfrac{1}{B}\sum_{b=1}^B \hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)| > \epsilon]\\
        &\leq P[\sum_{b=1}^B |\hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)| > B\epsilon]\\
        &\leq P\left[\bigcup_{b=1}^B |\hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)| > \epsilon\right]\\
        &\leq \sum_{b=1}^B P[|\hat{p}_{b,n}(y \mid x) - \tilde{p}_{b,n}(y \mid x)| > \epsilon]\\
        &= B \cdot P[|\hat{p}_{1,n}(y \mid x) - \tilde{p}_{1,n}(y \mid x)| > \epsilon]\\
        &= B \cdot \big( P[|\hat{p}_{1,n}(y \mid x) - \tilde{p}_{1,n}(y \mid x)| > \epsilon \cap \tilde{p}_{1,n}(y \mid x) = 0]\\
        &\quad + P[|\hat{p}_{1,n}(y \mid x) - \tilde{p}_{1,n}(y \mid x)| > \epsilon \cap \tilde{p}_{1,n}(y \mid x) \neq 0] \big)\\
        &\overset{n}{\rightarrow} 0
    \end{align*}
\end{proof}

Second, we require that honestly estimated posteriors $\tilde{p}_n(y \mid x)$ (without the finite sample correction) converge to the true posteriors. This can be done by application of Theorem 3 from \citet{Athey2019-uw}.

\begin{lemma}
    \label{thm:prob}
    For all $y \in [K]$ and $x \in \mathbb{R}^d$, $\tilde{p}_n(y \mid x) \overset{P}{\rightarrow} p(y \mid x)$ as $n \rightarrow \infty$.
\end{lemma}
\begin{proof}
    For a fixed $(x, y)$, we can define $\theta$ as the solution to
    \begin{align*}
        M_\theta(x) = p(y \mid x) - \theta = 0.
    \end{align*}
    The sample equivalent is thus
    \begin{align*}
        \psi_\theta(Y) = \ensuremath{\mathbbm{1}}[Y = y] - \theta,
    \end{align*}
    and we have that
    \begin{align*}
        \ensuremath{\mathbb{E}}[\psi_\theta(Y) \mid X = x] = M_\theta(x).
    \end{align*}
    Because our forest is learned according to Specification 1 in \citet{Athey2019-uw}, we must confirm Assumptions 1 - 6 from Section 3 of that paper to apply their result. Only Assumption 1 is a proper assumption on the distribution, whereas the remaining are confirmed true based on our choice of $M_\theta$ and $\psi_\theta$.
    \begin{enumerate}
        \item For fixed $\theta$, $M_\theta(x) = p(y \mid x) - \theta$ is Lipschitz in $x$. We took this as a standard assumption on the joint distribution of $(X,Y)$.
        \item For fixed $x$, $M_\theta(x)$ is twice continuously differentiable in $\theta$, $\frac{\partial^2 M_\theta}{\partial \theta^2} = 0$ and $\frac{\partial}{\partial \theta}M_\theta = -1$, which is invertible.
        \item The function $\psi_\theta$ satisfies
        \begin{align*}
            \sup_{x \in \mathcal{X}}(\text{Var}[\psi_\theta(Y) - \psi_\theta(Y)]) &= \sup_{x \in \mathcal{X}}(\text{Var}[\ensuremath{\mathbbm{1}}[Y = y] - \theta - (\ensuremath{\mathbbm{1}}[Y = y] - \theta')])
            &= \sup_{x \in \mathcal{X}}(\text{Var}[\theta' - \theta'])
            &= 0,
        \end{align*}
        and hence $\sup_{x \in \mathcal{X}}(\text{Var}[\psi_\theta(Y) - \psi_{\theta'}(Y)]) \leq L||\theta - \theta'||$ for all $\theta, \theta'$ and some $L \geq 0$.
        \item The function $\psi_\theta$ is itself Lipschitz in $\theta$.
        \item The solution of
        \begin{align*}
            \sum_{i=1} \alpha_i(x) \psi_{\theta}(Y_i) = 0
        \end{align*}
        in $\theta$ exists, and is equal to
        \begin{align*}
            \hat{\theta}(x) = \sum_{i=1}^n \alpha_i(x) \ensuremath{\mathbbm{1}}[Y_i = y]
        \end{align*}
        \item The function $\psi_\theta$ is the negative subgradient of a convex function, and $M_\theta$ is the negative subgradient of a strongly convex function (with respect to $\theta$). Choose
        \begin{align*}
            \bf{\Psi}(\theta) &= \frac{1}{2}(\ensuremath{\mathbbm{1}}[Y = y] - \theta)^2,\\
            \bf{M}(\theta) &= \frac{1}{2}(p(y \mid x) - \theta)^2
        \end{align*}
        as these functions.
    \end{enumerate}
    Granted the above assumptions, by Theorem 3 of \citet{Athey2019-uw} we have for every $x, y$,
    \begin{align*}
        \hat{p}_n(y \mid x) \overset{P}{\rightarrow} p(y \mid x).
    \end{align*}
\end{proof}
 By Lemma \ref{thm:prob} and continuity, we have the consistency of the honest forest estimate of conditional entropy.
 Let $\tilde{H}(Y \mid X = x) = -\sum_{y \in [K]} \tilde{p}_n(y \mid x) \log \tilde{p}_n(y \mid x)$.
\begin{corollary} For each $x\in \mathbb{R}^d$,
\label{thm:no_finite}
\begin{align*}
    \tilde{H}(Y \mid X = x) \overset{P}{\rightarrow} H(Y \mid X = x).
\end{align*}
\end{corollary}
\begin{proof}
    The function
    \begin{align*}
        h(p) = \begin{cases}
        0 &\text{ if } p = 0\\
        -p \log p &\text{ otherwise}
        \end{cases}
    \end{align*}
    is continuous on $[0,1]$. Similarly, the finite sum $\sum_{k=1}^K h(p_k)$ is continuous on $\{(p_1, ..., p_K) : 0 \leq p_k \leq 1, \sum_{k=1}^K p_k = 1\}$. Thus, by Lemma \ref{thm:prob} and the continuous mapping theorem, we have the desired result.
\end{proof}

\begin{lemma}
    \label{thm:function}
    The Uncertainty Forest function estimate $\hat{H}(Y \mid X = x)$ converges in probability to $H(Y \mid X = x)$ for each $x \in \mathbb{R}^d$.
\end{lemma}
\begin{proof}
     By continuity of $\sum_{k=1}^K h(p_k)$ on $\{(p_1, ..., p_K) : 0 \leq p_k \leq 1, \sum_{k=1}^K p_k = 1\}$, and Lemma \ref{thm:finite} we have that for any $x$,
    \begin{align}
        \left|\hat{H}(Y \mid X = x) - \tilde{H}(Y \mid X = x)\right| \overset{P}{\rightarrow} 0
        \label{eqn:entropy_correction}
    \end{align}
    Then, given $\epsilon > 0$,
    \begin{align*}
        P[|\hat{H}(Y \mid X = x) - H(Y \mid X = x)| > \epsilon] &\leq P[|\hat{H}(Y \mid X = x) - \tilde{H}(Y \mid X = x)|\\
        &\quad + |\tilde{H}(Y \mid X = x) - H(Y \mid X = x)| > \epsilon]\\
        &\leq P[|\hat{H}(Y \mid X = x) - \tilde{H}(Y \mid X = x)| > \frac{\epsilon}{2}]\\
        &\quad + P[|\tilde{H}(Y \mid X = x) - H(Y \mid X = x)| > \frac{\epsilon}{2}]\\
        &\overset{n}{\rightarrow} 0,
    \end{align*}
    by Lemma \ref{thm:no_finite} and \ref{eqn:entropy_correction}.
\end{proof}
\setcounter{theorem}{0}

For simplicity of notation, let $n$ refer only to the total sample size of the partition $\mathcal{D}^{\text{P}}$ and vote $\mathcal{D}^{\text{V}}$ sets. Let $\nu$ be the size of the evaluation set $\mathcal{D}^{\text{E}}$. If a subset of the labelled data is being used, then this may be some fraction $0 < \gamma < 1$ of the size of the full training data. In this case, we consider any growing evaluation set size $\nu$.

\begin{theorem}[Consistency of the conditional entropy estimate] Suppose the conditions of the preceding lemmas. Then the conditional entropy estimate is consistent as $n, \nu \rightarrow \infty$, that is,
    \begin{equation*}
        \hat{H}_{n, \nu}(Y \mid X) \overset{P}{\rightarrow} H(Y \mid X).
    \end{equation*}

\end{theorem}
\begin{proof}[Proof of Theorem \ref{thm:cond_ent}]
    By Lemma \ref{thm:function} we have the pointwise consistency of $\hat{H}_n(Y \mid X = x)$ (now indexed by $n$ explicitly) for $H(Y \mid X = x)$. We compute the limits with respect to $n$ and $\nu$ in either order to achieve the result. Let $\hat{H}_{n, \nu}(Y \mid X) = \frac{1}{\nu} \sum_{i \in \mathcal{D}^{\text{E}}} \hat{H}_n(Y \mid X = X_i)$.
    \begin{align*}
        \lim_{\nu \rightarrow \infty} \lim_{n \rightarrow \infty} \hat{H}_{n, \nu}(Y \mid X) &= \lim_{\nu \rightarrow \infty} \frac{1}{\nu} \sum_{i \in \mathcal{D}^{\text{E}}} \lim_{n \rightarrow \infty} \hat{H}_n(Y \mid X = X_i)\\
        &= \lim_{\nu \rightarrow \infty} \frac{1}{\nu} \sum_{i \in \mathcal{D}^{\text{E}}} H(Y \mid X = X_i)\\
        &= H(Y \mid X)
    \end{align*}
    by the Weak Law of Large Numbers, where the limits are taken in probability. On the other hand,
    \begin{align*}
        \lim_{n \rightarrow \infty} \lim_{\nu \rightarrow \infty} \hat{H}_{n, \nu}(Y \mid X) &= \lim_{n \rightarrow \infty} \lim_{\nu \rightarrow \infty} \frac{1}{\nu} \sum_{i \in \mathcal{D}^{\text{E}}} \hat{H}_n(Y \mid X = X_i)\\
        &= \lim_{n \rightarrow \infty} \ensuremath{\mathbb{E}}_{X'}[\hat{H}_n(Y \mid X = X')]\\
        &= \lim_{n \rightarrow \infty} \int_{x \in \mathbb{R}^d} \hat{H}_n(Y \mid X = x) \ dF_X,
    \end{align*}
    also by the Weak Law. Because of the finiteness of $[K]$, the entropy-like term $|\hat{H}_n(Y \mid X = x)|$ is bounded by $\log K$ for all $n$ and $x$. Therefore, by the Dominated Convergence Theorem,
    \begin{align*}
        \lim_{n \rightarrow \infty} \int_{x \in \mathbb{R}^d} \hat{H}_n(Y \mid X = x) \ dF_X &= \int_{x \in \mathbb{R}^d} \lim_{n \rightarrow \infty}  \hat{H}_n(Y \mid X = x) \ dF_X\\
        &= \int_{x \in \mathbb{R}^d} H(Y \mid X = x) \ dF_X\\
        &= \ensuremath{\mathbb{E}}_{X'}[H(Y \mid X = X')]\\
        &= H(Y \mid X)
    \end{align*}
    Thus, $\hat{H}(Y \mid X)$ is consistent for $H(Y \mid X)$.
\end{proof}
Letting the $n$ and $\nu$ be a fixed fraction of the training set size is a special case of both growing to infinity.

The second result concerns the mutual information estimate. The entropy $H(Y)$ is estimated in the natural way, as in
\begin{align*}
    \hat{H}(Y) &= -\sum_{y \in [K]} \hat{p}(y) \log \hat{p}(y)
\end{align*}
where $\hat{p}(y) = \frac{1}{n}\sum_{i=1}^n \ensuremath{\mathbbm{1}}\{Y_i = y\}$ (and set to zero when appropriate). Consequently, the mutual information is estimated as
\begin{align*}
    \hat{I}_{n, \nu}(X,Y) &= \hat{H}(Y) - \hat{H}(Y \mid X).
\end{align*}
Consider the same assumptions as Theorem \ref{thm:cond_ent}.
\begin{theorem}[Consistency of the mutual information estimate]
    $\hat{I}_{n, \nu}(X,Y)\rightarrow_p I(X; Y)$ as $n, \nu \rightarrow \infty$.
\end{theorem}
\begin{proof}
    For $i.i.d.$ observations of $Y_i$, it is clear that $\hat{H}(Y)$ is a consistent estimate for $H(Y)$. By Theorem \ref{thm:cond_ent} we have the consistency of $\hat{H}(Y \mid X)$ for $H(Y \mid X)$. Thus, the consistency of $\hat{I}(X,Y)$ follows immediately.
\end{proof}

redpajama_set_name: RedPajamaArXiv


text:
Home New Music STEPHANIE HATZINIKOLIS Releases New Single "Weed and Whisky"
STEPHANIE HATZINIKOLIS Releases New Single "Weed and Whisky"
Indie singer-songwriter Stephanie Hatzinikolis is thrilled to release her new single "Weed and Whisky" that is out now.
Stephanie says, "The dark and moody groove full of attitude "Weed and Whisky," not only represents the struggles of alcohol abuse and the choice to choose the stark reality of sobriety, it also marks the one year goal of sobriety for artist and singer- songwriter, Stephanie Hatzinikolis. Contemplating the need to make big changes , the track is a coming – of – age story in a sense, telling the struggles of isolation and depression while also acknowledging the common trend of self – medication in modern life. Though the mood may feel serious, the underlying message is still one of hope as she invites the listener to take stock, accept responsibility, and to turn towards the possibility of building a better life."
Stephanie Hatzinikolis is an artist and songwriter with a focus on creating personal, yet catchy and mesmerizing indie pop rock. Hatzinikolis' sound is heavily influenced by artists such as Elliott Smith, Sufjan Stevens, and Broken Bells, to name a few. Being a survivor of a traumatic upbringing, Stephanie has found it very easy to get swept up in alcohol abuse as a distraction.
In September 2021, Hatzinikolis made the personal choice to attempt a full year completely sober to improve her mental and physical health. She has shared her sobriety story and all the obstacles she has faced along the way on social media, where she actively tries to inspire and support fans in their own struggles. Marking her sobriety goal of one year, Hatzinikolis has released the song, "Weed and Whisky," a dark and moody groove full of attitude.
As with all her music, Stephanie aims to help others to feel understood, loved, and more optimistic towards themselves and their lives.
"Weed and Whisky" is out everywhere at https://linktr.ee/Hatzinikolis
https://www.instagram.com/hatzinikolis/?hl=en
https://www.tiktok.com/@hatzinicookies
Facci PR
Stephanie Hatzinikolis
Previous articleGOOD KID Releases New Single "No Time To Explain"
Next articlePillbox Patti Releases Latest Track, "Eat Pray Drugs"

redpajama_set_name: RedPajamaCommonCrawl


This Football Bear is the perfect furry friend for any football fan. It features soft brown fur, strong muscles, a leather-like football print on the paw pads and ears, and "eye black" streaks under its eyes. With a Miami Hurricanes Hoody.


text:
A (7563) 1988 BC a Naprendszer kisbolygóövében található aszteroida. Takuo Kojima fedezte fel 1988. január 16-án.

Kapcsolódó szócikkek
 Kisbolygók listája (7501–8000)

Jegyzetek

A Naprendszer kisbolygói

redpajama_set_name: RedPajamaWikipedia


Here's your video introduction to Section 7.3 in the ARRL General Class License Manual for ham radio. The website mentioned in the video that provides NVIS maximum usable frequencies is www.spacew.com/www/fof2.html. You may return to the list of General Class training videos by clicking here.
This entry was posted in General Class License Training, Ham Radio blog entries, Video and tagged NVIS, Scatter Modes. Bookmark the permalink.


Additional flowers may be purchased. A rehearsal is suggested if you have 3 or more bridesmaids/groomsmen. You can add more chairs, grooms cake, bigger wedding cake, waters, beach decorations etc... I am happy to customize as needed.
Call for additional details (956) 433-9448.
In order to book I require a $250.00 non-refundable deposit that does go towards your total. Six weeks before the wedding 1/2 is due. This payment is non-refundable. The final payment is due the day before the wedding. I do not take a personal check for the final payment. I will cover the initial pay pal fees on the $250.00 deposit. If you choose to make further payments through pay pal you will need to cover the fees. Call today to book your wedding on the shores of Padre (956)433-9448.


On weekdays, I usually keep my makeup pretty simple. Foundation, a bit of bronzer and mascara, but when I have a few extra minutes, I like to wear some eyeshadow. I love a good eyeshadow palette. Even more, I love an eyeshadow palette with neutral colours which is great for everyday use. So when I received the HD Brows Eyeshadow Palette in Bombshell*, I was really excited to give it a go. I had already heard lots of good reviews about HD Brows and their brow products, so I was expecting the eyeshadow palette to be just as good.
The HD Brows Eyeshadow Palettes are available in Bombshell which includes beige and ash shades, Foxy which is more honey and bronze, and finally Vamp which contains colours like berry and aubergine. The palette I received is in the shade Bombshell, which is designed for fairer skintones and lighter hair. Now although I don't fit ideally into the target audience, the colours are still beautiful on my skin. The palette contains 5 different shades ranging from light to dark letting you do a complete makeup look with just this palette. The colours are: Naked, Ash, Burnished, Graphite and Midnight.
As you can see from my swatches, the powders are highly pigmented and have a satin or matte finish. I use the colour 'Naked' which is a glittery beige, on the inner corners of my eyes, and apply a mixture of 'Burnished' and 'Granite' to my crease. The powders blend easily and once done, the final result is gorgeous. Without an eyeshadow primer, the eyeshadows last about 6 hours. But if you prep your eyelids with an eyeshadow base prior to applying the eyeshadow, it stays put for about 10 hours without creasing or coming off. I am very impressed it stayed so well, considering the current weather in Barcelona, which makes me believe it would last even longer in the winter months. I'd really recommend this as a travel eyeshadow kit because it's easy to carry around and the quality is amazing.
Dislikes - a bit pricey.
Have you tried the HD Brows Eyeshadow Palette? What is your fave palette?


text:
John Leslie Miller
M, #9052, b. 11 April 1939, d. February 1980
Father: Living
Mother: Helen Marie McGeorge (b. 23 May 1918)
Birth: John Leslie Miller was born on 11 April 1939 in Buffalo, Erie County, New York, United States.1,2
Death: He died in February 1980 at age 40.1
Person Source: John Leslie Miller had sources that helped establish his name.1
[S1160] "U.S., Social Security Applications and Claims Index, 1936-2007," database, Ancestry.com (http://www.ancestry.com : accessed ); Social Security Administration, Baltimore, Maryland, 23 July 2016; original application, John Leslie Miller (June 1955); no. 112-30-3190; 4 January 1988, name listed as John Miller
[S1112] Frank L Taylor, History of John Taylor of Hadley, p. [number cut off], under IX. Helen Marie McGeorge
Raymond A Garlapo
M, #9053, b. 24 April 1915, d. 20 November 2009
Family: Shirley Harriet Davis (b. 29 March 1918, d. December 1976)
Birth: Raymond A Garlapo was born on 24 April 1915 in North Tonawanda, Niagara County, New York, United States.1
Marriage: He and Shirley Harriet Davis were married on 28 June 1941.2
Death: Raymond A Garlapo died on 20 November 2009 at age 94 in Naples, Collier County, Florida, United States.1
Burial: He was buried on 25 November 2009 at Arcade Rural CemeteryG+, in Arcade, Wyoming County, New York, United States.1
Person Source: Raymond A Garlapo had sources that helped establish his name.2
[S29] Find a Grave, database and images (http://www.findagrave.com : accessed ), memorial page for , Find A Grave Memorial # , citing , 23 July 2016; Raymond A Garlapo; 6495131; Arcade Rural Cemetery; Arcade, Wyoming County, New York; Marsha Redden; Jim and Elizabeth, no. 47230507
[S1112] Frank L Taylor, History of John Taylor of Hadley, p. [number cut off], under IX. Shirley Harriet Davis
Dorothy Lauck
Family: Dana Elbert Freeman (b. 19 December 1917, d. 25 March 1988)
Marriage: Dana Elbert Freeman and Dorothy Lauck were married on 2 October 1943.1
Person Source: Dorothy Lauck had sources that helped establish her name.1
[S1112] Frank L Taylor, History of John Taylor of Hadley, p. [number cut off], under IX. Dana Albert Freeman
Robert Bookmiller
Family: Jeanne Mildred Howlett (b. 14 June 1914, d. 1985)
Marriage: Robert Bookmiller and Jeanne Mildred Howlett were married on 21 October 1934.1
Person Source: Robert Bookmiller had sources that helped establish his name.1
[S1112] Frank L Taylor, History of John Taylor of Hadley, p. [number cut off], under IX. Jeanne M Howlett
Ann Nell Reese
F, #9063, b. 20 April 1909, d. 20 December 2005
Birth: Ann Nell Reese was born on 20 April 1909 in Alabama, United States.1
Marriage: Lawrence Charles Older and Ann Nell Reese were married on 25 February 1933 in Cuba, Allegany County, New York, United States. Officiated by Robert M Purly, minister1
Death: She died on 20 December 2005 at age 96 in Clarksville, Allegany County, New York, United States.2
Burial: She was buried at Cuba CemeteryG+, in Cuba, Allegany County, New York, United States.2
Person Source: Ann Nell Reese had sources that helped establish her name.1
[S958] "New York, County Marriages, 1847-1849; 1907-1935," Database and images, The Church of Jesus Christ of Latter-day Saints, FamilySearch.org (http://www.familysearch.org : accessed ); County Clerk Offices, state-wide, New York, not all counties represented, 24 July 2016; marriage record, no. 8035, Lawrence Charles Older and Ann Nell Reese (1933); Allegany County, "Marriage records no. 7184-7952 1932-1935"
[S29] Find a Grave, database and images (http://www.findagrave.com : accessed ), memorial page for , Find A Grave Memorial # , citing , 24 July 2016; Ann Nell (Reese) Older; 154368375; Cuba Cemetery; Cuba, Allegany County, New York; Terry Winchell; Terry Winchell, no. 48397452
Ruth E Shannon
Birth: Ruth E Shannon was born about 1903 in Haskell Flats, Cattaraugus County, New York, United States.1
Marriage: Lynn R Carrier and Ruth E Shannon were married on 11 February 1922 in Olean, Cattaraugus County, New York, United States. Officiated by G R Williamson, minister1
Person Source: Ruth E Shannon had sources that helped establish her name.1
[S958] "New York, County Marriages, 1847-1849; 1907-1935," Database and images, The Church of Jesus Christ of Latter-day Saints, FamilySearch.org (http://www.familysearch.org : accessed ); County Clerk Offices, state-wide, New York, not all counties represented, 24 July 2016; marriages, vol. 2, p. 326 no. 4636 for Lynn R Carrier and Ruth E Shannon (1922); Allegany County, "Marriages v. 2, 1917-1932"
Clarence Everett Bailey
M, #9070, b. 30 April 1905, d. 7 August 1980
Family: Catherine M Carriers (b. 6 June 1908, d. October 1977)
Donna Joan Bailey (b. 31 August 1930, d. 17 February 2009)
Nora Dawne Bailey (b. 2 August 1938, d. 19 August 2014)
Birth: Clarence Everett Bailey was born on 30 April 1905 in Burns, Allegany County, New York, United States.1,2
Marriage: He and Catherine M Carriers were married on 3 August 1929 in Richburg, Allegany County, New York, United States. Officiated by W. N. Mason, minister1
Death: Clarence Everett Bailey died on 7 August 1980 at age 75 in Youngsville, Warren County, Pennsylvania, United States.2
Burial: He was buried at Chandlers Valley Cemetery+ in Chandlers Valley, Warren County, Pennsylvania, United States.2
Person Source: Clarence Everett Bailey had sources that helped establish his name.1
[S958] "New York, County Marriages, 1847-1849; 1907-1935," Database and images, The Church of Jesus Christ of Latter-day Saints, FamilySearch.org (http://www.familysearch.org : accessed ); County Clerk Offices, state-wide, New York, not all counties represented, 24 July 2016; marriages, vol. 2, p. 779 no. 6901 for Clarence Everett Bailey and Catherine M Carrier (1929); Allegany County, "Marriages v. 2, 1917-1932"
[S29] Find a Grave, database and images (http://www.findagrave.com : accessed ), memorial page for , Find A Grave Memorial # , citing , 6 June 2017; Clarence Everett Bailey; 8148628; Chandlers Valley Cemetery; Chandlers Valley, Warren County, Pennsylvania; Peg Stino; CABrown, no. 47000639
Agnes B Rooney
Birth: Agnes B Rooney was born about 1890 in Cuba, Allegany County, New York, United States.1
Marriage: Earl V Pettit and Agnes B Rooney were married on 28 June 1916 in Cuba, Allegany County, New York, United States. Officiated by Frank O Leonard, minister1
Person Source: Agnes B Rooney had sources that helped establish her name.1
[S958] "New York, County Marriages, 1847-1849; 1907-1935," Database and images, The Church of Jesus Christ of Latter-day Saints, FamilySearch.org (http://www.familysearch.org : accessed ); County Clerk Offices, state-wide, New York, not all counties represented, 24 July 2016; marriage records, vol. 1, p. 559, no. 2800 for Earl V Pettit and Agnes B Rooney (1916); Allegany County, "Marriages v. 1, 1908-1917"


Mercury switches were used in several Tektronix products for generating pulses with clean edges.
The switches are electromagnetic relays with mercury-wetted contacts inside glass envelopes that contain pressurized gas. The gas prevents arcing of the contacts. Mercury switches have limited life.
In pulse generators, the coil of the relay is typically driven by a multivibrator or other periodic current source. Turning off the coil drive waveform when the pulse generator is not in use is recommended.
This page was last edited on 26 March 2017, at 12:00.


There really has never been a better time than now to start an Internet business. The Internet explosion has allowed people who were sick of working the regular 9 to 5 schedule to start their own online businesses. Online businesses have a good potential to make money if they're marketed correctly because there are less overhead expenses.
The one thing you must have if you expect to receive payments directly from customers visiting your website is a merchant account. A merchant account is an account that online business owners use to send and receive payments. All you'll need to open one is some personal information and a separate bank account. You'll need a bank account so that once you've received payments, they can be transferred to your business account.
There are a number of benefits to opening up your own account. They're easy to apply for and set up but here are some of the other reasons why they're excellent for online retailers.
They're safe. Identity theft is of great concern to online shoppers but a merchant account keeps all personal information safe. Consumer information doesn't float out into cyberspace, rather it's simply collected in order to obtain a payment and is not shared whatsoever.
Customers Don't Have to Leave Your Site. Have you ever been to a website that makes you leave the site to pay for your items? That can be a bit annoying and can leave customers wondering if their information is safe. With a merchant account your customers won't have to worry about that because the checkout process can be done entirely on your website CBD merchant account processor.
Generate More Sales. If you accept credit card and debit card payments you are automatically going to increase the potential number of sales for your business. People shopping online love to pay electronically because it is quick and that allows them to receive their goods or services faster.
Receive Payments from Around the World. Merchant account providers can work with you to determine what's best for your business but there is the possibility of being able to accept payments from all around the world. This is another easy way to be able to generate more sales on your website by using a merchant account.
Receive Excellent Customer Support. Providers know how important customer service is which is why most of them provide 24/7 customer service. Any time an issue should arise with your site's checkout system or payment gateway, they will be on hand to help you right away. The quicker you can get everything working again the sooner you'll be able to make more sales.
Low Fees. Merchants are usually charged for each transaction that involves an electronic payment. This is because the merchant account provider is also charged for processing payments, however the fee that you will experience is a very, very small percentage of the transaction. Some providers may even provide a flat fee. It all depends on your company's sales volume and how many transactions you process each month.
As you can see, it's almost silly not to have some kind of merchant account if you're conducting business online. These accounts make it easy to receive payments and generate refunds to your customers as needed. You don't need to be a "techie" either because the merchant account provider will be there to help you every step of the way.


I agree that having a lot of books, all of whom sell from 1-10 copies a month, is probably the Way To Go for actually making money "passively." (I.e., minimizing hustle.) And writing what you Want To Write is likely to be important for avoiding burn-out, so from an outside perspective, this looks very logical to me!
I think I need to have baseline sales of more than 1-10 copies a month in order for this to get up to a living wage. :D But the baseline may inch up as my catalog grows.
So to make $31,200 (the equivalent of a full time job at $!5 an hour, since that seems the popular wage point these days) on $4.99 e-books, one would need to sell ~890 books per month. That'd require 178 titles if they each moved ~5 copies per month.
I'm not sure I'm getting to 178 titles in my remaining lifetime. I mean, maybe? But it's not happening any time soon. I gotta hope to get to the point where they each sell more like 1 per day, and then I only need 30-ish titles.
Alllleeeegedly -- extrapolating from Kris Rusch, who seems useful even if she's got a few pet peeves that I feel may be more nuanced -- if you get enough books, especially in enough different series, they start serving as entry points to readers, who then are more likely to go pick up the others. So if I'm parsing the theory correctly, the more books you write, the more likely you are to hit the 1-per-day mark.
Either way, full time or Helpful Supplement, the answer seems to be Rite Moar Bookz. Without burning out.
And really, with that A and B, can't go wrong, say I!


All our self– drive tours are available with a variety of accommodation and car rental options.
Your Honeymoon Vacation price will vary, depending on your needs.
We take all of the above into account when creating your Custom Quote & Itinerary.
We also run Special Offers on a regular basis, and may be able to apply generous discounts to your tour cost. Please contact us for more information and to receive a custom quote.


text:
Home Politics Adwoa Safo expected to return this week – Majority Leader
Adwoa Safo expected to return this week – Majority Leader
Sarah Adwoa Safo
The Majority Leader in Parliament, Osei Kyei-Mensah-Bonsu has revealed that Dome-Kwabenya Member of Parliament (MP) Sarah Adwoa Safo will be returning to Ghana this week.
The Leader of the Government Business told TV3's Benjamin Aidoo in interview on Monday February 28, after he was asked when the Gender Minister was returning to Ghana that " In the course of the week she may come down hopefully."
Asked again whether she will be punished for absenting herself from Parliament for more than fifteen days, the Suame lawmaker said "where we are, our orders have adequate provisions to deal with some of these matters but it depends on whatever excuse she has, if it is plausible.
"Which is why the orders provide that a person who absents himself or herself for more than fifteen days in the meeting is unable to provide plausible justification for the absence, then the committee of Privileges could propose sanctions, so it depends on the information she is coming with."
Some members of the New Patriotic Party (NPP) believe Adwoa Safo is sabotaging the government. For instance, the MP for New Juabeng South, Michael Okyere Baafi, said she is making the work of the government in Parliament difficult.
"All the problems government is facing are attributable to Adwoa Safo. Clearly, her intention and posture show that she wants to sabotage NPP," he said on Kumasi-based Hello FM.
The MP for Assin Central, Kennedy Agyapong also revealed that the Chief of Staff, Akosua Frema Osei-Opare gave him ¢120,000 to be deposited into the bank account of Adwoa Safo for her to attend to Parliamentary duties to work for the government.
READ ALSO: Don't be too excited about electricity tariff reduction promise - NDC Chair cautions
"The Chief of Staff called me and I went, I swear my mother's grave, Chief of Staff gave me ¢120,000 and deposited into Adwoa Safo's Fidelity Bank account.
"I took the money there into the Fidelity Account, ask her personal assistant if what I am saying is not the truth," a furious Kennedy Agyapong said.
He added "Now we are in opposition, the way Adwoa Safo is treating the party, it is clearly showing that we are in opposition and this shouldn't be tolerated at all," he told Asaase Radio on Tuesday February 22.
Mr Agyapong had raised issues against the conduct of Sarah Adwoa Safo.
He said that the Minister of Gender, Children and Social Protection has failed in life for requesting that she should be made a Deputy Majority Leader in Parliament.
He wondered why a cabinet minister will now want to occupy a position that is below the rank of a Minister.
Speaking on her rampant absence from Parliament, Mr Agyapong, who has two children with her, said: "She says she wants to be Deputy Majority, that woman has failed in life. A whole cabinet minister now demanding that she should be made a Deputy Majority Leader before she comes, she should go to hell.
"You don't come to Parliament and you are on TikTok dancing? Dome Kwabenya is not for Apostle Kwadwo Safo, get it straight. I am very furious because people are insulting me because I went there to campaign for her. Mike Oquaye the Speaker, the man that I respect very well, I campaigned against his son because Adwoa used my kids, calling me and begging me to help her. And now everybody is insulting me for doing that but I have not regretted it, she is very responsible when it comes to the kids," he said on GTV Monday, February 21.
READ ALSO: Martin Amidu promises to vindicate Ghanaians' trust at swearing-in
Asked how many days she has been absent from Parliament, he answered: "More than 15 days and the law says 15 days."
Asked again whether Ghanaians should expect her seat to be declared vacant, he said: "Why not? She should go and contest on TikTok."
Previous articleMan who faked robbery of his boss's GH¢69k with intention to travel with it grabbed
Next articleAU shocked at reported ill-treatment of Africans leaving Ukraine
NDC only talks because talk is cheap – Ahiagbah reacts to press conference on Covid funds
Akufo-Addo's govt is scared of probe into Covid funds utilization – Sammy Gyamfi
Nyaho Nyaho-Tamakloe 'storms' Addai-Nimoh's campaign launch
I was sidelined by Akufo-Addo even though I helped in NPP victory – Addai-Nimoh
A new face without blemish must lead NPP – Addai-Nimoh to contest
I received thunderous applause and massive show of support in Kumasi – Alan
NPP was going to lose 2024 miserably because of individual bondholders – Kennedy Agyapong
With me NPP can break the 8; without me it'll be tough – Kennedy Agyapong
Mahama is not an alternative, he hasn't proven to be a good leader – Kennedy Agyapong
NDC gives Akufo-Addo gov't 30-day ultimatum over Ayawaso West Wuogon by-election victims
Elect a presidential candidate from the heartbeat of the party – Catherine Afeku to NPP delegates


Lane Cove Senior Citizens Centre was not only completed on time and to the client's satisfaction and however formed part of the lives of all involved for the duration of the works. During the project, all contractors and employees involved in Lane Cove came together and worked with one another to meet the tight deadline and handover a project that would surpass the client's expectations. The project was a difficult and challenging build which was completed with teamwork and passion for construction.


text:
NASA Mission: Cloud Governance
Here's how the space agency's Jet Propulsion Laboratory addressed governance, risk, and compliance for its critical public cloud services.
Download the new issue of InformationWeek Government Tech Digest (registration required).
NASA is known for big science, both on Earth and in the farthest reaches of the solar system, and all of its research requires a lot of computer processing. Like many federal government agencies, NASA and its laboratories have tapped cloud computing and virtualization to cut costs and improve efficiency.
NASA's Jet Propulsion Laboratory relies on the cloud to help process and manage the massive amounts of data collected by its various science programs, including the Mars rovers. But in setting up and launching those cloud capabilities, JPL's IT staff had to meet a number of compliance, governance, and risk management requirements.
JPL, based at Caltech University in Pasadena, Calif., selected Amazon Web Services in 2009 because it wanted the flexibility to scale up its infrastructure to meet high-demand computing needs and then scale down or recommit resources to other projects on short notice, says Jonathan Chiang, JPL's chief IT engineer. Previously, NASA would buy data storage and processing capabilities up front for the requirements of an entire mission. So if a planned scientific satellite was forecast to require 100 TB of storage, "we would buy 100 TB today, and that storage would go minimally utilized for years," Chiang says.
But to use the AWS cloud to do this rapid research and analysis work, JPL had to develop a governance model that met NASA compliance requirements such as Federal Information Security Management Act rules and National Institute of Standards and Technology (NIST) guidelines. Auditability was another requirement; JPL is audited both by NASA's Office of the Inspector General and by Caltech.
Teething Problems
Setting up governance and compliance models proved useful because the initial cloud deployment ran into problems over account credentials. When JPL launched its cloud services, it issued a number of accounts to its in-house developers and project managers, but it quickly became evident that this model was flawed.
Specifically, NASA administrators realized that they had to apply security credentials and auditability to each account. What had initially been about 60 accounts bloomed into more than 250 sublevel accounts, none of which could be traced back to a JPL user, Chiang says.
These subaccounts were created when account holders gave access privileges to colleagues. But the use charges for the subaccounts all came out of JPL's funds, Chiang says. To stop this activity, JPL's IT staff had to contact all of the initial account users and gain full administrative access to their accounts, he says.
After the IT group realized that there were multiple root accounts, it had to design and implement a governance model for managing the accounts and apply that model to the existing network infrastructure and cloud deployment, says JPL cyber security engineer Matt Derenski.
The highly automated AWS cloud system helped JPL's IT staff sort out and manage the accounts issue and then other governance and compliance requirements. Chiang notes that JPL IT staffers sometimes have difficulty tracking usage and application data on the laboratory's internal network, but the Amazon cloud offers complete visibility into the number of active accounts and which servers they're running on.
JPL maintains an IT security database that tracks all running instances in its physical and virtual servers. Previously, staff had to manually enter CPU makes, models, and memory statistics into the system. This function is now handled by an AWS API, which provides updated instance information. The details are then fed into the IT security database, Chiang says. Using AWS cloud services was a double-edged sword, he says. It initially created a problem, but it also quickly provided JPL with a means to solve it -- an API for automated servicing.
Read the rest of the story in the new issue of
InformationWeek Government Tech Digest (free registration required).
GovernmentEnterprise ArchitectureCybersecurityLeadership
Evaluator's Guide for Managed Detection and Response (MDR) Services
Protecting Every Edge To Make Hackers' Jobs Harder, Not Yours
State of IT Team Resource Drain
Better Together: ITSM & PPM on One Platform
7 Key Takeaways to Passwordless Authentication


Your search for Shore in All Royalty Free Files returned 1937 results. Showing the first 1000.
Windermere lakeside with sounds of footsteps walking on gravel shore.
Waves of ocean sound effects, sea, shore, seagulls, atmosphere, ambiance, background, beach.


The Burmese government continues to neglect its people, particularly minor ethnic groups and Christians. In recent years there have been more than 3,000 Christian villages destroyed and burned (source: Impact Report).
Yet, despite the oppression in this country, there is hope and freedom being declared by brave soldiers of Jesus Christ.
Through Thirteen Three, youth from Australia alongside Soldiers of Jesus Christ founded by Issachar, sponsor students at a Bible College in Burma. As one of the lecturers, Issachar equips and trains the students to witness in these dangerous areas.
Recently, Issachar had the privilege of delivering Bibles that were sponsored by Thirteen Three's Badges for Bibles project. One family that received Bibles has agreed to meet every Saturday with the Soldiers of Jesus Christ team. This is particularly exciting as only the mother is a Christian at this stage.
Watch Issachar explain the recent outreach they had and one family receiving Bibles.
Please continue to pray for Burma and the work that these brave soldiers of Jesus Christ are doing.


text:
Q: Python async redis gives error AttributeError: __aexit__ I'm new to Python and trying to build a simple app to retrieve data from a redis database. But when i try to retrieve the data it gives me an "AttributeError: __aexit_" error.
Does anyone have an idea what i'm doing wrong? Thanks in advance!
My code:
from sanic import Sanic
from sanic.response import json
import redis

app = Sanic()

# request.args['token']

@app.route('/<id>')
async def test(request, id):
    async with redis.StrictRedis(host='0.0.0.0', port=6379, db=0) as r:
        data = await r.get("test")
        # print(data)

    return json({
        'data': data
    })

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9988)

Error traceback:
[17203] [ERROR] Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sanic/app.py", line 556, in handle_request
    response = await response
  File "main.py", line 11, in test
    async with redis.StrictRedis(host='0.0.0.0', port=6379, db=0) as r:
AttributeError: __aexit__

A: You should use an async implementation of redis library like aioredis.
https://github.com/aio-libs/aioredis

redpajama_set_name: RedPajamaStackExchange


This entry was posted on January 4, 2013, 8:58 pm and is filed under chick lit, deception, excerpt, fiction, friendship, love, lying, novel, parents, story, Uncategorized, women, writing. You can follow any responses to this entry through RSS 2.0. You can leave a response, or trackback from your own site.


J.B. Mauney recently won the 2013 World Professional Bull Riding Championship, making him the first American to do so since 2009. The North Carolina native spoke with TIME's Belinda Luscombe for this week's magazine about the intelligence of bulls, disregard for broken bones and his many tattoos.
Plus, hear why Mauney thinks New York is his favorite city for bull riding.


At Dumpster America, we strive to bring you the best service at the best price available. We have a wide selection of dumpster sizes in stock in Buffalo Gap, TX, with flexible & timley pickup and delivery. Give us a call today for all your dumpster rental and roll off needs.
Trash era in Guam will be inevitable since, it is made as part of regular hotel remodelling and upgrading of accommodations according to modifying trends. Resort industry inside Guam is experiencing serious undesirable effects thanks to down submit the tourist industry in the state. As a way to with remain the competition as well as attract this limited number of clients who visit a state, hotels out there are at regular renovation. It's good sign that they can try to maintain your hotels and also tidy by removing the crap on reasonable basis as well as upgrading the place with the advanced technology with respect to upcoming trends. These methods from the hotel industry associated with Guam will produce huge a great deal of trash inside the state this also trash provides to be disposed effectively.
Be it renovating your home or developing a new one, it's surely going to design your life simpler if you hire a company to help you with this garbage. How can a real service become a time and effort short-cut? Check out the following. You can do your own backyard washing, work place garbage disposal and much more on this service. Using disposal container and dumpster with rent now you can design your job associated with keeping the house or job clean much simpler.
By exploring the above informative tips, you ought to easily find an incredible company to rent overnight dumpsters for your building business.
There are also the companies that are so devoted to having a low base value that they have to put fees intended for things like shipping, pick-up, weekend premiums and so on.
How frequently have you purchased something as well as put something to the side because you might use it one day? This stuff can be smaller, like coupons, or large like storage area boxes and even machinery as well as furniture. Stop keeping these matters that you don't require! You are merely collecting trash! For high-priced items, sell them on the web or from the classified area of your local papers. When advertising things, restrict how long you are going to wait just before dropping your own price and just how long you are going to keep the merchandise. Less expensive goods can either go to charity as well as go straight to the trash, depending on whether another individual could use that.
You can have your own den renovated to fit regarding how you use it. If you want it bigger, knock out a wall as mentioned above. If you want it much more professional looking you'll probably decide to to have wood made bookshelves set up. You may need a more substantial area so your clients can certainly sit much more comfortably. In case you have more than one shopper from the similar business come at a time, you do not need one person or maybe more sitting away from your family room in yet another room.
The actual 20 cubic property dumpster is used intended for major building you may be accomplishing on the tastes your house. The actual 20 cubic property dumpster is identical to 10 pickup truck loads.
If you are impatiently planning on upgrading your kitchen in comparison with you have been hectic picking out completely new kitchen cabinets, completely new flooring, quite possibly new devices and more. With your large your kitchen remodeling task you will definitely require to rent a dumpster to eradicate all of the trash.
Prefer a dumpster permanently, may buy is from this company. A similar option for this is to rental the dumpster for just a certain amount associated with years, that is close to merely buying the dumpster. All needs might be accommodated intended for.
With springtime arriving, a number of minds convert to cleaning out your homes as well as selling what exactly we will no longer need in our lives. A new yard sale is an excellent thing on its own, but when you have the whole neighborhood in on it, your forces can merge to make a really fun event for your spouse and children, your neighbors, as well as anyone who prevents by to buy goods. And here is how you can get your location together for just a community garage sale.
To choose the right sort of container to suit your needs, you will need to determine the size of the project. Most people tend to overestimate their needs as well as, in this particular scenario, it would be a sensible thing to perform. You do not want to have got to pay nowadays than required, but the cost you would find for overfilling your container might be much greater than it would expense to rent a larger jar.
Have you looked at your car port or basement lately? Nearly all of us use these areas as indefinite home safe-keeping for points we think organic beef eventually utilize. But there will always be more points we could perhaps need in the future, and these spaces fill up rapidly, often to the point where we couldn't find the things we need amongst the clutter.
No actual home owner or even construction challenge will be alert to the right industry that can delete the trash within an effective manner or have plenty of time to make the junk reach the industrialist. Discarding the trash to the best location is only a mouse click away. There are amount of Rent a Large Dumpster in Buffalo Gap companies available online to handle the job for you.
Experience with a company in regards to this business also needs to be factored in. You could think that going waste is something that does not want much practical experience but the truth is in any industry or subject of work there are challenges then when someone really does work there, they will have developed ways of overcoming those challenges and ideal their work. Thus, when using a skilled company, it is possible to get top class services in comparison to the one that is only getting started. Consider this among other things such as proximity of a company while searching for Dumpster rental services.
Dumpsters are employed when there is additional garbage than one can readily take to your dump. Many trips might normally be expected. Few people stay close to some sort of dump, and they can be a international calls away. Operating garbage to them is not free. Persons must pay to the gas each trip to your dump needs. With the soaring cost of gasoline and the diminished fuel economy of a fully-loaded vehicle, most of these costs might be significant. When they're multiplied by a number of trips, the price of driving to some sort of dump is normally surprisingly substantial.
Currently people see the business on the web so it is simple for potential customers to pull-up your business label and see what exactly others currently have said about your service. Once your customers are done using your product or service you should motivate those that are extremely happy with your items to post an evaluation on your job site. Maybe it's helpful to place your website handle and make sure you leave an evaluation right on niche information.
Kind: Depending on your own waste management needs, it is possible looking to rent whether roll-off or a front loading package. A roll-off package is an open-top dumpster looks like a substantial metal container. It sheets of the distribution truck to the street, your own backyard, or even your driveway. Such a dumpster is ideal for momentary debris variety and eradication; you complete it yourself and therefore the trash is picked up in addition to hauled apart. Sizes for roll-off containers usually range from 10 to Forty five cubic yards. Some sort of front-load container is frequently smaller than the roll-off dumpster, has a cover, and is great for regularly timetabled garbage pick-up. Sizes for front-loading containers usually range from 1 to 10 cubic meters.
Larger dumpsters are also available as a next-day Rent a Large Dumpster in Buffalo Gap, TX and so they require virtually no additional ordering time. They can be positioned in the designated place chosen by the homeowner or even by the employed remodeling corporation. These dumpsters usually are large enough to manage huge amounts of spend such as is experience within a large redesigning job that will need the removal of old cabinets in addition to flooring. Regional trash collection cannot manage the excess trash involved in the construction undertaking and a Rent a Large Dumpster in Buffalo Gap, TX becomes necessary. A Rent a Large Dumpster in Buffalo Gap, TX works because it can be used for all types of trash and resources such as damaged glass, precious metals and plastics which can all be thrown in to one trash receptacle. Many of the convenient as it saves time period that would be necessary for sorting in addition to separating recyclables. Leasing a dumpster is usually convenient because it saves effort and time that would be needed to break down bigger items in to small enough dimensions to bag and put out by the path.
The throw off pots available at these Rent a Large Dumpster in Buffalo Gap, TX companies are placed on wheels which enable it to handle big tons of spend and are best suits for large assignments like building, house remodelling, yard cleanup. By employing a dumpster for your trash disposal needs, you will be arranged free from unsafe conditions which enable it to get your trash disposed from affordable expense.
Finally, when you are finished while using the rented dumpster, an appointment to the rental corporation for a pick-up becomes necessary. All companies have different rental periods, in addition to knowing the length of time allowed for a rental is imperative to know prior to making a repayment.
Before you go on the web and to a Rent a Large Dumpster in Buffalo Gap, TX internet site, try and execute a rough computation on how expensive is coming out of the building, so that you can obtain the right-sized dumpster. If you rent one that is too small you will possess to have it removed, emptied, and then brought back. This can cost a thing extra. Get one that's too big and that's a waste of space and cash. Most Rent a Large Dumpster in Buffalo Gap, TX companies will do their very best to help you assess your trash and recycling load.


text:
Stephen honoured to be invited to HMS Hermes service
A former Royal Navy crewman is heading out to India for a special service for an aircraft carrier close to his heart.
Stephen Robertson, from Newbiggin, is flying out to India for a de-commissioning ceremony for HMS Hermes.
Stephen Robertson, of Newbiggin, is one of 18 former crewmen invited out to India for the decommissioning service of the HMS Hermes.
The aircraft carrier played a major role in the Falklands War before it was sold to India in 1986.
He flew out to India today (Thursday) ahead of the special ceremony in Mumbai on Monday, and is delighted to know that although it is being decommissioned, HMS Hermes is to be turned into a museum.
Stephen, who left the Royal Navy in 1988, said: "I was only 19 when I served on her during the Falklands.
"There are 18 former crewmen going over, eight of us signed up to the Royal Navy at the same time and worked in the same department.
"I'm the only one from the North East going.
"I was quite surprised and honoured to receive the invite. It will be a privilege to see the old lady again.
"I came off the ship in 1982 after the War but it will still be an honour to attend her last service.
"She is going to be preserved as a museum in India, it is good that she is not going to the scrapyard."
Stephen, who is now a HGV driver working for Fergusons of Blyth, added: "It will be good to see my ex-Navy colleagues again. I do stay in touch with most of them but it will be good to see the others."
HMS Hermes was launched in 1954 and is in the Guinness World Records for the longest serving aircraft carrier before being sold to India in 1986.


allard architecture was selected to submit a design for an extension of the clubhouse for the tennis club Festina, in the middle of Amsterdam's Vondelpark. L.T.C. Festina is considered to be one of the most iconic and largest tennis clubs in the Netherlands. The club is hoping to expand its demographics by attracting younger members.
The existing part of the clubhouse dates back to 1926. The proposed extension follows the outline of the old sheepfold, but unlike the endoskeleton structure of the historical building, the new extension will be constructed with an exoskeleton. Thus the original building contrasts with the new, contemporary, "inside out" extension. This architectural contrast is intended to reflect the character of the younger generation and celebrate both the history and the current cultural relevance of the club.