text:
\subsubsection*{Acknowledgments}
This research was supported by DARPA (PPAML program, contract number FA8750-14-2-0004), IARPA (under research contract 2015-15061000003), the Office of Naval Research (under research contract N000141310333), the Army Research Office (under agreement number W911NF-13-1-0212), and gifts from Analog Devices and Google.
This research was conducted with Government support under and awarded by DoD, Air Force Office of Scientific Research, National Defense Science and Engineering Graduate (NDSEG) Fellowship, 32 CFR 168a.

\bibliographystyle{unsrtnat}

\section{Background}
\label{sec:background}
\vspace{-2mm}

Consider a generative probabilistic model with latent variables $X$ and observed variables $Y$.
We denote assignments to these variables by $x \in \mathcal{X}$ and $y \in \mathcal{Y}$.
Let $p(x,y)$ denote the joint density of the generative model.
The posterior density is $p(x|y) := p(x,y) / p(y)$ where $p(y) = \int p(x, y) dx$ is the marginal likelihood, or `evidence'.

Sampling-based approximate inference strategies including Markov chain Monte Carlo (MCMC, \citep{metropolis1953equation, hastings1970monte}), sequential Monte Carlo (SMC, \citep{del2006sequential}), annealed importance sampling (AIS, \citep{neal2001annealed}) and importance sampling with resampling (SIR, \citep{rubin1988using,smith1992bayesian}), generate samples of the latent variables that are approximately distributed according to $p(x|y)$.
Use of a sampling-based inference algorithm is often motivated by theoretical guarantees of exact convergence to the posterior in the limit of infinite computation (e.g. number of transitions in a Markov chain, number of importance samples in SIR).
However, how well the sampling distribution approximates the posterior distribution for finite computation is typically difficult to analyze theoretically or estimate empirically with confidence.

Variational inference \citep{jordan1999introduction} explicitly minimizes the approximation error of the approximating distribution $q_{\theta}(x)$ over parameters $\theta$ of a variational family.
The error is usually quantified using the Kullback-Leibler (KL) divergence from the approximation $q_{\theta}(x)$ to the posterior $p(x|y)$, denoted $\kl{q_{\theta}(x)}{p(x|y)}$.
Unlike sampling-based approaches, variational inference does not generally give exact results for infinite computation because the variational family does not include the posterior.
Minimizing the KL divergence is performed by maximizing the `evidence lower bound' (ELBO) $\mathcal{L} = \log p(y) - \kl{q_{\theta}(x)}{p(x|y)}$ over $\theta$.
Since $\log p(y)$ is usually unknown, the actual error (the KL divergence) of a variational approximation is also unknown.

\section{Discussion}
AIDE makes it practical to estimate bounds on the error of a broad class of approximate inference algorithms including sequential Monte Carlo (SMC), annealed importance sampling (AIS), sampling importance resampling (SIR), and variational inference.
AIDE's reliance on a gold-standard inference algorithm raises two questions that merit discussion:

\emph{If we already had an acceptable gold-standard, why would we want to evaluate other inference algorithms?}
Gold-standard algorithms such as very long MCMC runs, SMC runs with hundreds of thousands of particles, or AIS runs with a very fine annealing schedule, are often too slow to use in production.
AIDE make it possible to use gold-standard algorithms during an offline design and evaluation phase to quantitatively answer questions like ``how few particles or rejuvenation steps or samples can I get away with?'' or ``is my fast variational approximation good enough?''.
AIDE can thus help practitioners confidently apply Monte Carlo techniques in challenging, performance constrained applications, such as probabilistic robotics or web-scale machine learning.
In future work we think it will be valuable to build probabilistic models of AIDE estimates, conditioned on features of the data set, to learn offline what problem instances are easy or hard for different inference algorithms.
This may help practitioners bridge the gap between offline evaluation and production more rigorously.

\emph{How do we ensure that the gold-standard is accurate enough for the comparison with it to be meaningful?}
This is an intrinsically hard problem---we are not sure that near-exact posterior inference is really feasible, for most interesting classes of models.
In practice, we think that gold-standard inference algorithms will be calibrated based on a mix of subjective assumptions and heuristic testing---much like models themselves are tested.
For example, users could initially build confidence in a gold-standard algorithm by estimating the symmetric KL divergence from the posterior on simulated data sets (following the approach of \citet{grosse2016measuring}), and then use AIDE with the trusted gold-standard for a focused evaluation of target algorithms on real data sets of interest.
We do not think the subjectivity of the gold-standard assumption is a unique limitation of AIDE.

A limitation of AIDE is that its bias depends on the accuracy of meta-inference, i.e. inference over the auxiliary random variables used by an inference algorithm.
We currently lack an accurate meta-inference algorithm for MCMC samplers that do not employ annealing, and therefore AIDE is not yet suitable for use as a general MCMC convergence diagnostic.
Research on new meta-inference algorithms for MCMC and comparisons to standard convergence diagnostics \citep{gelman1992inference, geweke2004getting} are needed.
Other areas for future work include understanding how the accuracy of meta-inference depends on parameters of an inference algorithm, and more generally what makes an inference algorithm amenable to efficient meta-inference.

Note that AIDE does not rely on asymptotic exactness of the inference algorithm being evaluated.
An interesting area of future work is in using AIDE to study the non-asymptotic error of scalable but asymptotically biased sampling algorithms \citep{angelino2016patterns}.
It also seems fruitful to connect AIDE to results from theoretical computer science, including the computability \citep{ackerman2010computability} and complexity \citep{freer2010probabilistic,huggins2015convergence,chatterjee2015sample,agapiou2017importance} of probabilistic inference.
It should be possible to study the computational tractability of approximate inference empirically using AIDE estimates, as well as theoretically using a careful treatment of the variance of these estimates.
It also seems promising to use ideas from AIDE to develop Monte Carlo program analyses for samplers written in probabilistic programming languages.

\section{Experiments}

\subsection{Comparing the bias of AIDE for different types of inference algorithms}
\vspace{-1mm}
We used a Bayesian linear regression inference problem where exact posterior sampling is tractable to characterize the bias of AIDE when applied to three different types of target inference algorithms: sequential Monte Carlo (SMC), Metropolis-Hastings (MH), and variational inference.
For the gold-standard algorithm we used a posterior sampler with a tractable output density $q_{\GOLD}(x)$, which does not introduce bias into AIDE, so that the AIDE's bias could be completely attributed to the approximation error of meta-inference for each target algorithm.
Figure~\ref{fig:linreg} shows the results.
The bias of AIDE is acceptable for SMC, and AIDE is unbiased for variational inference, but better meta-inference algorithms for MCMC are needed to make AIDE practical for estimating the accuracy of MH.

\subsection{Evaluating approximate inference in a Hidden Markov model}
\vspace{-1mm}
We applied AIDE to measure the approximation error of SMC algorithms for posterior inference in a Hidden Markov model (HMM).
Because exact posterior inference in this HMM is tractable via dynamic programming, we used this opportunity to compare AIDE estimates obtained using the exact posterior as the gold-standard with AIDE estimates obtained using a `best-in-class' SMC algorithm as the gold-standard.
Figure~\ref{fig:hmm} shows the results, which indicate AIDE estimates using an approximate gold-standard algorithm can be nearly identical to AIDE estimates obtained with an exact posterior gold-standard.

\begin{figure}[ht]
\centering
\begin{minipage}[c]{0.20\textwidth}
\fontsize{8pt}{8pt}
\selectfont
\centering
\includegraphics[width=1.0\textwidth]{images/hmm/ground_truth_states.pdf}
Ground truth states\\
\vspace{0.1\textwidth}
\includegraphics[width=1.0\textwidth]{images/hmm/exact_posterior_marginals.pdf}
Posterior marginals\\
\vspace{0.1\textwidth}
\includegraphics[width=1.0\textwidth]{images/hmm/smc_cond_1000p_marginals.pdf}
SMC optimal proposal\\
1000 particles\\
(SMC gold standard)
\end{minipage}
\hspace{0.02\textwidth}
\begin{minipage}[c]{0.20\textwidth}
\small
\centering
Target algorithms\\
\hrule
\vspace{0.05\textwidth}
\fontsize{8pt}{8pt}
\selectfont
\includegraphics[width=1.0\textwidth]{images/hmm/smc_prior_1p_marginals.pdf}\\
SMC prior proposal\\
1 particle\\
\vspace{0.05\textwidth}
\includegraphics[width=1.0\textwidth]{images/hmm/smc_prior_10p_marginals.pdf}\\
SMC prior proposal\\
10 particles\\
\vspace{0.05\textwidth}
\includegraphics[width=1.0\textwidth]{images/hmm/smc_cond_100p_marginals.pdf}\\
SMC optimal proposal\\
100 particles
\end{minipage}
\hspace{0.02\textwidth}
\begin{minipage}[c]{0.53\textwidth}
\begin{minipage}[c]{0.49\linewidth}
\small
\centering
Measuring accuracy\\
of target algorithms using\\
posterior as gold-standard\\
\includegraphics[width=1.0\textwidth]{images/hmm/divergences_exact_gold_standard.pdf}
\end{minipage}
\begin{minipage}[c]{0.49\linewidth}
\small
\centering
Measuring accuracy\\
of target algorithms using\\
SMC gold-standard\\
\includegraphics[width=1.0\textwidth]{images/hmm/divergences_approximate_gold_standard.pdf}
\end{minipage}
\includegraphics[width=1.0\textwidth]{images/hmm/divergences_legend.pdf}
\end{minipage}
\caption{
Comparing use of an exact posterior as the gold-standard and a `best-in-class' approximate algorithm as the gold-standard, when measuring accuracy of target inference algorithms with AIDE.
We consider inference in an HMM, so that exact posterior sampling is tractable using dynamic programming.
Left: Ground truth latent states, posterior marginals, and marginals of the the output of a gold-standard and three target SMC algorithms (A,B,C) for a particular observation sequence.
Right: AIDE estimates using the exact gold-standard and using the SMC gold-standard are nearly identical.
The estimated divergence bounds decrease as the number of particles in the target sampler increases.
The optimal proposal outperforms the prior proposal.
Increasing $M_{\TARGET}$ tightens the estimated divergence bounds.
We used $M_{\GOLD} = 1$.
}
        \label{fig:hmm}
\end{figure}

\begin{figure}[ht]
        \centering
        \includegraphics[width=0.35\textwidth]{images/dpmm/aide.pdf}
        \includegraphics[width=0.35\textwidth]{images/dpmm/num_clusters.pdf}
        \includegraphics[width=0.28\textwidth]{images/dpmm/legend.pdf}
        \caption{
Contrasting AIDE against a heuristic convergence diagnostic for evaluating the accuracy of approximate inference in a Dirichlet process mixture model (DPMM).
The heuristic compares the expected number of clusters under the target algorithm to the expectation under the gold-standard algorithm \citep{ulker2010sequential}.
White circles identify single-particle likelihood-weighting, which samples from the prior.
AIDE clearly indicates that single-particle likelihood-weighting is inaccurate, but
the heuristic suggests it is accurate.
Probe functions like the expected number of clusters can be error prone measures of convergence because they only track convergence along a specific projection of the distribution.
In contrast, AIDE estimates a joint KL divergence.
Shaded areas in both plots show the standard error.
The amount of target inference computation used is the same for the two techniques, although AIDE performs a gold-standard meta-inference run for each target inference run.
}
	\label{fig:dpmm_first}
\end{figure}

\subsection{Comparing AIDE to alternative inference evaluation techniques}
\vspace{-1mm}
A key feature of AIDE is that it applies to different types of inference algorithms.
We compared AIDE to two existing techniques for evaluating the accuracy of inference algorithms that share this feature:
(1) comparing log marginal likelihood (LML) estimates made by a target algorithm against LML estimates made by a gold-standard algorithm, and
(2) comparing the expectation of a probe function under the approximating distribution to the same expectation under the gold-standard distribution \citep{ulker2010sequential}.
Figure~\ref{fig:bimodal} shows a comparison of AIDE to LML, on a inference problem where the posterior is bimodal.
Figure~\ref{fig:dpmm_first} shows a comparison of AIDE to a `number of clusters' probe function in a Dirichlet process mixture model inference problem for a synthetic data set.
We also used AIDE to evaluate the accuracy of several SMC algorithms for DPMM inference on a real data set of galaxy velocities \citep{drinkwater2004large} relative to an SMC gold-standard.
This experiment is described in the supplement due to space constraints.

\section{Estimating the symmetrized KL divergence between inference algorithms}
This section defines our mathematical formalism for analyzing inference algorithms;
shows how to represent SMC, MCMC, rejection sampling, and variational inference in this formalism;
and introduces the Auxiliary Inference Divergence Estimator (AIDE), an algorithm for estimating the symmetrized KL divergence between two inference algorithms.

\subsection{Generative inference models and meta-inference algorithms} \label{sec:formalism}
We define an inference algorithm as a procedure that produces a single approximate posterior sample.
Repeated runs of the algorithm give independent samples.
For each inference algorithm, there is an `output density' $q(x)$ that represents the probability that the algorithm returns a given sample $x$ on any given run of the algorithm.
Note that $q(x)$ depends on the observations $y$ that define the inference problem, but we suppress that in the notation.
The inference algorithm is accurate when $q(x) \approx p(x|y)$ for all $x$.
We denote a sample produced by running the algorithm by $x \sim q(x)$.

A naive simple Monte Carlo estimator of the KL divergence between the output distributions of two inference algorithms requires the output densities of both algorithms.
However, it is typically intractable to compute the output densities of sampling-based inference algorithms like MCMC and SMC, because that would require marginalizing over all possible values that the random variables drawn during the algorithm could possibly take.
A similar difficulty arises when computing the marginal likelihood $p(y)$ of a generative probabilistic model $p(x, y)$.
This suggests that we treat the inference algorithm as a probabilistic model, estimate its output density using ideas from marginal likelihood estimation, and use these estimates in a Monte Carlo estimator of the divergence.
We begin by making the analogy between an inference algorithm and a probabilistic model explicit:

\theoremstyle{definition}
\begin{definition}[Generative inference model] \label{def:geninfmodel}
A \emph{generative inference model} is a tuple $(\mathcal{U}, \mathcal{X}, q)$ where $q(u, x)$ is a joint density defined on $\mathcal{U} \times \mathcal{X}$.
A generative inference model \emph{models} an inference algorithm if the output density of the inference algorithm is the marginal likelihood $q(x) = \int q(u, x) du$ of the model for all $x$.
An element $u \in \mathcal{U}$ represents a complete assignment to the internal random variables within the inference algorithm, and is called a `trace'.
The ability to simulate from $q(u, x)$ is required, but the ability to compute the density $q(u, x)$ is not.
A simulation, denoted $u, x \sim q(u, x)$, may be obtained by running the inference algorithm and recording the resulting trace $u$ and output $x$.\footnote{The trace data structure could in principle be obtained by writing the inference algorithm in a probabilistic programming language like Church \citep{goodman2008church}, but the computational overhead would be high.}
\end{definition}

A generative inference model can be understood as a generative probabilistic model where the $u$ are the latent variables and the $x$ are the observations.
Note that two different generative inference models may use different representations for the internal random variables of the same inference algorithm.
In practice, constructing a generative inference model from an inference algorithm amounts to defining the set of internal random variables.
For marginal likelihood estimation in a generative inference model, we use a `meta-inference' algorithm:

\theoremstyle{definition}
\begin{definition}[Meta-inference algorithm] \label{def:metainference}
For a given generative inference model $(\mathcal{U}, \mathcal{X}, q)$, a \emph{meta-inference algorithm} is a tuple $(r, \xi)$ where $r(u;x)$ is a density on traces $u \in \mathcal{U}$ of the inference algorithm, indexed by outputs $x \in \mathcal{X}$ of the inference algorithm, and where $\xi(u, x)$ is the following function of $u$ and $x$ for some $Z > 0$:
{\setlength{\mathindent}{5.5cm}
\begin{equation}
\xi(u, x) := Z \frac{q(u, x)}{r(u; x)}
\end{equation}}%
We require the ability to sample $u \sim r(u;x)$ given a value for $x$, and the ability to evaluate $\xi(u, x)$ given $u$ and $x$.
We call a procedure for sampling from $r(u;x)$ a `meta-inference sampler'.
We do not require the ability to evaluate the density $r(u;x)$.
\end{definition}
A meta-inference algorithm is considered accurate for a given $x$ if $r(u;x) \approx q(u|x)$ for all $u$.
Conceptually, a meta-inference sampler tries to answer the question `how could my inference algorithm have produced this output $x$?'
Note that if it is tractable to evaluate the marginal likelihood $q(x)$ of the generative inference model up to a normalizing constant, then it is not necessary to represent internal random variables for the inference algorithm, and a generative inference model can define the trace as an empty token $u = ()$ with $\mathcal{U} = \{()\}$.
In this case, the meta-inference algorithm has $r(u;x) = 1$ for all $x$ and $\xi(u, x) = Z q(x)$.

\subsection{Examples} \label{sec:examples}
We now show how to construct generative inference models and corresponding meta-inference algorithms for SMC, AIS, MCMC, SIR, rejection sampling, and variational inference.
The meta-inference algorithms for AIS, MCMC, and SIR are derived as special cases of a generic SMC meta-inference algorithm.

\paragraph{Sequential Monte Carlo.}
We consider a general class of SMC samplers introduced by \citet{del2006sequential}, which can be used for approximate inference in both sequential state space and non-sequential models.
We briefly summarize a slightly restricted variant of the algorithm here, and refer the reader to the supplement and \citet{del2006sequential} for full details.
The SMC algorithm propagates $P$ weighted particles through $T$ steps, using proposal kernels $k_t$ and multinomial resampling based on weight functions $w_1(x_1)$ and $w_t(x_{t-1}, x_t)$ for $t > 1$ that are defined in terms of `backwards kernels' $\ell_t$ for $t=2\ldots T$.
Let $x_t^i$, $w_t^i$ and $W_t^i$ denote the value, unnormalized weight, and normalized weight of particle $i$ at time $t$, respectively.
We define the output sample $x$ of SMC as a single draw from the particle approximation at the final time step, which is obtained by sampling a particle index $I_T \sim \mbox{Categorical}(W_T^{1:P})$ where $W_T^{1:P}$ denotes the vector of weights $(W_T^1,\ldots,W_T^P)$, and then setting $x \gets x^{I_T}_T$.
The generative inference model uses traces of the form $u = (\mathbf{x}, \mathbf{a}, I_T)$, where $\mathbf{x}$ contains the values of all particles at all time steps and where $\mathbf{a}$ (for `ancestor') contains the index $a_t^i \in \{1\ldots P\}$ of the parent of particle $x_{t+1}^i$ for each particle $i$ and each time step $t = 1\ldots T-1$.
Algorithm~\ref{alg:smc-metainference} defines a canonical meta-inference sampler for this generative inference model that takes as input a latent sample $x$ and generates an SMC trace $u \sim r(u;x)$ as output.
The meta-inference sampler first generates an ancestral trajectory of particles $(x_1^{I_1}, x_2^{I_2}, \ldots, x_T^{I_T})$ that terminates in the output sample $x$, by sampling sequentially from the backward kernels $\ell_t$, starting from $x_T^{I_T} = x$.
Next, it runs a conditional SMC update \citep{andrieu2010particle} conditioned on the ancestral trajectory.
For this choice of $r(u;x)$ and for $Z = 1$, the function $\xi(u, x)$ is closely related to the marginal likelihood estimate $\widehat{p(y)}$ produced by the SMC scheme:\footnote{AIDE also applies to approximate inference algorithms for undirected probabilistic models; the marginal likelihood estimate is replaced with the estimate of the partition function.}
 $\xi(u, x) = p(x, y) / \widehat{p(y)}$.
See supplement for derivation.

\begin{algorithm}[h]
\footnotesize
\caption{Generalized conditional SMC (a canonical meta-inference sampler for SMC)}
\label{alg:smc-metainference}
\begin{minipage}{0.5\textwidth}
\begin{algorithmic}
    \Require Latent sample $x$, SMC parameters
    \State $I_T \sim \mbox{Uniform}(1\ldots P)$
    \State $x_T^{I_T} \gets x$
    \For{$t \gets T-1\ldots 1$}
        \State $I_t \sim \mbox{Uniform}(1\ldots P)$
        \State \LeftComment{Sample from backward kernel}
        \State $x_t^{I_t} \sim \ell_{t+1}(\cdot; x_{t+1}^{I_{t+1}})$
    \EndFor
    \For{$i \gets 1\ldots P$}
        \IIf{$i \ne I_1$} $x^i_1 \sim k_1(\cdot)$
        \State $w^i_1 \gets w_1(x^i_1)$
    \EndFor
    \For{$t \gets 2\ldots T$}
        \State $W_{t-1}^{1:P} \gets w_{t-1}^{1:P} / (\sum_{i=1}^Pw_{t-1}^i )$
        \For{$i \gets 1\ldots P$}
            \If{$i = I_t$} $a_{t-1}^i \gets I_{t-1}$
            \Else
                \State $a_{t-1}^i \sim \mbox{Categorical}(W_{t-1}^{1:P})$
                \State $x^i_t \sim k_t(\cdot;x_{t-1}^{a_{t-1}^i})$
            \EndIf
            \State $w_t^i \gets w_t(x_{t-1}^{a_{t-1}^i}, x_t^i)$
        \EndFor
    \EndFor
    \State $u \gets (\mathbf{x}, \mathbf{a}, I_T)\quad$ \LeftComment{Return an SMC trace}
    \State \Return $u$
\end{algorithmic}
\end{minipage}%
\begin{minipage}{0.5\textwidth}
\begin{flushright}
\resizebox{0.8\textwidth}{!}{
\begin{tikzpicture}
\tikzstyle{line} = [draw, thick]
\tikzstyle{particle} = [circle, draw, thick, align=center]

\node[particle, fill={rgb:red,1;white,4}] (x11) at (1, 1) {$x_1^1$};
\node[particle] (x12) at (2, 1) {$x_1^2$};
\node[particle] (x13) at (3, 1) {$x_1^3$};
\node[particle] (x14) at (4, 1) {$x_1^4$};

\node[particle] (x21) at (1, 2.5) {$x_2^1$};
\node[particle] (x22) at (2, 2.5) {$x_2^2$};
\node[particle, fill={rgb:red,1;white,4}] (x23) at (3, 2.5) {$x_2^3$};
\node[particle] (x24) at (4, 2.5) {$x_2^4$};

\node[particle] (x31) at (1, 4) {$x_3^1$};
\node[particle, fill={rgb:red,1;white,4}] (x32) at (2, 4) {$x_3^2$};
\node[particle] (x33) at (3, 4) {$x_3^3$};
\node[particle] (x34) at (4, 4) {$x_3^4$};

\node[circle, draw, thick, align=center, fill={rgb:black,1;white,4}, minimum size=22] (output) at (2.5, 5.5) {$x$};
\node[right=1pt of output, align=left] {\small Latent sample\\\small(input to meta-inference sampler)};

\draw [->, line width=1.75] (node cs:name=x23,angle=225) -- node[above,pos=0.7] {$\ell_2$} (node cs:name=x11,angle=45);
\draw [->, line width=1.75] (node cs:name=x32,angle=315) -- node[below,pos=0.1,shift={(-2pt,0pt)}] {$\ell_3$} (node cs:name=x23,angle=135);
\draw [->, line width=1.75] (node cs:name=output,angle=250) -- node[below,pos=0.1,shift={(3pt,0pt)}] {$\delta$} (node cs:name=x32,angle=70);

\draw [->, thick] (node cs:name=x11,angle=90) -- (node cs:name=x21,angle=270);
\draw [->, thick] (node cs:name=x13,angle=135) -- (node cs:name=x22,angle=315);
\draw [->, thick] (node cs:name=x14,angle=90) -- (node cs:name=x24,angle=270);

\draw [->, thick] (node cs:name=x23,angle=45) -- (node cs:name=x34,angle=225);
\draw [->, thick] (node cs:name=x23,angle=90) -- (node cs:name=x33,angle=270);
\draw [->, thick] (node cs:name=x22,angle=135) -- (node cs:name=x31,angle=315);

\node at (5.3, 1) {$I_1 = 1$};
\node at (5.3, 2.5) {$I_2 = 3$};
\node at (5.3, 4) {$I_3 = 2$};

\node (label) at (2.5, 0) {$T = 3$};

\node[particle, fill={rgb:red,1;white,4}, below=0.5cm of label] (example) {$x_t^i$};
\node[right=0.1cm of example, align=left] {Member of ancestral\\trajectory};

\end{tikzpicture}
}
\end{flushright}
\end{minipage}
\end{algorithm}

\paragraph{Annealed importance sampling.} \label{sec:ais}
When a single particle is used ($P = 1$), and when each forward kernel $k_t$ satisfies detailed balance for some intermediate density, the SMC algorithm simplifies to annealed importance sampling (AIS, \citep{neal2001annealed}), and the canonical SMC meta-inference inference (Algorithm~\ref{alg:smc-metainference}) consists of running the forward kernels in reverse order, as in the reverse annealing algorithm of \citet{grosse2015sandwiching, grosse2016measuring}.
The canonical meta-inference algorithm is accurate ($r(u;x) \approx q(u;x)$) if the AIS Markov chain is kept close to equilibrium at all times.
This is achieved if the intermediate densities form a sufficiently fine-grained sequence.
See supplement for analysis.

\paragraph{Markov chain Monte Carlo.} \label{sec:mcmc}
We define each run of an MCMC algorithm as producing a single output sample $x$ that is the iterate of the Markov chain produced after a predetermined number of burn-in steps has passed.
We also assume that each MCMC transition operator satisfies detailed balance with respect to the posterior $p(x|y)$.
Then, this is formally a special case of AIS.
However, unless the Markov chain was initialized near the posterior $p(x|y)$, the chain will be far from equilibrium during the burn-in period, and the AIS meta-inference algorithm will be inaccurate.

\paragraph{Importance sampling with resampling.}
Importance sampling with resampling, or SIR \citep{rubin1988using} can be seen as a special case of SMC if we set the number of steps to one ($T = 1$).
The trace of the SIR algorithm is then the set of particles $x_1^i$ for $i \in \{1,\ldots,P\}$ and output particle index $I_1$.
Given output sample $x$, the canonical SMC meta-inference sampler then simply samples $I_1 \sim \mbox{Uniform}(1\ldots P)$, sets $x_1^{I_1} \gets x$, and samples the other $P - 1$ particles from the importance distribution $k_1(x)$.

\paragraph{Rejection sampling.} \label{sec:rejection}
To model a rejection sampler for a posterior distribution $p(x|y)$, we assume it is tractable to evaluate the unnormalized posterior density $p(x, y)$.
We define $\mathcal{U} = \{()\}$ as described in Section~\ref{sec:formalism}.
For meta-inference, we define $Z = p(y)$ so that $\xi(u, x) = p(y) p(x|y) = p(x, y)$.
It is not necessary to represent the internal random variables of the rejection sampler.

\paragraph{Variational inference.}
We suppose a variational approximation $q_{\theta}(x)$ has been computed through optimization over the variational parameters $\theta$.
We assume that it is possible to sample from the variational approximation, and evaluate its normalized density.
Then, we use $\mathcal{U} = \{()\}$ and $Z = 1$ and $\xi(u, x) = q_{\theta}(x)$.
Note that this formulation also applies to amortized variational inference algorithms, which reuse the parameters $\theta$ for inference across different observation contexts $y$.

\subsection{The auxiliary inference divergence estimator} \label{sec:estimator}
Consider a probabilistic model $p(x, y)$, a set of observations $y$, and two inference algorithms that approximate $p(x|y)$.
One of the two inference algorithms is considered the `gold-standard', and has a generative inference model $(\mathcal{U},\mathcal{X},q_{\GOLD})$ and a meta-inference algorithm $(r_{\GOLD}, \xi_{\GOLD})$.
The second algorithm is considered the `target' algorithm, with a generative inference model $(\mathcal{V},\mathcal{X},q_{\TARGET})$ (we denote a trace of the target algorithm by $v \in \mathcal{V}$), and a meta-inference algorithm $(r_{\TARGET}, \xi_{\TARGET})$.
This section shows how to estimate an upper bound on the symmetrized KL divergence between $q_{\GOLD}(x)$ and $q_{\TARGET}(x)$, which is:
\begin{equation} \label{eq:symkl}
    \kl{q_{\GOLD}(x)}{q_{\TARGET}(x)} + \kl{q_{\TARGET}(x)}{q_{\GOLD}(x)} = \E_{x \sim q_{\GOLD}(x)} \left[ \log \frac{q_{\GOLD}(x)}{q_{\TARGET}(x)} \right] + \E_{x \sim q_{\TARGET}(x)} \left[ \log \frac{q_{\TARGET}(x)}{q_{\GOLD}(x)} \right]
\end{equation}
We take a Monte Carlo approach.
Simple Monte Carlo applied to the Equation~(\ref{eq:symkl}) requires that $q_{\GOLD}(x)$ and $q_{\TARGET}(x)$ can be evaluated, which would prevent the estimator from being used when either inference algorithm is sampling-based.
Algorithm~\ref{alg:estimator} gives the Auxiliary Inference Divergence Estimator (AIDE), an estimator of the symmetrized KL divergence that only requires evaluation of $\xi_{\GOLD}(u, x)$ and $\xi_{\TARGET}(v, x)$ and not $q_{\GOLD}(x)$ or $q_{\TARGET}(x)$, permitting its use with sampling-based inference algorithms.

\input{algorithm.tex}
The generic AIDE algorithm above is defined in terms of abstract generative inference models and meta-inference algorithms.
For concreteness, the supplement contains the AIDE algorithm specialized to the case when the gold-standard is AIS and the target is a variational approximation.

\begin{restatable}{theorem}{elboconsistency}
\label{thm:elbo-estimator}
    The estimate $\hat{D}$ produced by AIDE is an upper bound on the symmetrized KL divergence in expectation, and the expectation is nonincreasing in AIDE parameters $M_{\GOLD}$ and $M_{\TARGET}$. \end{restatable}
See supplement for proof.
Briefly, AIDE estimates an upper bound on the symmetrized divergence in expectation because it uses unbiased estimates of $q_{\TARGET}(x_n)$ and $q_{\GOLD}(x_n)^{-1}$ for $x_n \sim q_{\GOLD}(x)$, and unbiased estimates of $q_{\GOLD}(x_n')$ and $q_{\TARGET}(x_n')^{-1}$ for $x_n' \sim q_{\TARGET}(x)$.
For $M_{\GOLD} = 1$ and $M_{\TARGET} = 1$, AIDE over-estimates the true symmetrized divergence by:
{\setlength{\mathindent}{0cm}
\begin{equation} \label{eq:general_gap}
\begin{array}{ll}
&\E[ \hat{D} ] - (\kl{q_{\GOLD}(x)}{q_{\TARGET}(x)} + \kl{q_{\TARGET}(x)}{q_{\GOLD}(x)}) =\\[5pt]
    &\left(\begin{array}{ll}
        &\E_{x \sim q_{\GOLD}(x)} \left[ \kl{q_{\GOLD}(u|x)}{r_{\GOLD}(u;x)} + \kl{r_{\TARGET}(v;x)}{q_{\TARGET}(v|x)} \right]\\
        + &\E_{x \sim q_{\TARGET}(x)} \left[ \kl{q_{\TARGET}(v|x)}{r_{\TARGET}(v;x)} + \kl{r_{\GOLD}(u;x)}{q_{\GOLD}(u|x)} \right]
    \end{array}\right)
\end{array}
\begin{array}{c}
\text{Bias of AIDE}\\
\text{for } {\scriptstyle M_{\GOLD} = M_{\TARGET} = 1}
\end{array}
\end{equation}}%
Note that this expression involves KL divergences between the meta-inference sampling densities ($r_{\GOLD}(u;x)$ and $r_{\TARGET}(v;x)$) and the posteriors in their respective generative inference models
($q_{\GOLD}(u|x)$ and $q_{\TARGET}(v|x)$).
Therefore, the approximation error of meta-inference determines the bias of AIDE.
When both meta-inference algorithms are exact ($r_{\GOLD}(u;x) = q_{\GOLD}(u|x)$ for all $u$ and $x$ and $r_{\TARGET}(v;x) = q_{\TARGET}(v|x)$ for all $v$ and $x$), AIDE is unbiased.
As $M_{\GOLD}$ or $M_{\TARGET}$ are increased, the bias decreases (see Figure~\ref{fig:linreg} and Figure~\ref{fig:hmm} for examples).
If the generative inference model for one of the algorithms does not use a trace (i.e. $\mathcal{U} = \{()\}$ or $\mathcal{V} = \{()\}$), then that algorithm does not contribute a KL divergence term to the bias of Equation~(\ref{eq:general_gap}).
The analysis of AIDE is equivalent to that of \citet{grosse2016measuring} when the target algorithm is AIS and $M_{\TARGET} = M_{\GOLD} = 1$ and the gold-standard inference algorithm is a rejection sampler.

\section{Introduction}
\label{sec:intro}
Approximate probabilistic inference algorithms are central to diverse disciplines, including statistics, robotics, machine learning, and artificial intelligence.
Popular approaches to approximate inference include sequential Monte Carlo, variational inference, and Markov chain Monte Carlo.
A key problem faced by practitioners is measuring the accuracy of an approximate inference algorithm on a specific data set.
The accuracy is influenced by complex interactions between the specific data set in question, the model family, the algorithm tuning parameters such as the number of iterations, and any associated proposal distributions and/or approximating variational family.
Unfortunately, practitioners assessing the accuracy of inference have to rely on heuristics that are either brittle or specialized for one type of algorithm \citep{cowles1996markov}, or both.
For example, log marginal likelihood estimates can be used to assess the accuracy of sequential Monte Carlo and variational inference, but these estimates can fail to significantly penalize an algorithm for missing a posterior mode.
Expectations of probe functions do not assess the full approximating distribution, and they require design specific to each model.

This paper introduces an algorithm for estimating the symmetrized KL divergence between the output distributions of a broad class of exact and approximate inference algorithms.
The key idea is that inference algorithms can be treated as probabilistic models and the random variables used within the inference algorithm can be viewed as latent variables.
We show how sequential Monte Carlo, Markov chain Monte Carlo, rejection sampling, and variational inference can be represented in a common mathematical formalism based on two new concepts: \emph{generative inference models} and \emph{meta-inference algorithms}.
Using this framework, we introduce the Auxiliary Inference Divergence Estimator (AIDE), which estimates the symmetrized KL divergence between the output distributions of two inference algorithms that have both been endowed with a meta-inference algorithm.
We also show that the conditional SMC update of \citet{andrieu2010particle} and the reverse AIS Markov chain of \citet{grosse2015sandwiching} are both special cases of a `generalized conditional SMC update', which we use as a canonical meta-inference algorithm for SMC.
AIDE is a practical tool for measuring the accuracy of SMC and variational inference algorithms relative to gold-standard inference algorithms.
Note that this paper does not provide a practical solution to the MCMC convergence diagnosis problem.
Although in principle AIDE can be applied to MCMC, to do so in practice will require more accurate meta-inference algorithms for MCMC to be developed.

\begin{figure}[t]
\setlength{\abovecaptionskip}{10pt}
\centering
\resizebox{\textwidth}{!}{
\begin{tikzpicture}
\tikzstyle{line} = [draw, thick]

\node (aide) at ( 0, 0) [rectangle, draw, text width=4cm, align=center]
    {\bf AIDE\\ Auxiliary Inference Divergence Estimator};

\node (gold) at ( -2.5, 2.5) [align=center]
    {Gold standard\\inference algorithm};

\node (target) at ( 2.5, 2.5) [align=center]
    {Target inference algorithm\\(the algorithm being measured)};

\node (estimate) at (0, -2)  [align=center]
    {Symmetrized KL divergence estimate $\hat{D}$\\
     $\hat{D} \approx D_{\mbox{\scriptsize KL}}(\mbox{gold-standard} || \mbox{target})
                    + D_{\mbox{\scriptsize KL}}(\mbox{target} || \mbox{gold-standard})$};

\draw [->, thick] (node cs:name=gold, anchor=south)
    .. controls +(0,-1) and +(0,1)
    .. (node cs: name=aide,angle=110);

\draw [->, thick] (node cs:name=target, anchor=south)
    .. controls +(0,-1) and +(0,1)
    .. (node cs: name=aide,angle=70);

\draw [->, thick] (aide.south) -- (estimate.north);

\node (Ngold) at (-3, 0.7) [align=right]
    {$N_{\GOLD}$};
\node (Ngoldtext) [align=right, left=0mm of Ngold]
    {Number of gold-standard\\inference runs};

\draw [->, thick] (Ngold.east)
    .. controls +(0.5, 0) and +(-0.5, 0)
    .. (node cs: name=aide, angle=175);

\node (Ntarget) at (3, 0.7) [align=left]
    {$N_{\TARGET}$};
\node (Ntargettext) [align=left, right=0mm of Ntarget]
    {Number of target\\inference runs};

\draw [->, thick] (Ntarget.west)
    .. controls +(-0.5, 0) and +(0.5, 0)
    .. (node cs:name=aide, angle=5);

\node (Mgold) at (-3, -0.7) [align=right]
    {$M_{\GOLD}$};
\node (Mgoldtext) [align=right, left=0mm of Mgold]
    {Number of meta-inference\\runs for gold-standard};

\draw [->, thick] (Mgold.east)
    .. controls +(0.5, 0) and +(-0.5, 0)
    .. (node cs: name=aide, angle=185);

\node (Mtarget) at (3, -0.7) [align=left]
    {$M_{\TARGET}$};
\node (Mtargettext) at (5, -0.7) [align=left, right=0mm of Mtarget]
    {Number of meta-inference\\runs for target};

\draw [->, thick] (Mtarget.west)
    .. controls +(-0.5, 0) and +(0.5, 0)
    .. (node cs: name=aide, angle=-5);

\end{tikzpicture}
}
\caption{
Using AIDE to estimate the accuracy of a target inference algorithm relative to a gold-standard inference algorithm.
AIDE is a Monte Carlo estimator of the symmetrized Kullback-Leibler (KL) divergence between the output distributions of two inference algorithms.
AIDE uses \emph{meta-inference}: inference over the internal random choices made by an inference algorithm.
}
\vspace{4mm}
\setlength{\abovecaptionskip}{5pt}
    \includegraphics[width=1.0\textwidth]{images/linear_regression_combined.pdf}
    \caption{
{\bf AIDE applies to SMC, variational, and MCMC algorithms.}
Left: AIDE estimates for SMC converge to zero, as expected.
Right: AIDE estimates for variational inference converge to a nonzero asymptote that depends on the variational family.
Middle: The symmetrized divergence between MH and the posterior converges to zero, but AIDE over-estimates the divergence in expectation.
Although increasing the number of meta-inference runs $M_{\TARGET}$ reduces the bias of AIDE, AIDE is not yet practical for measuring MH accuracy due to inaccurate meta-inference for MH.
}
\label{fig:linreg}
\end{figure}

\section{Related Work}
Diagnosing the convergence of approximate inference is a long-standing problem.
Most existing work is either tailored to specific inference algorithms \citep{kong1992note}, designed to detect lack of exact convergence \citep{cowles1996markov}, or both.
Estimators of the non-asymptotic approximation error of general approximate inference algorithms have received less attention.
\citet{gorham2015measuring} propose an approach that applies to arbitrary sampling algorithms but relies on special properties of the posterior density such as log-concavity.
Our approach does not rely on special properties of the posterior distribution.

Our work is most closely related to Bounding Divergences with REverse Annealing (BREAD, \citep{grosse2016measuring}) which also estimates upper bounds on the symmetric KL divergence between the output distribution of a sampling algorithm and the posterior distribution.
AIDE differs from BREAD in two ways:
First, whereas BREAD handles single-particle SMC samplers and annealed importance sampling (AIS), AIDE handles a substantially broader family of inference algorithms including SMC samplers with both resampling and rejuvenation steps, AIS, variational inference, and rejection samplers.
Second, BREAD estimates divergences between the target algorithm's sampling distribution and the posterior distribution, but the exact posterior samples necessary for BREAD's theoretical properties are only readily available when the observations $y$ that define the inference problem are simulated from the generative model.
Instead, AIDE estimates divergences against an exact or approximate gold-standard sampler on real (non-simulated) inference problems.
Unlike BREAD, AIDE can be used to evaluate inference in both generative and undirected models.

AIDE estimates the error of sampling-based inference using a mathematical framework with roots in variational inference.
Several recent works have treated sampling-based inference algorithms as variational approximations.
The Monte Carlo Objective (MCO) formalism of \citet{maddison2017filtering} is closely related to our formalism of generative inference models and meta-inference algorithms---indeed a generative inference model and a meta-inference algorithm with $Z=1$ give an MCO defined by: $\mathcal{L}(y, p) = \mathbb{E}_{u, x \sim q(u, x)} [\log (p(x, y) / \xi(u, x))]$, where $y$ denotes observed data.
In independent and concurrent work to our own, \citet{naesseth2017variational, maddison2017filtering} and \citet{le2017auto} treat SMC as a variational approximation using constructions similar to ours.
In earlier work, \citet{salimans2015markov} recognized that MCMC samplers can be treated as variational approximations.
However, these works are concerned with optimization of variational objective functions instead of estimation of KL divergences, and do not involve generating a trace of a sampler from its output.

\section{Sequential Monte Carlo}
An SMC sampler template based on \citep{del2006sequential} is reproduced in Algorithm~\ref{alg:smc}.
The algorithm evolves a set of $P$ particles to approximate a sequence of target distributions using a combination of proposal kernels, weighting, and resampling steps.
The final target distribution in the sequence is typically the posterior $p(x|y)$.
In our version, the algorithm resamples once from the final weighted particle approximation and returns this particle as its output sample $x$.
Specifically, the algorithm uses a sequence of unnormalized target densities $\tilde{p}_t$ defined on spaces $\mathcal{X}_t$ for $t=1\ldots T$, with $\mathcal{X}_T = \mathcal{X}$ (the original space of latent variables in the generative model) and $\tilde{p}_T(x) := p(x, y)$.
The algorithm also makes use of an initialization kernel $k_1$ defined on $\mathcal{X}_1$, proposal kernels $k_t$ defined on $\mathcal{X}_t$ and indexed by $\mathcal{X}_{t-1}$ for $t=2\ldots T$, and backward kernels $\ell_t$ defined on $\mathcal{X}_{t-1}$ and indexed by $\mathcal{X}_t$ for $t=2\ldots T$.
For simplicity of analysis we assume that resampling occurs at every step in the sequence.
The weight functions used in the algorithm are:
\begin{equation}
w_1(x_1^i) := \frac{\tilde{p}_1(x_1^i)}{k_1(x_1^i)}\;\;\;\;\;\;\;\;\;
w_t(x_{t-1}^j, x_t^i) := \frac{\tilde{p}_t(x_t^i) \ell_t\left(x_{t-1}^j;x_t^i\right)}{\tilde{p}_{t-1}\left(x_{t-1}^j\right) k_t\left(x_t^i;x_{t-1}^j\right)}
\end{equation}
Note that Algorithm~\ref{alg:smc} does not sample from the backward kernels, which serve to define the extended target densities $\tilde{p}_t(x_t) \prod_{s=2}^t \ell_s(x_{s-1}; x_s)$ that justify the SMC sampler as a sequential importance sampler \citep{del2006sequential}.
When $P = 1$, $\mathcal{X}_t = \mathcal{X}$ for all $t$, $k_t(x_t; x_{t-1})$ is a detailed balance transition operator for $p_{t-1}$, and $\ell_t = k_t$, the algorithm reduces to AIS.\footnote{More generally, the proposal kernel $k_t$ needs to have stationary distribution $p_{t-1}$. The backward kernel $\ell_t$ is the `reversal' of $k_t$ as defined in \cite{neal2001annealed}. When the proposal kernel satisfies detailed balance, it is its own reversal, and therefore sampling from the backward kernel is identical to sampling from the forward kernel.}
The particle filter without rejuvenation \citep{gordon1993novel} is also a special case of Algorithm~\ref{alg:smc}.
A variety of other SMC variants can also be seen to be special cases of this formulation \citep{del2006sequential}.
\begin{algorithm}
\caption{Sequential Monte Carlo} \label{alg:smc}
\begin{algorithmic}
    \For{$i \gets 1\ldots P$}
        \State $x^i_1 \sim k_1(\cdot)$ \Comment{Initialize particle $i$}
        \State $w^i_1 \gets w_1(x_1^i)$ \Comment{Initial weight for particle $i$}
    \EndFor
    \For{$t \gets 2\ldots T$}
        \State $W_{t-1}^{1:P} \gets w_{t-1}^{1:P} / (\sum_{i=1}^Pw_{t-1}^i )$ \Comment{Normalize weights $w_{t-1}^{1:P} = (w_{t-1}^1, \ldots, w_{t-1}^P)$}
        \For{$i \gets 1\ldots P$}
            \State $a_{t-1}^i \sim \mbox{Categorical}(W_{t-1}^{1:P})$ \Comment{Sample index of parent for particle $i$}
            \State $x^i_t \sim k_t\left(\cdot;x_{t-1}^{a_{t-1}^i}\right)$ \Comment{Sample value for new particle $i$}
            \State $w_t^i \gets w_t(x_{t-1}^{a_{t-1}^i}, x_t^i)$ \Comment{Compute weight for particle $i$}
        \EndFor
    \EndFor
    \State $I_T \sim \small\mbox{Categorical}(W_T^{1:P})$ \Comment{Sample particle index for output sample}
    \State $x \gets x_T^{I_T}$
    \State \Return $x$  \Comment{Return the output sample}
\end{algorithmic}
\end{algorithm}
The SMC marginal likelihood estimate $\widehat{p(y)}$ is computed from the weights $w_t^i$ generated during the SMC algorithm according to:
\begin{align}
\quad\quad\quad\quad
\quad\quad\quad\quad
\quad\quad\quad\quad
\widehat{p(y)}
&= \prod_{t=1}^T \frac{1}{P} \sum_{i=1}^P w_t^i
\end{align}
Note that an SMC marginal likelihood estimate can also be computed from the weights $w_t^i$ generated during the generalized conditional SMC algorithm.
Note that $\widehat{p(y)}$ a function of the SMC trace $u$.
To relate $\xi(u, x)$ to $\widehat{p(y)}$, we write the joint density of the generative inference model for SMC:
\begin{align}
q(u, x) := \left[ \prod_{i=1}^P k_1(x^i_1) \right] \left[ \prod_{t=2}^T \prod_{i=1}^P \frac{w_{t-1}^{a^i_{t-1}}}{\sum_{j=1}^P w_{t-1}^j} k_t(x^i_t; x_{t-1}^{a_{t-1}^i})\right]
\left[ \frac{w_T^{I_T}}{\sum_{j=1}^P w_T^j}\right]  \delta(x, x_T^{I_T})
\end{align}
The the canonical meta-inference sampler (Algorithm~1) for SMC takes as input a latent sample $x$ and returns a trace $u = (\mathbf{x}, \mathbf{a}, I_T)$ of Algorithm~\ref{alg:smc}, containing all particles at all time steps $\mathbf{x}$, all parent indices $\mathbf{a}$, and the final output particle index $I_T$.
The density on outputs of the meta-inference sampler is given by:
\begin{align}
r(u; x)
:= \delta(x_T^{I_T}, x) \frac{1}{P^T} \left[ \prod_{t=2}^T \ell_t(x_{t-1}^{I_{t-1}}; x_t^{I_t}) \right] \left[ \prod_{\substack{i=1\\i \ne I_1}}^P k_1(x^i_1) \right]
\left[ \prod_{t=2}^T \prod_{\substack{i=1\\i \ne I_t}}^P \frac{w_{t-1}^{a^i_{t-1}}}{\sum_{j=1}^P w_{t-1}^j} k_t(x^i_t; x_{t-1}^{a_{t-1}^i}) \right]
\end{align}
Taking the ratio $q(u,x)/r(u;x)$ and simplifying gives $p(x, y) / \widehat{p(y)}$.
Therefore, the quantity $\xi(u, x) = q(u, x) / r(u; x)$ can be computed from an SMC trace $u$, in terms of the SMC marginal likelihood estimate and the unnormalized posterior density $p(x, y)$.

\section{Proofs} \label{sec:proofs}

\begin{restatable}{theorem}{elboconsistency2}
    The estimate $\hat{D}$ produced by AIDE is an upper bound on the symmetrized KL divergence in expectation, and the expectation is nonincreasing in AIDE parameters $M_{\GOLD}$ and $M_{\TARGET}$. \end{restatable}
\begin{proof}
We consider the general case of two inference algorithms $a$ and $b$ with generative inference models $(\mathcal{U}, \mathcal{X}, q_a)$ and $(\mathcal{V}, \mathcal{X}, q_b)$, and meta-inference algorithms $(r_a, \xi_a)$ and $(r_b, \xi_b)$ with normalizing constants $Z_a$ and $Z_b$ respectively.
For example $a$ may be `target' inference algorithm and $b$ may be the `gold standard' inference algorithm.
Note that the analysis of AIDE is symmetric in $a$ and $b$.
First, we define the following quantity relating $a$ and $b$:
\begin{equation}
\mathcal{L}_{ab} := \E_{x \sim q_a(x)} \left[ \log \frac{Z_b q_b(x)}{Z_a q_a(x)} \right] = \log \frac{Z_b}{Z_a} - \kl{q_a(x)}{q_b(x)}
\end{equation}
When $Z_a = 1$ and when $b$ is a rejection sampler for the posterior $p(x|y)$, we have that $Z_b = p(x, y)$, and $\mathcal{L}_{ab}$ is the `ELBO' of inference algorithm $a$ with respect to the posterior.
We also define the quantity:
\begin{equation}
\mathcal{U}_{ab} := -\mathcal{L}_{ba}  = \log \frac{Z_b}{Z_a} + \kl{q_b(x)}{q_a(x)}
\end{equation}
Note that $\mathcal{U}_{ab} - \mathcal{L}_{ab} = \mathcal{U}_{ba} - \mathcal{L}_{ba}$ is the symmetrized KL divergence between $q_a(x)$ and $q_b(x)$.
The AIDE estimate can be understood as a difference of an estimate of $\mathcal{U}_{ab}$ and an estimate of $\mathcal{L}_{ab}$.
Specifically, we define the following estimator of $\mathcal{L}_{ab}$:
\begin{align}
\hat{\mathcal{L}}_{ab}^{N_a,M_a,M_b} &:= \frac{1}{N_a} \sum_{n=1}^{N_a} \log \left(
\frac
{\frac{1}{M_b} \sum_{k=1}^{M_b} \xi_b(v_{n,k},x_n)}
{\frac{1}{M_a} \sum_{k=1}^{M_a} \xi_a(u_{n,k},x_n)} \right)\;\;
\\
&= \frac{1}{N_a} \sum_{n=1}^{N_a} \log \left(
\frac
{\frac{1}{M_b} \sum_{k=1}^{M_b} Z_b\frac{q_b(v_{n,k},x_n)}{r_b(v_{n,k};x_n)}}
{\frac{1}{M_a} \sum_{k=1}^{M_a} Z_a \frac{q_a(u_{n,k},x_n)}{r_a(u_{n,k};x_n)}} \right)\\
&\mbox{where: }\begin{array}{l}
x_n \sim q_a(x)\; \mbox{for }n = 1\ldots N_a\\
u_{n,1} | x_n \sim q_a(u|x)\; \mbox{for }n = 1\ldots N_a\\
u_{n,k} | x_n \sim r_a(u;x)\; \mbox{for } n = 1\ldots N_a \mbox{ and } k=2\ldots M_a\\
v_{n,k} | x_n \sim r_b(v;x)\; \mbox{for } n = 1\ldots N_a \mbox{ and } k=1\ldots M_b
\end{array}\nonumber
\end{align}
\noindent We now analyze the expectation $\E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ]$ and how it depends on $M_a$ and $M_b$.
We use the notation $u_{i:j} = (u_i, \ldots, u_j)$.
First, note that:
\begin{align}
\E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ]
&= \log \frac{Z_b}{Z_a} + \E_{x \sim q_a(x)} \left[ \log \frac{q_b(x)}{q_a(x)} \right]\nonumber\\
&\quad + \E_{\begin{subarray}{l}x \sim q_a(x)\\v_{1:M_b}|x\overset{iid}{\sim} r_b(v;x)\end{subarray}} \left[ \log \frac{1}{M_b} \sum_{k=1}^{M_b} \frac{q_b(v_k|x)}{r_b(v_k;x)} \right]\nonumber\\
&\quad -\E_{\begin{subarray}{l}x \sim q_a(x)\\u_1|x \sim q_a(u|x)\\u_{2:M_a}|x \overset{iid}{\sim} r_a(u;x)\end{subarray}} \left[ \log \frac{1}{M_a} \sum_{k=1}^{M_a} \frac{q_a(u_k|x)}{r_a(u_k;x)} \right]\\
&= \mathcal{L}_{ab}
+\E_{\begin{subarray}{l}x \sim q_a(x)\\v_{1:M_b}|x\overset{iid}{\sim} r_b(v;x)\end{subarray}} \left[ \log \frac{1}{M_b} \sum_{k=1}^{M_b} \frac{q_b(v_k|x)}{r_b(v_k;x)} \right]\nonumber\\
&\quad\quad\quad-\E_{\begin{subarray}{l}x \sim q_a(x)\\u_1 \sim q_a(u|x)\\u_{2:M_a}|x \overset{iid}{\sim} r_a(u;x)\end{subarray}} \left[ \log \frac{1}{M_a} \sum_{k=1}^{M_a} \frac{q_a(u_k|x)}{r_a(u_k;x)} \right] \label{eq:elbo-unroll}
\end{align}
We define the following families of densities on $v_{1:M_b}$, indexed by $x$:
\begin{align}
    \eta_b^{M_b}(v_{1:M_b}; x) &= \frac{1}{M_b} \sum_{k=1}^{M_b} q_b(v_k | x) \prod_{\substack{\ell = 1\\\ell \ne k}}^{M_b} r_b(v_{\ell}; x)\\
    \lambda_b^{M_b}(v_{1:M_b}; x) &= \prod_{k=1}^{M_b} r_b(v_k; x)
\end{align}
and similarly for $u_{1:M_a}$:
\begin{align}
    \eta_a^{M_a}(v_{1:M_a}; x) &= \frac{1}{M_a} \sum_{k=1}^{M_a} q_a(u_k | x) \prod_{\substack{\ell = 1\\\ell \ne k}}^{M_a} r_a(u_{\ell}; x)\\
    \lambda_a^{M_a}(u_{1:M_a}; x) &= \prod_{k=1}^{M_a} r_a(u_k; x)
\end{align}

\noindent Taking the first expectation in Equation~(\ref{eq:elbo-unroll}):\\
\begin{align}
    &\E_{\begin{subarray}{l}x \sim q_a(x)\\v_{1:M_b}\overset{iid}{\sim} r_b(v;x)\end{subarray}}\left[ \log \frac{1}{M_b} \sum_{k=1}^{M_b} \frac{q_b(v_k|x)}{r_b(v_k;x)} \right]\\
    &= \E_{\begin{subarray}{l}x \sim q_a(x)\\v_{1:M_b}\overset{iid}{\sim} r_b(v;x)\end{subarray}} \left[ \log \frac{\frac{1}{M_b} \sum_{k=1}^{M_b} q_b(v_k|x) \prod_{\substack{\ell = 1\\\ell \ne k}}^{M_b} r_b(v_{\ell};x)}{\prod_{k=1}^{M_b} r_b(v_k;x)} \right]\\
        &= \E_{\begin{subarray}{l}x \sim q_a(x)\\v_{1:M_b} \sim \lambda_b^{M_b}(v_{1:M_b}; x)\end{subarray}} \left[ \log \frac{\eta_b^{M_b}(v_{1:M_b};x)}{\lambda_b^{M_b}(v_{1:M_b};x)}\right]\\
    &= - \E_{x \sim q_a(x)} \left[ \kl{\lambda_b^{M_b}(v_{1:M_b};x)}{\eta_b^{M_b}(v_{1:M_b};x)} \right] \label{eq:substitute-for-b}
\end{align}

\noindent Taking the second expectation in Equation~(\ref{eq:elbo-unroll}):
\begin{align}
    &\E_{\begin{subarray}{l}x \sim q_a(x)\\u_1 \sim q_a(u|x)\\u_{2:M_a} \overset{iid}{\sim} r_a(u;x)\end{subarray}} \left[ \log \frac{1}{M_a} \sum_{k=1}^{M_a} \frac{q_a(u_k|x)}{r_a(u_k;x)} \right]\\
    &= \E_{\begin{subarray}{l}x \sim q_a(x)\\u_1 \sim q_a(u|x)\\u_{2:M_a} \overset{iid}{\sim} r_a(u;x)\end{subarray}}\left[ \log \frac{\frac{1}{M_a} \sum_{k=1}^{M_a} q_a(u_k|x) \prod_{\substack{\ell = 1\\\ell \ne k}}^{M_a} r_a(u_{\ell};x)}{\prod_{k=1}^{M_a} r_a(u_k;x)} \right]\\
        &= \E_{\begin{subarray}{l}x \sim q_a(x)\\u_1 \sim q_a(u|x)\\u_{2:M_a} \overset{iid}{\sim} r_a(u; x)\end{subarray}} \left[ \log \frac{\eta_a^{M_a}(u_{1:M_a}; x)}{\lambda_a^{M_a}(u_{1:M_a}; x)} \right]\\
    &= \E_{\begin{subarray}{l}x \sim q_a(x)\\u_{1:M_a} \sim \eta_a^{M_a}(u_{1:M_a}; x)\end{subarray}} \left[ \log \frac{\eta_a^{M_a}(u_{1:M_a}; x)}{\lambda_a^{M_a}(u_{1:M_a}; x)} \right] \label{eq:mixture-step}\\
    &= \E_{x \sim q_a(x)} \left[ \kl{\eta_a^{M_a}(u_{1:M_a}; x)}{\lambda_a^{M_a}(u_{1:M_a}; x)} \right] \label{eq:substitute-for-a}
\end{align}
\noindent where to obtain Equation~(\ref{eq:mixture-step}) we used the fact that $\log (\eta_a^{M_a}(u_{1:M_a}; x) / \lambda_a^{M_a}(u_{1:M_a}; x))$ is invariant to permutation of its arguments $u_{1:M_a}$.
Substituting the expression given by Equation~(\ref{eq:substitute-for-b}) and the expression given by Equation~(\ref{eq:substitute-for-a}) into Equation~(\ref{eq:elbo-unroll}), we have:
\begin{align}
\E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ]
    &= \mathcal{L}_{ab} - \mathbb{E}_{x \sim q_a(x)} \left[ \kl{\lambda_b^{M_b}(v_{1:M_b};x)}{\eta_b^{M_b}(v_{1:M_b};x)} \right]\\
        &\quad\quad\quad- \mathbb{E}_{x \sim q_a(x)} \left[ \kl{\eta_a^{M_a}(u_{1:M_a}; x)}{\lambda_a^{M_a}(u_{1:M_a}; x)} \right] \label{eq:est-kl}
\end{align}
From non-negativity of KL divergence:
\begin{align}
\E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ] \le \mathcal{L}_{ab}
\end{align}
\noindent Next, we show that $\E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ]$ is nondecreasing in both $M_a$ and $M_b$.
First, we show this for $M_a$.
We introduce the notation $u_{1:k-1:k+1:M_a} := (u_1, \ldots, u_{k-1}, u_{k+1}, \ldots, M_a)$ to denote the subvector of length $M_a - 1$ obtained by removing element $k$ from vector $u_{1:M_a}$, where $u_{1:0:2:M_a} := u_{2:M_a}$ and $u_{1:M_a-1:M_a+1:M_a} := u_{1:M_a-1}$.
Note that:
\begin{equation}
    \eta_a^{M_a}(u_{1:M_a}; x) = \frac{1}{M_a} \sum_{k=1}^{M_a} \eta_a^{M_a - 1}(u_{1:k-1:k+1:M_a}; x) r_a(u_k; x)
\end{equation}
By convexity of KL divergence, we have:
\begin{align}
    &\kl{\eta_a^{M_a}(u_{1:M_a}; x)}{\lambda_a^{M_a}(u_{1:M_a}; x)}\\
    &\quad\le \frac{1}{M_a} \sum_{k=1}^{M_a} \kl{\eta_a^{M_a-1}(u_{1:k-1:k+1:M_a}; x) r_a(u_k; x)}{\lambda_a^{M_a}(u_{1:M_a}; x)}\\
    &\quad= \frac{1}{M_a} \sum_{k=1}^{M_a} \kl{\eta_a^{M_a-1}(u_{1:M_a-1}; x)}{\lambda_a^{M_a-1}(u_{1:M_a-1}; x)}\\
    &\quad= \kl{\eta_a^{M_a-1}(u_{1:M_a-1}; x)}{\lambda_a^{M_a-1}(u_{1:M_a-1}; x)}
\end{align}
A similar argument can be used to show that:
\begin{align}
&\kl{\lambda_b^{M_b}(v_{1:M_b};x)}{\eta_b^{M_b}(v_{1:M_b};x)}\\
\quad\quad&\le \kl{\lambda_b^{M_b-1}(v_{1:M_b-1};x)}{\eta_b^{M_b-1}(v_{1:M_b-1};x)}
\end{align}
Applying these inequalities to Equation~(\ref{eq:est-kl}), we have:
\begin{align}
    \mathcal{L}_{ab} \ge \E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ] &\ge \E [ \hat{\mathcal{L}}_{ab}^{1,M_a-1,M_b} ]\\
    \mathcal{L}_{ab} \ge \E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ] &\ge \E [ \hat{\mathcal{L}}_{ab}^{1,M_a,M_b-1} ]
\end{align}
To conclude the proof we apply these inequalities to the expectation of the AIDE estimate:
\begin{align}
    \hat{D}^{N_a,N_b,M_a,M_b} &= -\hat{\mathcal{L}}_{ab}^{N_a,M_a,M_b} - \hat{\mathcal{L}}_{ba}^{N_b,M_b,M_a}\\
    \E[\hat{D}^{N_a,N_b,M_a,M_b}] &=\E [ -\hat{\mathcal{L}}_{ab}^{1,M_a,M_b} ] + \E[-\hat{\mathcal{L}}_{ba}^{1,M_b,M_a}]\\
                &\ge -\mathcal{L}_{ab} - \mathcal{L}_{ba}\\
                &= \kl{q_a(x)}{q_b(x)} + \kl{q_b(x)}{q_a(x)}\\
    \E[\hat{D}^{N_a,N_b,M_a,M_b}] &\le  \E[\hat{D}^{N_a,N_b,M_a-1,M_b}]\\
    \E[\hat{D}^{N_a,N_b,M_a,M_b}] &\le  \E[\hat{D}^{N_a,N_b,M_a,M_b-1}]
\end{align}
\end{proof}

\section{Bias of AIDE for AIS and MH}
When the generic SMC algorithm (Algorithm~\ref{alg:smc}) is used with a single particle ($P = 1$), the algorithm becomes a Markov chain that samples from transition kernels $k_t$, and the canonical SMC meta-inference algorithm also becomes a Markov chain that samples from transition kernels $\ell_t$ in reverse order.
For this analysis we assume that $k_t = \ell_t$ and that $k_t$ satisfies detailed balance with respect to intermediate distribution $p_{t-1}$ for $t=2\ldots T$.
Then, the incremental weight simplifies to:
\begin{align}
w_t(x_{t-1},x_t)
&= \frac{\tilde{p}_t(x_t) k_t(x_{t-1};x_t)}{\tilde{p}_{t-1}(x_{t-1}) k_t(x_t;x_{t-1})}\\
&= \frac{\tilde{p}_t(x_t) }{\tilde{p}_{t-1}(x_{t-1})} \frac{\tilde{p}_{t-1}(x_{t-1})}{\tilde{p}_{t-1}(x_t)}\\
&= \frac{\tilde{p}_t(x_{t})}{\tilde{p}_{t-1}(x_{t})}
\end{align}
Under the limiting assumption that $k_t(x_t; x_{t-1}) = p_{t-1}(x_t)$, the approximation error of the canonical meta-inference algorithm becomes:
\begin{align}
\kl{q(u|x)}{r(u;x)} = \sum_{t=2}^T \kl{p_{t-1}(x)}{p_t(x)}
\end{align}
where $p_0$ is defined to be the initialization distribution $k_1$, and where $p_T(x) = p(x|y)$.
A similar result can be obtained for the other direction of divergence.
If the intermediate distributions are sufficiently fine-grained, then empirically this divergence converges to zero (as demonstrated in e.g. \citep{grosse2016measuring}).
However, in standard Markov chain Monte Carlo practice, without annealing, the intermediate distributions are $p_t = p_T$ for all $t > 0$.
In this case, the approximation error of meta-inference is the divergence between the initializing distribution and the posterior, which is generally large.
Better meta-inference algorithms that do not rely on the AIS assumption that the chain is near equilibrium at all times are needed in order for AIDE to be a practical tool for measuring the accuracy of standard, non-annealed Markov chain Monte Carlo.

\section{AIDE specialized for evaluating variational inference using AIS}
To make AIDE more concrete for the reader, we provide the AIDE algorithm when specialized to measure the symmetrized KL divergence between a variational approximation $q_{\theta}(x)$ and an annealed importance sampler (AIS).
For variational inference, there is no meta-inference sampler necessary because we can evaluate the variational approximation density as discussed in the main text.
The meta-inference sampler for AIS consists of running the AIS chain in reverse, starting from a latent sample.
The trace $u$ that is generated is the vector of intermediate states $\mathbf{x} = (x_1, \ldots, x_T)$ in the chain.
The AIS marginal likelihood estimate $\widehat{p(y)}$ can be computed from a trace $u$ of the AIS algorithm in terms of the weights (which are themselves deterministic functions of the trace):
\begin{equation}
\quad\quad\quad\quad
\quad\quad\quad\quad
\quad\quad\quad\quad
    \widehat{p(y)} = \frac{\tilde{p}_1(x_1)}{k_1(x_1)} \prod_{t=2}^T \frac{\tilde{p}_t(x_t)}{\tilde{p}_{t-1}(x_t)}
\end{equation}
Note that $\widehat{p(y)}$ can be computed from a trace $u$ that is generated either by a `forward' run of AIS, or a reverse run of AIS.
Algorithm~\ref{alg:ais-variational-aide} gives a concrete instantiation of AIDE (Algorithm~2) simplified for the case when the gold-standard algorithm is an AIS sampler, and the target algorithm being evaluated is a variational approximation.
We further simplify the algorithm by fixing $M_{\GOLD} = 1$, where AIS is the gold-standard.
The AIS algorithm must support two primitives: $\textproc{ais}.\textproc{forward}()$, which runs AIS forward and returns the resulting output sample $x$ and the resulting marginal likelihood estimate $\widehat{p(y)}$, and $\textproc{ais}.\textproc{reverse}(x)$, which takes as input a sample $x$ and runs the same AIS chain in reverse order, returning the resulting marginal likelihood estimate $\widehat{p(y)}$.
\begin{algorithm}
\footnotesize
\caption{AIDE specialized for measuring divergence between variational inference and AIS} \label{alg:ais-variational-aide}
\begin{algorithmic}
    \Require
        \begin{varwidth}[t]{\linewidth}
        \begin{tabular}[t]{ll}
        AIS algorithm& $\textproc{ais}.\textproc{forward}()$ and $\textproc{ais}.\textproc{reverse}(x)$\\
        Trained variational approximation &$q_{\theta}(x)$\\
        Number of AIS forward samples & $N_{\GOLD}$\\
        Number of variational samples & $N_{\TARGET}$
        \end{tabular}
        \end{varwidth}
    \Statex
    \For{$n \gets 1\ldots N_{\GOLD}$}
        \LineComment{Run AIS forward, record the marginal likelihood estimate $\widehat{p(y)}_{n}$ and the final state in chain $x_n$}
        \State $\left( \widehat{p(y)}_{n}, x_n\right) \sim \textproc{ais}.\textproc{forward}()$
    \EndFor
    \Statex
    \For{$n \gets 1\ldots N_{\TARGET}$}
        \LineComment{Generate sample $x_n'$ from the variational approximation}
        \State $x_n' \sim q_{\theta}(x)$
        \LineComment{Run AIS in reverse, starting from $x_n'$, and record resulting marginal likelihood estimate $\widehat{p(y)}^{\prime}_{n}$}
        \State $\widehat{p(y)}^{\prime}_{n} \sim \textproc{ais}.\textproc{reverse}(x_n')$
    \EndFor
    \Statex
    \LineComment{Compute AIDE estimate}
    \State $
\hat{D} \gets
            \displaystyle \frac{1}{N_{\GOLD}} \sum_{n=1}^{N_{\GOLD}}
                \log \left(
                    \frac{p(x_n, y)}{q_{\theta}(x_n)\widehat{p(y)}_{n}}
                \right)
            - \frac{1}{N_{\TARGET}} \sum_{n=1}^{N_{\TARGET}}
                \log \left(
                    \frac{p(x_n', y)}{q_{\theta}(x_n') \widehat{p(y)}_{n}^{\prime}}
                \right)$
    \State \Return $\hat{D}$
\end{algorithmic}
\end{algorithm}

\section{Evaluating SMC inference in DPMM for galaxy velocity data}
We obtained a data set of galaxy velocities based on redshift \citep{drinkwater2004large}, and randomly subsampled down to forty of the galaxies for analysis.
A histogram of the data set is shown in Figure~\ref{fig:galaxy-dataset}(a).
We consider the task of inference in a collapsed normal-inverse-gamma DPMM.
We used SMC with 100 particles, optimal (Gibbs) proposal for cluster assignments, and Metropolis-Hastings rejuvenation kernels over hyperparameters and Gibbs kernels over cluster assignments as the gold-standard inference algorithm.
Using this gold-standard, we evaluated the accuracy of SMC inference with the prior proposal, and without rejuvenation kernels using AIDE and using an alternative diagnostic based on comparing the average number of clusters in the sampling distribution relative to the average number under the gold-standard sampling distribution.
Results are shown in Figure~\ref{fig:galaxy-dataset}(b) and Figure~\ref{fig:galaxy-dataset}(c).
\begin{figure}[t]
    \centering
        \begin{minipage}[c]{0.65\textwidth}
        \begin{subfigure}[b]{1.0\textwidth}
            \centering
            \includegraphics[width=1.0\textwidth]{images/dpmm/galaxies_histogram.pdf}
            \caption{~}
        \end{subfigure}\\
        \begin{subfigure}[b]{0.49\textwidth}
            \centering
            \includegraphics[width=1.0\textwidth]{images/dpmm/galaxies_aide.pdf}
            \caption{~}
        \end{subfigure}%
        \begin{subfigure}[b]{0.49\textwidth}
            \centering
            \includegraphics[width=1.0\textwidth]{images/dpmm/galaxies_num_clusters.pdf}
            \caption{~}
        \end{subfigure}%
        \end{minipage}%
        \begin{minipage}[c]{0.35\textwidth}
        \includegraphics[width=1.0\linewidth]{images/dpmm/galaxies_legend.pdf}
        \end{minipage}
        \caption{
	(a) shows a histogram of velocities of galaxies from \citep{drinkwater2004large}.
	We model this data set using a Dirichlet process mixture, and evaluate the accuracy of SMC inference algorithms relative to a gold-standard, using AIDE and using a heuristic diagnostic based on measuring the average number of clusters in the approximating distribution and the gold-standard distribution.
(b) shows results of AIDE.
(c) shows result of the heuristic diagnostic.
Both techniques indicate that rejuvenation kernels are important for fast convergence.
Unlike the heuristic diagnostic, AIDE does not require custom design of a probe function for each model.
We envision AIDE being used in concert with heuristic diagnostics like (c).
In our experience, AIDE provides more conservative quantification of accuracy than heuristic diagnostics.
The experiment was performed on a subsampled set of 40 data points from the data set in (a).
}
    \label{fig:galaxy-dataset}
\end{figure}

redpajama_set_name: RedPajamaArXiv


There's still snow on the ground, but we're headed into another lovely, spring-like week. Here's some news to get you started.
Mar. 19, 2018, 10:36 a.m.
Hey, howdy. Did everyone enjoy last night's snow storm? I think it was Denver's way of saying, I'll teach you to discount my winters.


text:
Blake Elarbee of 'The Bachelorette' Once Arrested For DUI
Now that Rachel Lindsay is filming "The Bachelorette" 2017, the guys who are on their show are starting to come out. Blake Elarbee is one of the guys trying to win her over this season, and Radar Online is now sharing that he was arrested in the past for DUI.
His name is William Blake Elarbee, but he goes by the name Blake. Back on December 13, 2010, Blake was arrested for DUI. The police report from the Leon County Sheriff's Office shows that Blake was seen driving with a tire that was deflated and almost off the rim. The officer explained what happened.
"I detected a very strong odor of alcoholic beverage coming from the cabin of the vehicle. Elarbee's eyes were very red and watery, and his pupils were dilated. Elarbee stated he was coming from Friday's, where he was attending a work party. Elarbee said he had one beer at the party."
Elarbee entered a plea of no contest and was found guilty. Want another little bit of information on him? Blake was actually on another reality show. He was on "Ex Isle" which aired on WE.
What do you think of Blake Elarbee as a match for Rachel Lindsay? Sound off in the comments section below on your thoughts, and don't miss new episodes of "The Bachelorette" 2017 starting in May on ABC.
Tags: Blake Elarbee, Rachel Lindsay, The Bachelorette
Duggar's Share How They All Wear Grandma Duggar's Wedding Dress In Their Own Way
Review: The Naked Dog For Grain Free Dog Food Delivered To Your Door

redpajama_set_name: RedPajamaCommonCrawl


God's never been fazed with our rejections of him; he's allowed them and even used them to further his plan to restore his erring children to his side. At the beginning, Adam and Eve decided on independence and rejected the rule of God. No doubt the serpent chortled, 'Gotcha God!' Did God really say 'Oh whoops, it's all gone wrong?' Of course he didn't—he had a rescue plan in place long before the pair jumped. It's a plan which, of all things, uses the very human race that rejected him to bring it about.
On their way to the Promised Land, the Israelites had a mind-blowing national short-term memory loss—they rejected God, who had just performed some spectacular deeds to rescue them from slavery and death, and decided they wanted what all the peoples around them preferred, and got a nice shiny animal statue for their god. God simply made them all tour the wilderness till they died and the next generation entered the Promised Land.
Now after being installed in the land for a while, they decided they wanted to have a human king like all the other nations. But look at how God dealt with this one: You can have your king, and from one of these kings one day will come the righteous King of the whole world. God uses their rejection for his own ends to further his rescue.
Skipping down the centuries, when the chief priest brought Jesus before Pilate, he asked them, Shall I crucify your king? So they are forced to choose which king they will serve: 'We have no king but Caesar' they shout back. For four hundred years after returning from exile the nation had suffered oppression by foreign powers, violence and death, the enforcement of Hellenistic values, and attempts to pollute and eradicate their religion. The Jews longed for their liberty and hated their gentile rulers, yet there and then they were willing to bow to Tiberius Caesar.
It's the biggest about-turn in history, the greatest capitulation and the ultimate rejection of their God and King in the world since Adam and Eve. Yet Jesus does not defend himself. God does nothing and Jesus dies. For in that ultimate rejection and betrayal was the very way that God would reconcile his rebellious and hostile children. Three days later Jesus is resurrected. As Peter said to the rulers of Israel a few weeks later 'the stone which the builders rejected has become the chief cornerstone' (Acts 4:11).
Glory to you, our Father, who with infinite power and all-encompassing love accomplishes our reconciliation and salvation, in spite of what we do.


Get ready to eat your way through the best places Michigan has to offer!
It's the "taste of the bay" at a lighthouse. Enjoy the culinary experience with a view!
the scenes" kitchen tour. We'll dish it up for you!
nourishment, bakery tours, and the "Beggin' for Bacon" experience.
Edible flowers 101? Yes, it's true! Baskets for fruit picking, Herbs 101 and more – Orchard tours here are all part of the lore.
artwork, homemade beers, baskets for fruit picking and spiritual nourishment!
and decorating with ice carvings. What's up with that?
treat and fill your basket with the fruits of the land.


We love new techniques that take your quilting to the next level.
Whether it be embroidery, fabric painting, or other surprise techniques, Candy Apple Quilts always has ideas for taking your quilts from GREAT to EXTRAORDINARY -- and you will have a finished project that will become a family favorite for years to come.
Not a quilter? That's Ok. We have completed quilts and gift items ready to ship to your door.
Are you an embroidery enthusiast? Quilted embroidery blanks will set you apart from the crowd. Finished embroidery blanks and kits are just what you need.


The 503 service unavailable error indicates that the server hosting the website you are trying to access is unavailable.
In March 1989 a British computer scientist Tim Berners-Lee, who was working at CERN at the time, came up with the idea for what later became the World Wide Web.
HTTP is the protocol that powers the web by allowing one computer (the client) to send a request to another computer (the server) and get a response.
When you type a website address into your browser's URL bar, it sends a request to the server on which that website is hosted, and then the server responds by sending your browser data that allows it to display the website.
A status code is a three digit number provided by a server to the client together with a short description of it known as a reason phrase (it may or may not be shown to the user depending on whether the situation requires the user's attention).
"503" is a status code that indicates a particular server side error and "service unavailable" is the reason phrase that describes it.
Remembering that 4xx codes show that the problem is on your side and 5xx codes show that the problem is on the other side is useful because it allows you to immediately know whether there's anything you can do solve it.
This can save you a lot of frustration.
503 service unavailable message is shown to the user when the server they are trying to reach is unavailable. What are the most common causes of this error?
Server is down because of a spike in traffic. An unexpected and sudden increase in traffic to a website, such as an article going viral on social media or a product being featured on a major publication, can crash the server on which that website is hosted.
Server is down because of maintenance. Servers need regular maintenance in order to work properly. That means they are occasionally unavailable for this reason.
Server is down because of a DDoS (Distributed Denial of Service) attack. DDoS refers to someone deliberately overwhelming and crashing a server by bombarding it with requests.
Generally, when you encounter the 503 service error, there's not much you can do about it. Your best bet is to wait for the server to become available again.
Although 503 service unavailable error is a server-side error, sometimes the problem might actually be on the client (your) side. This can occur when the DNS configuration on your router or on your computer is preventing you from accessing the website.
This is rare, but if you suspect that to be the case, you can simply restart your internet router and the computer and then try accessing the same website again.


A sleek and contemporary Rose tinted mirror with a lacquered wooden frame. The rose mirror gives a warm inviting reflection to the interior.
The Pebbles is a 100% hand knotted wool rug that has a heavy weight shaggy pile. This rug has a spectrum of coloured knots that makes a bold statement in any room.


Hello again. I'm writing to you early because a storm is expected soon and we might lose electricity.
Greetings from Mexico! This time, I am away to learn, not to teach.
Greetings again. I'm back from Nairobi, Kenya, where I helped give a workshop through AuthorAID.
Greetings! Welcome to the AuthorAID @ INASP blog, established to assist researchers in developing countries to publish and otherwise communicate their work.


text:
Atanas Kirov (, 24 September 1946 – 27 January 2017) was a Bulgarian bantamweight weightlifter. He was a three-time World Champion, winning the bantamweight title consecutively from 1973-75. Kirov won four European titles, winning the bantamweight class in 1969 and 1973-75. He also earned a World silver medal in 1969. Kirov set two official world records in the total, with 257.5 kg in September 1973 in Havana, Cuba; and 260.0 kg in February 1974 in his hometown of Burgas. For his achievements he was made an honorary citizen of Burgas. Kirov was responsible for starting the long-line of great Bulgarian lifters, especially in the lighter classes. He was born on September 24, 1946 in Burgas. He began competing in 1962 at the Chernomorets Club with coach Ilya Babachev, later transferred to Neftochimic Burgas. Only a 156 cm tall competitor, fitting perfectly into the bantam scales, he dominated the home scene for a decade. At the 1968 Olympics he took 7th place (335 kg). Then came September 1969 and the World and European Championships in Warsaw. 14 competitors took part in the bantam. The press was won by Földi with 125 kg, Kirov was third with 107.5 kg. The snatch was dominated by Trebicki (107.5 kg), second was Földi with 105 kg and Kirov registered 102.5 kg. In the clean and jerk Földi and Trebicki got zero and dropped, Kirov lifted 137.5 kg and that was enough for the European title (347.5 kg). He was second in the world behind Mohamed Nassiri (360 kg).  In the World Championships in 1971 Kirov finished 5th (352.5 kg). At the 1972 Olympic Games in Munich he was fourth with a Bulgarian record of 362.5 kg. Between 1973 and 1975, Kirov won the World Championships and started the era of domination of Bulgarian weightlifting: 1973 (110 + 147.5 = 257.5 kg), 1974 (110 + 145 = 255 kg) and 1975 (110 + 145 = 255 kg). His last World championships was in Stuttgart 1977 when he finishеd 9th. He competed in the 1968 and 1972 Summer Olympics and placed seventh and fifth, respectively.

References

1946 births
2017 deaths
Bulgarian male weightlifters
Olympic weightlifters of Bulgaria
Weightlifters at the 1968 Summer Olympics
Weightlifters at the 1972 Summer Olympics
European Weightlifting Championships medalists
World Weightlifting Championships medalists
20th-century Bulgarian people
21st-century Bulgarian people

redpajama_set_name: RedPajamaWikipedia


Weather. The weather drops to that perfect temperature that makes living in Arizona so well worth it after a long, hot summer. I get to finally wear layers. Boots, sweaters, scarves, oh my!
Mom's birthday. I get to celebrate my wonderful mom's birthday. I'm already thinking up gift ideas to give to such a loving lady.
Pumpkin treats. Everything is spiked with that delicious taste of pumpkin. From Starbucks Pumpkin Spice Lattes to baked pumpkin seeds to pumpkin muffins.
Decorations. All my surroundings get a little bit warmer with deep, burnt colors. It feels so homey. On top of that, there are Jack-o'-lanterns, black kitties, and ghosts littering stores and homes.
Pumpkin patches. I haven't been to one since I was younger and I'm really itching to go. I love the hay rides, picking out pumpkins, and walking through haunted houses. Any takers?
Scary movies. I'm not usually a fan of these creepy flicks, but when October comes it feels mandatory to pop one in the DVD player. Getting goosebumps, covering my eyes, and letting out a few little screams is fun every once in awhile.
Halloween. This is one of my favorite holidays. I love making my own costumes. I have tons from over the years and always lend them out to my friends. I take full advantage of Halloween weekend by dressing up as much as I can.


text:
Martin County man sentenced to 30 years in prison for producing child pornography
When he's released in 30 years, Rogers will be registered as a sexual offender and supervised for the rest of his life.
Martin County man sentenced to 30 years in prison for producing child pornography When he's released in 30 years, Rogers will be registered as a sexual offender and supervised for the rest of his life. Check out this story on tcpalm.com: https://www.tcpalm.com/story/news/crime/martin-county/2018/08/08/stuart-man-sentenced-30-years-prison-after-taping-sex-acts-teen/934424002/
Sara Marino, Treasure Coast Newspapers Published 1:26 p.m. ET Aug. 8, 2018 | Updated 1:55 p.m. ET Aug. 8, 2018
Joshua Rogers met the 16-year-old online.(Photo: Courtesy of the Martin County Sheriff's Office)
MARTIN COUNTY — A Martin County man was sentenced to 30 years in federal prison Tuesday after he and another man produced and distributed pornography of a teenage girl.
On Sept. 27, Martin County Sheriff's Office detectives found a 16-year-old girl with Joshua Laine Rogers, 35, of Stuart. The girl had run away 24 days before with Rogers, according to Sarah Schall, public information officer for the U.S. Attorney, Southern District of Florida, Miami.
More: Stuart men indicted with federal child porn charges
Detectives said Rogers met the girl online and let her stay at his home, where he and Richard Lockley, 34, also of Stuart, recorded sexual acts with the girl on multiple occasions, records showed.
A forensic analysis of Rogers' smart phone revealed he had used it to record sexually explicit activity with the girl. He sent explicit videos and pictures via text from his phone to other people and posted some of them on a social media account and in the personal section of a Treasure Coast website, according to a news release from Schall.
A forensic analysis of Lockley's smart phone revealed he had used it to record sexually explicit activity with the girl.
On March 22, Rogers pled guilty to two counts of producing visuals of sexual exploitation of minors and one count of distributing visuals of sexual exploitation of minors, according to Schall.
More: Man accused of 11 counts of sexual battery on teen girl in Martin County
After completing his prison term, Rogers will have a lifetime of supervised release and will be registered as a sexual offender.
In January, Lockley plead guilty to one count of producing pictures of sexual exploitation of minors and on April 20 was sentenced to 15 years in prison and 10 years of supervised release after that, records showed.
Read or Share this story: https://www.tcpalm.com/story/news/crime/martin-county/2018/08/08/stuart-man-sentenced-30-years-prison-after-taping-sex-acts-teen/934424002/
Walmart erred in NFL player's layaway donation
70-year-old man found dead next to overturned kayak
Vero Beach business owner killed in his Illinois home
Stuart considers suing U.S. Army Corps for Lake O
1 southbound lane now open on Turnpike after tractor-trailer fire


text:
Under Melitta Baumeister's Skin
Like Scarlett Johansson's emotionless alien character in Under the Skin, Melitta Baumeister's silicone pieces resembling leather, cotton, and knit — "casted garments," she calls them — imply an authenticity that their molecular sophistication can't give them. While they are made in recognizable shapes — a tank top, a dress, a biker jacket — their rubbery tactility and harsh, clinical shades of black and white give them away. The notion of going to extremes to perfect what has already been perfected is at once mesmerizing and mind-boggling.
"I consider what is fake and what is real," says the German-born, Parsons-taught, New York-based designer — who, at the upcoming spring 2015 shows, will present only her third collection since earning her MFA from Parsons. "And I consider the loss of tangibility and the importance of experience," by which she means she's playing to an online audience as much as an in-the-flesh audience. "I find advanced technology that is able to replace or simulate reality extremely interesting. What else could be in store in tomorrow's world?"
Melitta Baumeister is available at Dover Street Market New York
08.02.2014 – Supernova / biker jacket / German / Melitta Baumeister / New York / Parsons / silicone


text:
Q: Firebase Crashlytics crashes not being reported to the Firebase Console in Flutter I tried to setup Firebase Crashlytics for the iOS part of the flutter app. I get the following messages in my console, but I do not see anything in the Firebase Dashboard.

2019-08-23 14:31:33.648772+0530 Runner[87816:826824] flutter: Flutter error caught by Crashlytics plugin:
2019-08-23 14:31:33.734996+0530 Runner[87816:826824] flutter: Error reported to Crashlytics.

1) Pubspec.yaml
Added firebase_crashlytics: ^0.1.0+2 added to the dependencies list
2) Main.dart
import 'package:firebase_crashlytics/firebase_crashlytics.dart';

void main() async {
  Crashlytics.instance.enableInDevMode = true;

  // Pass all uncaught errors from the framework to Crashlytics.
  FlutterError.onError = Crashlytics.instance.recordFlutterError;
  runApp(MyApp());
}

3) Added run script in Build Phases
Script - ${PROJECT_DIR}/Fabric.framework/run
Input Files - $(SRCROOT)/$(BUILT_PRODUCTS_DIR)/$(INFOPLIST_PATH)
4) Adding the following to crash test on click of a button in dart.
throw exception ("This is a test crash");

5) I have also tried installing the app on the simulator, stop debug, force crash and restart again.
Here are my logs:
2019-08-23 14:31:29.162211+0530 Runner[87816:826814]  - <AppMeasurement>[I-ACS036002] Analytics screen reporting is enabled. Call +[FIRAnalytics setScreenName:setScreenClass:] to set the screen name or override the default screen class name. To disable screen reporting, set the flag FirebaseScreenReportingEnabled to NO (boolean) in the Info.plist
2019-08-23 14:31:29.321000+0530 Runner[87816:826819] 6.7.0 - [Firebase/Core][I-COR000001] Configuring the default app.
2019-08-23 14:31:29.339518+0530 Runner[87816:826819] 6.7.0 - [Firebase/Core][I-COR000033] Data Collection flag is not set.
2019-08-23 14:31:29.343569+0530 Runner[87816:826779] [Crashlytics] Version 3.13.4 (143)
2019-08-23 14:31:29.344190+0530 Runner[87816:826779] [Crashlytics] Running on iOS Simulator (iPhone), 12.2.0 (18F132)
2019-08-23 14:31:29.351346+0530 Runner[87816:826779] [Crashlytics:Crash:Reports:Event] Registered Firebase Analytics event listener
2019-08-23 14:31:29.352626+0530 Runner[87816:826779] [Answers] Initialized
2019-08-23 14:31:29.374874+0530 Runner[87816:826779] [Fabric]
    Initialized with kit versions: {
    "com.twitter.answers.ios" = "1.4.1";
    "com.twitter.crashlytics.ios" = "3.13.4";
    "io.fabric.sdk.ios" = "1.10.2";
    }
2019-08-23 14:31:29.441898+0530 Runner[87816:826819] 6.7.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    acct = "com.vts.vtsMobileCloud";
    class = genp;
    gena = "com.google.iid";
    "m_Limit" = "m_LimitAll";
    "r_Attributes" = 1;
    "r_Data" = 1;
    svce = "com.google.iid.checkin";
}
2019-08-23 14:31:29.459396+0530 Runner[87816:826819] 6.7.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    atag = <636f6d2e 676f6f67 6c652e69 69642d7c 4b7c>;
    class = keys;
    "r_Ref" = 1;
    type = 42;
}
2019-08-23 14:31:29.467260+0530 Runner[87816:826819] 6.7.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    atag = <636f6d2e 676f6f67 6c652e69 69642d7c 507c>;
    class = keys;
    "r_Ref" = 1;
    type = 42;
}
2019-08-23 14:31:29.469687+0530 Runner[87816:826819] 6.7.0 - [Firebase/InstanceID][I-IID009000] No keypair info is found with tag com.google.iid-|K|
2019-08-23 14:31:29.470959+0530 Runner[87816:826819] 6.7.0 - [Firebase/InstanceID][I-IID009006] There's no legacy keypair so no need to do migration.
2019-08-23 14:31:29.473193+0530 Runner[87816:826819] 6.7.0 - [Firebase/InstanceID][I-IID023000] Info is not found in Keychain. OSStatus: -25300. Keychain query: {
    acct = "com.vts.vtsMobileCloud";
    class = genp;
    gena = "com.google.iid-tokens";
    "m_Limit" = "m_LimitAll";
    "r_Attributes" = 1;
    "r_Data" = 1;
    svce = "346036413985:*";
}
2019-08-23 14:31:29.475434+0530 Runner[87816:826819] 6.7.0 - [GULReachability][I-REA902003] Monitoring the network status
2019-08-23 14:31:29.475923+0530 Runner[87816:826831] flutter: Observatory listening on http://127.0.0.1:59771/SLPgRYC-GPQ=/
2019-08-23 14:31:29.477698+0530 Runner[87816:826819] 6.7.0 - [Firebase/Analytics][I-ACS024000] Debug mode is on
2019-08-23 14:31:29.478350+0530 Runner[87816:826819] 6.7.0 -
2019-08-23 14:31:29.478853+0530 Runner[87816:826819] 6.7.0 - [Firebase/Analytics][I-ACS023007] Analytics v.60101000 started
2019-08-23 14:31:29.480269+0530 Runner[87816:826819] 6.7.0 - [Firebase/Analytics][I-ACS023008] To disable debug logging set the following application argument: -noFIRAnalyticsDebugEnabled (see http:)
2019-08-23 14:31:29.480999+0530 Runner[87816:826819] 6.7.0 - [Firebase/Analytics][I-ACS023009] Debug logging enabled
2019-08-23 14:31:29.550095+0530 Runner[87816:826819] [Answers] Answers is disabled. Cleaning up.
2019-08-23 14:31:29.689894+0530 Runner[87816:826814] 6.7.0 - [GoogleUtilities/AppDelegateSwizzler][I-SWZ001008] Successfully created App Delegate Proxy automatically. To disable the proxy, set the flag GoogleUtilitiesAppDelegateProxyEnabled to NO (Boolean) in the Info.plist
2019-08-23 14:31:29.692544+0530 Runner[87816:826814] 6.7.0 - [GULReachability][I-REA902004] Network status has changed. Code:2, status:Connected
2019-08-23 14:31:29.694005+0530 Runner[87816:826814] 6.7.0 - [Firebase/Analytics][I-ACS031010] Tracking view controller. Class, ID: FlutterViewController, -5765211381914463904
2019-08-23 14:31:29.696012+0530 Runner[87816:826814] 6.7.0 - [Firebase/Analytics][I-ACS031013] Screen view event not logged. App is not active.
2019-08-23 14:31:29.821624+0530 Runner[87816:826814] 6.7.0 - [Firebase/Analytics][I-ACS029014] Successfully parsed a configuration. Version: 1566456169535025
2019-08-23 14:31:29.826523+0530 Runner[87816:826814] 6.7.0 - [Firebase/Analytics][I-ACS023016] Analytics is ready to receive events
2019-08-23 14:31:29.946190+0530 Runner[87816:826819] 6.7.0 - [Firebase/Analytics][I-ACS032003] iAd framework is not linked. Search Ad Attribution Reporter is disabled.
2019-08-23 14:31:29.947722+0530 Runner[87816:826814] 6.7.0 - [Firebase/Analytics][I-ACS023024] No data to upload. Upload task will not be scheduled
2019-08-23 14:31:29.948368+0530 Runner[87816:826819] 6.7.0 - [Firebase/Analytics][I-ACS023012] Analytics enabled
2019-08-23 14:31:30.414665+0530 Runner[87816:826814] [Fabric] settings downloaded successfully
2019-08-23 14:31:30.415103+0530 Runner[87816:826814] [Fabric] Settings are available for consumption
2019-08-23 14:31:30.511026+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS033003] Scheduling user engagement timer
2019-08-23 14:31:30.513147+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS002002] Engagement timer scheduled to fire in approx. (s): 3600
2019-08-23 14:31:30.513984+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS023051] Logging event: origin, name, params: auto, screen_view (_vs), {
    ga_event_origin (_o) = auto;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) = -5765211381914463904;
}
2019-08-23 14:31:30.514647+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS023073] Debug mode is enabled. Marking event as debug and real-time. Event name, parameters: screen_view (_vs), {

        ga_debug (_dbg) = 1;
        ga_event_origin (_o) = auto;
        ga_realtime (_r) = 1;
        ga_screen_class (_sc) = FlutterViewController;
        ga_screen_id (_si) = -5765211381914463904;
    }
    2019-08-23 14:31:30.703851+0530 Runner[87816:826824] flutter: addToOverlay
    2019-08-23 14:31:31.040103+0530 Runner[87816:826835] 6.7.0 - [Firebase/Analytics][I-ACS023072] Event logged. Event name, event
    params: screen_view (_vs), {
    ga_debug (_dbg) = 1;
    ga_event_origin (_o) = auto;
    ga_realtime (_r) = 1;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) = -5765211381914463904;
    }

2019-08-23 14:31:31.766716+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS002002] Measurement timer scheduled to fire in approx. (s): -0.2573471069335938
2019-08-23 14:31:31.792724+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS023028] Upload task scheduled to be executed in approx. (s): -0.2573471069335938
2019-08-23 14:31:31.796034+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS002001] Measurement timer fired
2019-08-23 14:31:31.798589+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS002003] Measurement timer canceled
2019-08-23 14:31:31.799467+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS023033] Starting data upload
2019-08-23 14:31:31.799862+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS023105] Event is not subject to real-time event count daily limit. Marking an event as real-time. Event name,
   parameters: screen_view (_vs), {
    ga_debug (_dbg) = 1;
    ga_event_origin (_o) = auto;
    ga_realtime (_r) = 1;
    ga_screen_class (_sc) = FlutterViewController;
    ga_screen_id (_si) = -5765211381914463904;
    }
2019-08-23 14:31:31.801179+0530 Runner[87816:826844] 6.7.0 - [Firebase/Analytics][I-ACS012018] Saving bundle. size (bytes): 425
2019-08-23 14:31:31.802492+0530 Runner[87816:826844] 6.7.0 - [Firebase/Analytics][I-ACS023116] Bundle added to the upload queue. BundleID, timestamp (ms): 61, 1566550890505
2019-08-23 14:31:32.035619+0530 Runner[87816:826844] 6.7.0 - [Firebase/Analytics][I-ACS023038] Uploading events. Elapsed time since last successful upload (s): 242.4702870845795
2019-08-23 14:31:32.037031+0530 Runner[87816:826844] 6.7.0 - [Firebase/Analytics][I-ACS023039] Measurement data sent to network. Timestamp (ms), data: 1566550892032, <APMPBMeasurementBatch: 0x600000252560>
2019-08-23 14:31:32.050976+0530 Runner[87816:826840] 6.7.0 - [Firebase/Analytics][I-ACS900000] Uploading data. Host: https://app-measurement.com/a
2019-08-23 14:31:32.170464+0530 Runner[87816:826844] 6.7.0 - [Firebase/Analytics][I-ACS901006] Received SSL challenge for host. Host: https://app-measurement.com/a
2019-08-23 14:31:32.262611+0530 Runner[87816:826835] 6.7.0 - [Firebase/Analytics][I-ACS023044] Successful upload. Got network response. Code, size: 204, -1
2019-08-23 14:31:32.265501+0530 Runner[87816:826821] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C2.1:2][0x7fd16a4875c0] get output frames failed, state 8196
2019-08-23 14:31:32.265647+0530 Runner[87816:826821] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C2.1:2][0x7fd16a4875c0] get output frames failed, state 8196
2019-08-23 14:31:32.266164+0530 Runner[87816:826821] TIC Read Status [2:0x0]: 1:57
2019-08-23 14:31:32.266553+0530 Runner[87816:826821] TIC Read Status [2:0x0]: 1:57
2019-08-23 14:31:32.268991+0530 Runner[87816:826835] 6.7.0 - [Firebase/Analytics][I-ACS002002] Measurement timer scheduled to fire in approx. (s): -0.7637839317321777
2019-08-23 14:31:32.269088+0530 Runner[87816:826835] 6.7.0 - [Firebase/Analytics][I-ACS023028] Upload task scheduled to be executed in approx. (s): -0.7637839317321777
2019-08-23 14:31:32.358056+0530 Runner[87816:826821] 6.7.0 - [Firebase/Analytics][I-ACS023024] No data to upload. Upload task will not be scheduled
2019-08-23 14:31:32.358229+0530 Runner[87816:826821] 6.7.0 - [Firebase/Analytics][I-ACS002003] Measurement timer canceled
2019-08-23 14:31:33.648772+0530 Runner[87816:826824] flutter: Flutter error caught by Crashlytics plugin:
2019-08-23 14:31:33.734996+0530 Runner[87816:826824] flutter: Error reported to Crashlytics.
2019-08-23 14:35:59.541939+0530 Runner[87816:830344] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C3.1:2][0x7fd16a486530] get output frames failed, state 8196
2019-08-23 14:35:59.542135+0530 Runner[87816:830344] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C3.1:2][0x7fd16a486530] get output frames failed, state 8196
2019-08-23 14:35:59.542630+0530 Runner[87816:830344] TIC Read Status [3:0x0]: 1:57
2019-08-23 14:35:59.542764+0530 Runner[87816:830344] TIC Read Status [3:0x0]: 1:57

I am unable to figure out what I am doing wrong and have been stuck with this for 2 days now. Any help or advice is much appreciated. Thanks.

A: Make sure you are not crashing in debug mode.
I read this in a GitHub issue: In android, the app doesn't report Crash type events with debug APK
After building a release version and installing it on my device, viola the first crash was reported.
flutter build apk --release && flutter install

A: When you open the Firebase Crashlytics page, it defaults to applying the 'Event type = "Crashes"' filter. Unhandled exceptions in Flutter are technically not crashes - the Android activity is still running as the exception did not occur on the native UI thread. All Flutter exceptions are therefore logged as non-fatal crashes.
Simply remove the default filter to see both fatal and non-fatal crashes.

A: The errors in flutter aren't crashes so it won't be recorded as a crash.

*

*Turn the filter to non-fatals,

*The 2 apply plugin statements in app/build.gradle should be placed below dependencies section.
Make sure of that and you're good to go!

A: I had this same issue [but on Android]. The screen in the Firebase console was stuck on "We'll be listening for your app to communicate with our servers." I got it working by doing the following:

*

*Follow the instructions in FlutterFire docs to set up Crashlytics, but change the following. In app/build.gradle add the dependencies like this:
implementation platform('com.google.firebase:firebase-bom:26.1.0')
implementation 'com.google.firebase:firebase-crashlytics'
implementation 'com.google.firebase:firebase-analytics'
implementation 'com.android.support:multidex:2.0.1'

*And this is how I initialised Crashlytics instance in my main.dart
 Function originalOnError = FlutterError.onError;
 FlutterError.onError = (FlutterErrorDetails errorDetails) async {
    await FirebaseCrashlytics.instance.recordFlutterError(errorDetails);
 originalOnError(errorDetails);
};

*Also add the following if you want to test in debug mode:
await FirebaseCrashlytics.instance.setCrashlyticsCollectionEnabled(true);

A: As a side note: There is a limit/quota for the number of crashes/error reports, that firebase accepts, that is reset after a while after it was exceeded.
I reached the quota with around 100-200 error reports for multiple apps of the same firebase project. After that, no more errors showed up on a previously untested app. Changing the network did not help. Waiting a few hours did. After the period, errors showed up again.

A: I also found out that non-fatal exceptions are only displayed in the Crashlytics dashboard when a fatal exception is thrown, since this causes a report to be sent to Crashlytics.
More in Firebase docs: https://firebase.google.com/docs/crashlytics/customize-crash-reports?authuser=1&hl=en&platform=flutter#report-caught-exceptions

redpajama_set_name: RedPajamaStackExchange


a case for god Archives - Page 2 of 7 - God Evidence • Does God Exist?
What atheists present as "science" supporting their views is actually extremely shoddy philosophy posturing or posing as science…a posing and posturing that would make Elvis impersonators proud.
Why calling theism "primitive superstition" shows primitive understanding.
Atheistic reasoning which judges belief in God to be a "primitive superstition superseded by science" fails to understand the nature and limitations of scientific knowledge.
Atheistic reasoning often suggests that "science explains things without the need for God." But such a suggestion is what is known in philosophical terms as a "category error." Science describes things in terms of natural laws, but does not explain where natural laws come from or how they are enforced.
Atheistic reasoning often cites pure chance or luck as an alternative explanation to God for such phenomena as the origin of life and the origin of our universe.
Sure—the atheist argument goes—the probability of such things occurring naturally is very low…but with enough time, and even a slight probability, what is there to prevent virtually anything from happening?! But this atheist reasoning makes some very grave oversights. First of all, bare probability and large amounts of time, alone, cannot accomplish anything, ever. Period.


Motherhood is filled with demands. Babies cry. Diapers leak. Dishes pile. As our children grow, so does the number of activities clamoring for our attention. Yet one encounter can help shape every interaction: time in the Word. In this session, we'll consider how God's presence in our lives leaves a lasting impression on our children. While the specific application for different ages may vary, every mother needs wisdom, joy, patience, kindness, and love. We find these treasures and more as we walk with God in the season of motherhood.