text:
\section{Training Details}
\subsection{Loss functions}
There are three loss functions for the bidirectional training \cite{ardizzone2018analyzing} described in the main text. Without loss of generality, let us consider a single time point which can simply be extended to multiple time points by computing these losses to each of the entire time points.
\begin{enumerate}
	\item $\mathcal{L}_\mathbf{Z} ( p(\mathbf{y},\mathbf{z}), p(\mathbf{y})p(\mathbf{z}) )$: This is a loss in the forward mapping. Specifically, given a input $\mathbf{x}$, we first forward map it to $[\hat{\mathbf{y}},\mathbf{z}] = f(\mathbf{x})$ which corresponds to $p(\hat{\mathbf{y}},\mathbf{z})$ as our network maps both $\mathbf{y}$ and $\mathbf{z}$ with a single network. Our goal is to minimize the distance between this distribution resulting from our network ($p(\hat{\mathbf{y}},\mathbf{z})$) to the ideal joint distribution $p(\mathbf{y})p(\mathbf{z})$. But since we may not exactly know $p(\mathbf{y})$ and $p(\mathbf{z})$, a kernel-based moment matching measure called Maximum Mean Discrepancy (MMD) \cite{dziugaite2015training} is used which only uses the samples without explicitly requiring $p(\mathbf{y})$ and $p(\mathbf{z})$. Specifically, for each $\mathbf{x}$ and its corresponding forward map $[\hat{\mathbf{y}},\mathbf{z}]$, we also construct its ``counterpart'' sample $[\mathbf{y}_{gt},\mathbf{z} \sim \mathbf{Z}]$ which is simply the ground truth $\mathbf{y}_{gt}$ and a random sample $\mathbf{z}$ from a standard normal $\mathbf{Z}$. In other words, we construct a set of samples representing the joint distribution $p(\mathbf{y})p(\mathbf{z})$ by empirically setting $\mathbf{y}_{gt}$ and a sample $\mathbf{z}$ from the true prior $\mathbf{Z}$ which we have been assuming. Thus, the loss is fully expressed in practice as follows:
	\begin{align}
	\mathcal{L}_\mathbf{Z} ( p(\mathbf{y},\mathbf{z}), p(\mathbf{y})p(\mathbf{z}) ) &= MMD([\hat{\mathbf{y}},\mathbf{z}]_{i=1}^N \\
	&= f(\mathbf{x}_{i=1}^N),[ (\mathbf{y}_{gt})_{i=1}^N, \mathbf{z}_{i=1}^N \sim \mathbf{Z}] )
	\end{align}
	for $N$ samples in each mini-batch. The kernel used in the MMD is an inverse multiquadratic kernel
	\[
	k(\mathbf{x},\mathbf{x}') = \frac{\alpha}{\alpha  + ||\mathbf{x} - \mathbf{x}'||_2^2}
	\]
	where we used $\alpha = \{ 0.2, 0.5, 0.8, 1.0, 1.2 \}$ for multiple scales of $\alpha$ \cite{ardizzone2018analyzing,tolstikhin2017wasserstein}.
	\item $\mathcal{L}_\mathbf{Y}(\mathbf{y}_{gt}, \hat{\mathbf{y}})$: This is another loss in the forward mapping. Similar to typical supervised loss, it penalizes the difference between the true $\mathbf{y}_{gt}$ and the predicted $\hat{\mathbf{y}}$. We used the mean squared error (MSE) function.
	\item $\mathcal{L}_\mathbf{X}(p(\mathbf{x}), p_\mathbf{X})$: This is a loss in the inverse mapping (hence, the bidirectional training together with the above losses). Intuitively, this enforces the reconstructed $\mathbf{x}_{reconst}$ with known $\mathbf{y}$ ($p(\mathbf{x})$) and random $\mathbf{z}$ to follow a likely $\mathbf{x}$ with the same $\mathbf{y}$ ($p_\mathbf{X}$). Instead of maximizing the log likelihood of $p(\mathbf{x})$ directly, this is again achieved via MMD that for a given set of $\mathbf{x}_{i=1}^N$ (and their $\mathbf{y}_{i=1}^N$), we construct a set of samples with random $\mathbf{z}$ and the same set of $\mathbf{y}_{i=1}^N$ to perform the kernel-based distance measure. We use the same kernel function (and $\alpha$'s) as $\mathcal{L}_\mathbf{Z}$.
\end{enumerate}
For all these losses, the ratios of the terms were all equal throughout the experiments

\subsection{Jacobian determinant and Temporal Context Gating}
Here, we provide further explanations to the determinant of Jacobian computation that we promised in the main text along with the adjustment for the Temporal Context Gating.
\subsubsection{Chain rule of determinants}
When our network (or any other involving coupling layers) involves multiple coupling layers, the Jacobian determinant of the entire network can be computed by computing the individual Jacobian determinants. Let us consider only one coupling layer for now. We know that the Jacobian determinant of each coupling layer can be computed as the product of the diagonal entries of the Jacobian since the Jacobian is a square triangular matrix of the following form (without temporal context gating):
\begin{equation}\label{eq:jacobian}
\small
J_\mathbf{v} = \frac{\partial \mathbf{v}}{\partial \mathbf{u}} = \begin{vmatrix} \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_2} \\
\frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_2}
\end{vmatrix}
= \begin{vmatrix}
I & 0 \\
\frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & diag(\exp s(\mathbf{u}_1))
\end{vmatrix}
\end{equation}
thus $J_\mathbf{v} = \exp(\sum_i (s(\mathbf{u}_1))_i)$ and with the temporal context gating, it is
\begin{equation}\label{eg:jacobian-tcg}
\small
J_\mathbf{v} = \begin{vmatrix} \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_2} \\
\frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_2}
\end{vmatrix}
= \begin{vmatrix}
diag(cgate(\mathbf{h}^t)) & 0 \\
\frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & diag(\exp s(\mathbf{u}_1))
\end{vmatrix}
\end{equation}
where $J_\mathbf{v} = [\prod_j cgate(\mathbf{h}^t)_j]*[\exp(\sum_i (s(\mathbf{u}_1))_i)]$.
Here, we note that the upper left block of $J_\mathbf{v}$ in Eq.~\eqref{eg:jacobian-tcg} is diagonal since each output of $cgate(\mathbf{h}^t)$ is multiplied to each element of $\mathbf{u}_1$ in an element-wise manner. This behavior is identical to how the lower right block $J_\mathbf{v}$ in Eq.~\eqref{eg:jacobian-tcg} and Eq.~\eqref{eq:jacobian} is simply $diag(\exp s(\mathbf{u}_1))$ due to the element-wise product transform applied to $\mathbf{u}_1$.

The Jacobian determinants of the subsequent coupling layers can be computed consecutively using the output of the previous coupling layer as the input to the current coupling layers. In other words, for a series of composited formulations $f = f_1 \circ f_2 \circ \cdots \circ f_N$ where each $f_i$ is a coupling layer operation, then $\det(f)$ is
\begin{align}
\det(f) &= \det(f_1 \circ f_2 \circ \cdots \circ f_N) \\
&= \det(f_1) \det(f_2) \cdots \det(f_N)\\
&= \prod_{i=1}^N \det(f_i).
\end{align}
This allows an easy computation of the full Jacobian determinant regardless of the number of coupling layer operations.

\subsection{Basic training details}
We used NVIDIA 1080ti GPU to train all the models. ADAM optimizer with $\alpha=0.9$ and $\beta=0.999$ and the initial learning rate of $0.0005$ was used.

\section{Alzheimer's Disease Dataset}
We provide additional details regarding the Alzheimer's disease dataset we used. These are largely based on Alzheimer's Disease Neuroimaging Initiative (ADNI) databse (\url{adni.loni.usc.edu}) and The Alzheimer's Disease Prediction of Longitudinal Evolution (TADPOLE) \cite{marinescu2018tadpole}.

\subsection{Dataset (ADNI and TADPOLE)}
\subsubsection{Pre-processing}
The images were pre-processed with standard ADNI pipelines. For PET images, including AV45, were co-registered, averaged across the dynamic range, standardized with respect to the orientation and voxel size (see \url{adni.loni.usc.edu/methods/pet-analysis/pre-processing} for full details). For AV45, standardized uptake value ratio (SUVR) measures for Desikan ROIs \cite{desikan2006automated} after segmented and parcellated with Freesurfer (version 5.3.0) were measured (see \cite{jagust2010alzheimer} for full details).

\subsubsection{TADPOLE Datasets}
TADPOLE consists of multiple datasets, each serving different purposes with respect to the challenge itself. For our experiments, we used D1 and D2: (i) D1 is the standard training set consisting of individuals with at least two separate visits across three phases of the ADNI study (ADNI1, ADNI GO and ADNI2) and (ii) D2 is the longitudinal prediction set which have the rollovers (i.e., subjects from D1 with further visits) for the purpose of forecasting tasks. In our setup, we simply treated the subjects in D1 and D2 without distinctions to obtain the most number of subjects with (i) 3 time points with (ii) AV45 measures for all 3 time points and (iii) covariates of interests at each time point.

\subsubsection{Covariates (Conditions) Description}
Here, we describe the covariates (conditions) we used in the experiments to characterize the disease progression from varying perspectives. Full documents are available on \url{http://adni.loni.usc.edu/methods/documents} (e.g., ADNI Procedures Manual).
\begin{enumerate}
	\item \textbf{Diagnosis}: This is the diagnosis by each visit code. There are largely three categories: Control (CN, healthy), Mild Cognitive Impairment (MCI, combining early and late MCI) and Alzheimer's Disease (AD, diseased). In TADPOLE, it is provided with the baseline diagnosis along with the per-visit diagnosis change (DX) to indicate the change from previous diagnosis to the current diagnosis. We explicitly used the diagnosis change code to assign the current status (e.g., if DX is CN$\rightarrow$MCI, we assign MCI to the current diagnosis). Note that although it might be logical to think that the disease progresses monotonically (e.g., MCI$\rightarrow$AD but not AD$\rightarrow$MCI), the diagnosis is only with respect to the corresponding visit, so such revert cases are present in the dataset, capturing the progression especially more difficult.
	\item \textbf{ADAS13}: Alzheimer's Disease Assessment Scale - Cognitive (ADAS-COG) is a set of 13 assessments for learning and memory, language production, language comprehension, constructional parxis, ideational parxis, and orientation. The values (in the current dataset) range from 0 to 85 where higher number indicates worse cognitive functions (e.g., more number of errors made). While ADAS13 is not a physical measure of AD (compared to, for instance, amyloid), this measure the level of cognitive deficit which is a well-known symptom of dementia. Note that dementia is not specific to AD; thus more AD-specific pathologies such as amyloid and tau are observed with respect to these cognitive measures.
	\item \textbf{MMSE}: Mini Mental State Exam (MMSE) consists of a series of tasks testing the cognitive impairment. The score ranges from 0 to 30 where higher score indicates better cognitive function (opposite from ADAS13). MMSE has been used for reliable diagnosis of AD.
	\item \textbf{RAVLT-I}: Rey Auditory Verbal Learning Test - Immediate (RAVLT-I) tests learning ability which has been shown to be associated with AD \cite{moradi2017rey}. The score ranges from 0 to 75 where lower score indicates lower cognition.
	\item \textbf{CDR-SB}: Clinical Dementia Rating - Sum of Boxes (CDR-SB) measures scores from six different categories such as memory, orientation, judgment and problem solving and community affairs. The score ranges from 0 to 18 (from out samples) where higher score indicates higher dementia rating. This especially prioritizes memory functional over other categories.
\end{enumerate}

\section{Experiment Details}
Here, we provide specifics of the experiments such as the feature dimensions, feature splitting and network dimensions.

\begin{figure*}[!t]
	\centering
	\includegraphics[height=130pt]{figs/desikan_top.png}
	\includegraphics[height=130pt]{figs/desikan_left.png}
	\includegraphics[height=130pt]{figs/desikan_front.png}\\
	\includegraphics[height=130pt]{figs/desikan_diag_left.png}
	\includegraphics[height=130pt]{figs/desikan_diag_right.png}
	\caption{Desikan atlas ROIs. Top row (from left): top, side and front view. Bottom row: diagonal views.}
	\label{fig:desikan}
\end{figure*}
\subsection{Moving MNIST}
\begin{enumerate}
	\item Input/output dimensions: An input $\mathbf{x}^t$ at each time point $t$ was $\mathbb{R}^{400}$ (image of size 20 by 20 was vectorized). The condition $\mathbf{y}^t$ was $\mathbb{R}^2$ (onehot vector for 2 digit case) and the latent variable $\mathbf{z}^t$ was $\mathbb{R}^8$ which was chosen by us. For a smaller sized $\mathbf{z}^t$, the densities could not be accurately captured. On the other hand, for a larger sized $\mathbf{z}^t$, the latent information could potentially ``memorize'' the input to output mapping which is also undesirable. Both input and output were zero-padded to 512 dimensions.
	\item Input partitioning: We split $\mathbf{x}$ (i.e., $\mathbf{u}$ without zero padding) into $[\mathbf{x}_1,\mathbf{x}_2]$ ($\mathbf{x}_1$ into $\mathbf{u}_1$ and $\mathbf{x}_2$ into $\mathbf{u}_2$) in a ``checkerboard'' pattern. In other words,  given an image $\mathbf{x}$, the first half $\mathbf{x}_1$ consists of the pixels that are not directly adjacent to each other (i.e., black squares in a checkerboard) and the remaining pixels (which are also not directly adjacent to each other like the white squares in a checkerboard) are assigned to $\mathbf{x}_2$. This splitting scheme preserves the overall geometric structure of the image as much as possible in a simplistic manner as did Real-NVP \cite{dinh2016density}.
	\item Network setup: We used 3 coupling blocks (2 coupling layers in each block) where the input/output dimensions (i.e., $\mathbf{u}$ and $\mathbf{v}$ dimensions) are 512 (thus, zero padding of length 112 is needed for $\mathbf{x}^t$ and 502 is needed for $[\mathbf{y}^t,\mathbf{z}^t]$). Each subnetwork $q$ then has the input/output dimensions of 256 where it first starts with a GRU (256 input, 256 output, 256 hidden) followed by 3 residual layers (fully connected layers with ReLU non-linearity, all of 256 input and 256 output).
\end{enumerate}

\subsection{Neuroimaging experiments}
\begin{enumerate}
	\item Input/output dimensions: An input $\mathbf{x}^t$ at each time point $t$ is $\mathbb{R}^{82}$ for 82 brain regions. The condition $\mathbf{y}^t$ is $\mathbb{R}^3$ for diagnosis (onehot-vector over three possible diagnosis categories) and $\mathbb{R}^1$ for other continuous covariates. The latent variable $\mathbf{z}^t$ was $\mathbb{R}^4$. Both input and output were zero-padded to 150 dimensions.
	\item Input partitioning: We simply split the input $\mathbf{x}^t$ by assigning the left hemisphere features to $\mathbf{x}_1^t$ and the right hemisphere features to $\mathbf{x}_2^t$ (there are exactly 41 left and 41 right hemisphere features). This was the most logical setup since the brain regions are symmetrical.
	\item Network setup: The setup is exactly the same as the Moving MNIST setup except for the input/output zero-padded dimensions of 150.
\end{enumerate}

\section{Neuroimaging Experiment Result Details}

\subsection{Desikan ROIs}
Fig.~\ref{fig:desikan} is the enlarged version of the Fig.~6 in the main text (bottom two with diagonal views) showing the Desikan ROIs. The colors distinguish different ROIs (not reflecting any measures).
Note that the orange regions (cerebellums) are not shown in the AV45 figures since AV45 are not measured in those regions (standard procedure for treating cerebellums in other modalities as well).

\subsection{Significant ROIs}
Below, we show the list of the significant ROIs presented in Table 1 of the main text. These are the predefined names of the Desikan atlas \cite{desikan2006automated}. Some naming conventions are as follows: CTX: cortex, LH: left hemisphere, RH: right hemisphere. Significance threshold of $\alpha = 0.01$ with the Bonferroni multiple testing correction ($\alpha / 82$) was used for all the setups to control for family-wise error rate.

\subsubsection{AV45 vs. Diagnosis (CN$\rightarrow$CN$\rightarrow$CN vs. CN$\rightarrow$MCI$\rightarrow$AD)}
\begin{enumerate}
	\setlength\itemsep{0.01em}
	\item CTX$\_$LH$\_$LINGUAL
	\item CTX$\_$LH$\_$PARAHIPPOCAMPAL
	\item CTX$\_$LH$\_$PERICALCARINE
	\item LEFT$\_$AMYGDALA
	\item LEFT$\_$CAUDATE
	\item LEFT$\_$HIPPOCAMPUS
	\item LEFT$\_$PALLIDUM
	\item LEFT$\_$PUTAMEN
	\item LEFT$\_$THALAMUS$\_$PROPER
	\item LEFT$\_$VENTRALDC
	\item CTX$\_$RH$\_$ENTORHINAL
	\item CTX$\_$RH$\_$PARAHIPPOCAMPAL
\end{enumerate}

\subsubsection{AV45 vs. ADAS13 (10$\rightarrow$10$\rightarrow$10 vs. 10$\rightarrow$20$\rightarrow$30)}
\begin{enumerate}
	\setlength\itemsep{0.01em}
	\item CTX$\_$LH$\_$FUSIFORM
	\item CTX$\_$LH$\_$LATERALOCCIPITAL
	\item CTX$\_$LH$\_$PERICALCARINE
	\item LEFT$\_$HIPPOCAMPUS
	\item LEFT$\_$PALLIDUM
	\item LEFT$\_$THALAMUS$\_$PROPER
	\item LEFT$\_$VENTRALDC
	\item CTX$\_$RH$\_$CUNEUS
	\item CTX$\_$RH$\_$FUSIFORM
	\item CTX$\_$RH$\_$LATERALOCCIPITAL
	\item CTX$\_$RH$\_$LATERALORBITOFRONTAL
	\item CTX$\_$RH$\_$LINGUAL
	\item CTX$\_$RH$\_$PERICALCARINE
	\item CTX$\_$RH$\_$SUPERIORTEMPORAL
\end{enumerate}

\subsubsection{AV45 vs. MMSE (30$\rightarrow$30$\rightarrow$30 vs. 30$\rightarrow$26$\rightarrow$22)}
\begin{enumerate}
	\setlength\itemsep{0.01em}
	\item CTX$\_$LH$\_$ISTHMUSCINGULATE
	\item CTX$\_$RH$\_$FRONTALPOLE
\end{enumerate}

\subsubsection{AV45 vs. RAVLT-I (70$\rightarrow$70$\rightarrow$70 vs. 70$\rightarrow$50$\rightarrow$30)}
\begin{enumerate}
	\setlength\itemsep{0.01em}
	\item CTX$\_$LH$\_$ISTHMUSCINGULATE
	\item CTX$\_$LH$\_$PARACENTRAL
\end{enumerate}

\subsubsection{AV45 vs. CDR-SB (0$\rightarrow$0$\rightarrow$0 vs. 0$\rightarrow$5$\rightarrow$10)}
\begin{enumerate}
	\setlength\itemsep{0.01em}
	\item CTX$\_$LH$\_$PERICALCARINE
	\item LEFT$\_$HIPPOCAMPUS
	\item LEFT$\_$PALLIDUM
	\item LEFT$\_$THALAMUS$\_$PROPER
	\item LEFT$\_$VENTRALDC
	\item CTX$\_$RH$\_$CUNEUS
	\item CTX$\_$RH$\_$LATERALOCCIPITAL
	\item CTX$\_$RH$\_$LINGUAL
	\item CTX$\_$RH$\_$PERICALCARINE
\end{enumerate}

\section{Conclusion}
\vspace{-5pt}
Motivated by various experimental setups in recent computer vision and neuroimaging studies
that require large-scale longitudinal/temporal sequence of data analysis,
we study the problem of
generative models using neural networks that accounts for
progressive behavior of longitudinal data sequences.
By developing a novel architecture of an invertible neural network
that incorporates recurrent subnetworks and temporal context gating
to pass information within a sequence generation,
we enable a neural network to ``learn'' the conditional distribution of training data in a latent space
and generate a sequence of samples that demonstrate progressive behavior according to the given conditions
such as different levels of diagnosis labels (e.g., healthy to MCI to AD).
We demonstrate exhaustive experimental results with various experimental settings using two datasets
that validate such longitudinal progress in sequential image generation and AD pathology.
Especially in neuroimaging applications which often suffer from small sample sizes,
our results show promising evidence
that our model can provide sufficient number of generated samples
to obtain statistically robust results. The code will be publicly available on \url{https://github.com/shwang54}.

\section{Experiments}
To validate our framework in both qualitative and quantitative manners,
we demonstrate two sets of experiments (one from a image sequence dataset and the other from a neuroimaging study) that are carried by successfully generating conditional sequences of data
and estimating sequential densities on two separate datasets.

\begin{figure}[!t]
	\vspace{-15pt}
	\centering
	\raisebox{2\height}{Train: \hspace{4.9pt}} \includegraphics[width=0.85\columnwidth]{figs/mnist/0/training_sample_0.png} \\
	\raisebox{2\height}{cINN: \hspace{2.3pt}} \includegraphics[width=0.85\columnwidth]{figs/mnist/baseline/baseline_0_to_0.png} \\
	\raisebox{2\height}{Ours: \hspace{6.1pt}} \includegraphics[width=0.85\columnwidth]{figs/mnist/0/ours_0_to_0.png}
	\vspace{-20pt}
	\caption{Top: A training sequence example. Middle: A generated 0-to-0 sequence example using cINN. Bottom: A generated 0-to-0 sequence example using our model. ([digit label]: estimated density)}
	\label{fig:mnist1}

\end{figure}

\subsection{Conditional Moving MNIST Generation}
We first test our model on a controlled moving MNIST dataset \cite{srivastava2015unsupervised} that consists of sequences of images showing a hand-written digit from 0 to 9 starting moving from left to right and oscillating its trajectories once hitting the boundary.
This experiment qualitatively shows that a sequence of images generated with specific conditions (i.e., image labels) look smooth between sequences.
Here, we specifically construct $\sim$13K controlled sequences of length $T=6$ where each frame of a sequence is an image of size 20 by 20  (vectorized as $\mathbf{x}^t \in \mathbb{R}^{400}$) and a one-hot vector $\mathbf{y}^t \in \mathbb{R}^2$ labeling the digit at $t$ indicating two different digit values (i.e., $[1,0]$ and $[0,1]$ respectively).

Our model consists of three coupling layers as shown in Fig.~\ref{fig:network}.
An input is split into two halves $\mathbf{u}_1$ and $\mathbf{u}_2$,
and each layer performs two successive transformations on each half once.
In each transformation,
one half of the input are passed through temporal contextual gating as defined in Eq.~\eqref{eq:cgate}
and the other half through transformation blocks that contain one GRU cell and three layers of residual fully connected networks with ReLU non-linearity.

Models were trained on 6 time points, but since the model is recurrent, further time points can be generated.
The labels for each sequence of images were ``identical'', i.e., $\mathbf{y}_t = [1,0]$ for $t=1,\dots,6$ for training.
An example of training image sequences of 0 is given at the top panel of Fig.~\ref{fig:mnist1}.

The middle and bottom panels in Fig.~\ref{fig:mnist1} show a primitive comparison of
images sequences generated using cINN and our method.
While the sequences from cINN are independent from each other, i.e., sampled independently under the same condition,
the generated images using \textit{CRow{}} show smooth and natural ``progress'' across the sequence.

\begin{figure}[t]
	\centering

	\raisebox{2\height}{Ours: \hspace{2pt}} \includegraphics[width=0.87\columnwidth]{figs/mnist/0to1/ours_1_to_0.png}
	\raisebox{2\height}{cINN:} \hspace{1pt} \includegraphics[width=0.87\columnwidth]{figs/mnist/baseline/baseline_1_to_0.png}\hspace{-1pt}
	\vspace{-18pt}
	\caption{Examples of generated sequences using flipped observations. The number at the top of images show likelihood of the image belonging to the class.
		Top: 1-to-0 using our model. Bottom: 1-to-0 using cINN.
		Images from our model shows smooth transition while image using cINN show drastic change in the sequence. }
	\label{fig:good_2}
	\vspace{-15pt}
\end{figure}

In the next step, we changed the condition (i.e., image label) during a sequence generation (e.g., 0-to-1 or 5-to-3)
to visually check if the changes between the sequences look natural.
Note that the model was originally trained to learn the sequence of images with the same number.
Despite this setting was unseen during the training phase, we expected that the image sequence generated using our model
exhibit ``smooth'' progress of changes.
One demonstrative result is shown in Fig.~\ref{fig:good_2} where we compare the generated sequence of images with
condition (i.e., label) changes from 1 to 0.
Images using our model at the bottom of Fig.~\ref{fig:good_2} show gradual transition of the number in the middle of the sequence,
while the images generated using cINN does not show such behavior.
Furthermore, our model quantifies its output confidence in the form of density (i.e., likelihood) shown at the top of each generated images in Fig.~\ref{fig:good_2}.
Not only our model adjust generation based on inputs,
it also outputs relatively low density at first when encountering the change as such patterns were not observed during the training,
i.e., the likelihood decreases when condition changes and then increase as the sequences goes.
This means that our model not only shows the conditional generation ability,
it also estimates outputs' relative density given the training data seen.
Different from other generative models,
it allows conditional generation on sequential data while maintaining exact and efficient density estimation.
More figures to visualize qualitative results are shown in Fig.~\ref{fig:good_3} that shows natural transition from one number to the other.

\begin{figure}[!t]
	\centering
	\raisebox{2\height}{5-to-3:} \includegraphics[width=0.85\columnwidth]{figs/mnist/3to5/3_vs_5_good_5to3.png}
	\raisebox{2\height}{3-to-5:} \includegraphics[width=0.85\columnwidth]{figs/mnist/3to5/3_vs_5_good_3to5.png}
	\raisebox{2\height}{9-to-5:} \includegraphics[width=0.85\columnwidth]{figs/mnist/5to9/5_vs_9_good_9to5.png}
	\raisebox{2\height}{5-to-9:} \hspace{1.5pt}\includegraphics[width=0.85\columnwidth]{figs/mnist/5to9/5_vs_9_good_5to9.png}
	\vspace{-5pt}
	\caption{Examples of generated sequences with condition changes using our model.
		Top: 3-to-5,

		Bottom: 5-to-9.}
	\vspace{-5pt}
	\label{fig:good_3}

\end{figure}

\begin{table*}[t]
	\centering
	\setlength{\tabcolsep}{7pt}
	\renewcommand{\arraystretch}{1.2}

	\begin{tabular}{l|ccccc}
		&  Diagnosis            & ADAS13       & MMSE         & RAVLT-I        & CDR-SB        \\ \hline \hline
		Control     & CN$\rightarrow$CN$\rightarrow$CN  & 10$\rightarrow$10$\rightarrow$10 & 30$\rightarrow$30$\rightarrow$30 & 70$\rightarrow$70$\rightarrow$70 & 0$\rightarrow$0$\rightarrow$0  \\
		Progression           & CN$\rightarrow$MCI$\rightarrow$AD & 10$\rightarrow$20$\rightarrow$30 & 30$\rightarrow$26$\rightarrow$22 & 70$\rightarrow$50$\rightarrow$30 & 0$\rightarrow$5$\rightarrow$10 \\ \hline
		cINN    ($N_1$=100 / $N_2$= 100)      & 4             & 2            & 0            & 0            & 0          \\
		Ours    ($N_1$=100 / $N_2$= 100)     & 11            & 12           & \textbf{2}   & \textbf{2}   & 7          \\
		Ours + TCG ($N_1$=100 / $N_2$= 100)  & \textbf{12}   & \textbf{14}  & \textbf{2}   & \textbf{2}   & \textbf{9} \\ \hline \hline
		Control &            CN$\rightarrow$CN$\rightarrow$CN  & 10$\rightarrow$10$\rightarrow$10 & 30$\rightarrow$30$\rightarrow$30 & 70$\rightarrow$70$\rightarrow$70 & 0$\rightarrow$0$\rightarrow$0  \\
		Early-progression            & CN$\rightarrow$MCI$\rightarrow$MCI & 10$\rightarrow$13$\rightarrow$16 & 30$\rightarrow$28$\rightarrow$26 & 70$\rightarrow$60$\rightarrow$50 & 0$\rightarrow$2$\rightarrow$4  \\ \hline
		cINN  ($N_1$=150 / $N_2$= 150)        &  0             & 2            & 0            & 0            & 0          \\
		Ours  ($N_1$=150 / $N_2$= 150)         & 2             & 4            & \textbf{4}   & \textbf{1}   & 0          \\
		Ours + TCG ($N_1$=150 / $N_2$= 150)    & \textbf{4}    & \textbf{5}   & \textbf{4}   & \textbf{1}   & \textbf{1}\\

		\hline
	\end{tabular}
	\vspace{-5pt}
	\caption{Number of ROIs identified by statistical group analysis using the generated measures with respect to various covariates associated with AD.
		Each column represents sequences of disease progression represented by diagnosis or test scores.
		\textit{CRow{}} considers the progression sequences while cINN generates cross-sectional data in different conditions.
		In all cases, using \textit{CRow{}} with TCG yields the most number of statistically significant ROIs.

	}
	\label{table:neuroimaging}
\end{table*}
\subsection{Longitudinal Neuroimaging Analysis}

The goal of this neuroimaging experiment is to validate if our conditionally generated samples actually exhibit statistically robust and clinically sound characteristics
when trained with a longitudinal AD brain imaging dataset.
After training, we generate {\em sufficient} number of longitudinal brain imaging measures (i.e., $\{\mathbf{x}^t\}$) conditioned on various covariates (i.e., labels $\{\mathbf{y}^t\}$) associated with AD progression.
These data in sequence should show ``progress'' of pathology consistent with the covariates.
Then, with the generated sequences conditioned on two chosen groups (e.g., healthy group vs. disease progressing group),
we perform a statistical analysis to detect disease related features of the data measurements.
In the end, we expect that regions of interests (ROIs) identified by the statistical group analysis are consistent with other AD literature
with statistically stronger signal (i.e., lower $p$-value) than results using the original training data.

\textbf{Dataset.} The Alzheimer's Disease Neuroimaging Initiative (ADNI) database (\url{adni.loni.usc.edu}) is one of the largest and still growing neuroimaging databases.
Originated from ADNI, we use a longitudinal neuroimaging dataset called The Alzheimer's Disease Prediction of Longitudinal Evolution (TADPOLE) \cite{marinescu2018tadpole}.
We used data from $N$$=$$276$ participants with $T=3$ time points.

\textbf{Input.} For the longitudinal brain imaging sequence $\{\mathbf{x}^t\}$, we chose Florbetapir (AV45) Positron Emission Tomography (PET) scan measuring the level of amyloid-beta deposited in  brain which has been a known type of pathology associated with Alzheimer's disease \cite{wong2010vivo,joshi2012performance}.
The AV45 images were registered to a common brain template (MNI152) to derive the gray matter regions of interests (82 Desikan atlas ROIs \cite{desikan2006automated}, see Fig.~\ref{fig:desikan-atlas}).
Thus, each ROI entry of $\mathbf{x}^t \in \mathbb{R}^{82}$ holds an average Standard Uptake Value Ratio (SUVR) measure of AV45 where high AV45 implies more amyloid pathology.

\textbf{Condition.} For the corresponding labels $\{\mathbf{y}^t\}$ for longitudinal conditions, we chose five covariates with known associations to AD progression (parentheses for normal to impaired range): (1) \textit{Diagnosis}: Normal/Control (CN), Mild Cognitive Impairment (MCI) and Alzheimer's Disease (AD). (2) \textit{ADAS13}: Alzheimer's Disease Assessment Scale (0 to 85). (3) \textit{MMSE:} Mini Mental State Exam (0 to 30). (4) \textit{RAVLT-I:} Rey Auditory Verbal Learning Test - Immediate (normal (0 to 75). (5) \textit{CDR:} Clinical Dementia Rating (0 to 18).
These assessments impose disease \textit{progression} during the generation step. See supplement and \cite{marinescu2018tadpole} for details.

\begin{figure}[!t]
	\centering

	\includegraphics[height=70pt]{figs/neuroimaging/desikan_top.png}
	\hspace{10pt}
	\includegraphics[height=70pt]{figs/neuroimaging/desikan_left.png}
	\hspace{10pt}
	\includegraphics[height=70pt]{figs/neuroimaging/desikan_front.png}
	\caption{Left to right: Top, front and side view of Desikan brain atlas. Different ROIs have different colors.}
	\label{fig:desikan-atlas}
	\vspace{-10pt}
\end{figure}

\textbf{Analysis.}
We performed a statistical group analysis on each condition $\{\mathbf{y}^t\}$ independently with the following pipeline: \textit{(1) Training:} First, we trained our model (same subnetworks as the Moving MNIST experiment model) using
SUVR $\mathbf{x}^t \in \mathbb{R}^{82}$ for 82 ROIs with $\{\mathbf{y}^t\}$ as the `labels'. \textit{(2) Conditional longitudinal sample generation:} Then, we generate longitudinal samples $\{\ \hat{\mathbf{x}}^t \}$ conditioned on two distinct longitudinal conditions: Control (Healthy across all sequences) versus Progress (condition gets worse).

For each condition (e.g., Diagnosis), we generate $N_1$ samples of Control (e.g., $\{\hat{\mathbf{x}_1}^t \}$ conditioned on $\{\mathbf{y}_1^t \} =$ CN$\rightarrow$CN$\rightarrow$CN) and $N_2$ samples of Progress ($\{\hat{\mathbf{x}_2}^t \}$ conditioned on $\{\mathbf{y}_2^t \} =$ CN$\rightarrow$MCI$\rightarrow$AD).
Then, we perform a two sample t-test at $t=3$ between each entry of $\{\hat{\mathbf{x}_1}^3$ and $\{\hat{\mathbf{x}_2}^3$ and derive its multiple testing corrected $p$-value.

\begin{figure*}[!t]

	\centering

	\begin{minipage}{230pt}\centering
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t1_top_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t1_side_lh_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t1_front_scaled.png} \hspace{3pt} \\ \vspace{2pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t2_top_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t2_side_lh_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t2_front_scaled.png} \hspace{3pt} \\ \vspace{2pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t3_top_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t3_side_lh_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_gt/av45_gt_dx_t3_front_scaled.png} \hspace{3pt}
	\end{minipage}\hspace{10pt}

	\begin{minipage}{250pt}\centering
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t1_top_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t1_side_lh_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t1_front_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/colorbar.png} \\ \vspace{2pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t2_top_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t2_side_lh_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t2_front_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/colorbar.png} \\ \vspace{2pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t3_top_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t3_side_lh_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/av45_ours/av45_dx_t3_front_scaled.png} \hspace{3pt}
		\includegraphics[height=70pt]{figs/colorbar.png}
	\end{minipage}

	\caption{Training data vs. generated samples comparison for CN (top)$\rightarrow$MCI (middle)$\rightarrow$AD (bottom). \textbf{Left three columns:} The average of the samples CN$\rightarrow$MCI$\rightarrow$AD trajectory in the dataset. \textbf{Right three columns:} The average of the 100 generated sequences conditioned on CN$\rightarrow$MCI$\rightarrow$AD. Red and blue indicate high and low AV45 respectively, from top to bottom, ROIs are expected to turn more red (i.e., disease progression). We observe that the generated samples (right 3 columns) show magnitudes and sequential patterns similar to those of the real samples from the training data.}
	\label{fig:trajectory}
\end{figure*}

\begin{figure}[!t]
	\centering
	\includegraphics[height=62pt]{figs/av45_pval/av45_dx_t3_top_scaled_pval.png} \hspace{3pt}
	\includegraphics[height=62pt]{figs/av45_pval/av45_dx_t3_side_lh_scaled_pval.png} \hspace{3pt}
	\includegraphics[height=62pt]{figs/av45_pval/av45_dx_t3_front_scaled_pval.png} \hspace{3pt}
	\includegraphics[height=62pt]{figs/colorbar_pval.png} \hspace{3pt}
	\caption{12 ROIs found to be significantly different between two Diagnosis groups ( CN$\rightarrow$CN$\rightarrow$CN vs. CN$\rightarrow$MCI$\rightarrow$AD) at $t=3$ using our model under `Diagnosis' in Table~\ref{table:neuroimaging}. The colors denote the -log $p$-value. Several AD-related ROIs such as hippocampus, putamen, caudate and amygdala are included.}
	\label{fig:pvalue}
	\vspace{-15pt}
\end{figure}

\textbf{Result.}
\textit{Control vs. Progression (Top row block of Table~\ref{table:neuroimaging}):} We set longitudinal conditions for each covariate based on its associated to healthy progression (e.g., low ADAS13 throughout) and disease progression (e.g., high ADAS13 related to eventual AD onset).
We generated $N_1 = 100$ and $N_2=100$ samples for each group respectively.
Then, we performed the above statistical group difference analysis under 4 setups: (1) Raw training data, (2) cINN \cite{ardizzone2018analyzing}, (3) Our model, and (4) Our model + TCG.
With out sampling, the sample size of the desirable longitudinal conditions was extremely small for all setups, so no statistical significances were found.
With cINN, we occasionally found few significant ROIs, but the non-sequential samples with only $t=3$ could not generate realistic samples.
With CRow{} we consistently found significant ROIs
Further, CRow{} with the temporal context gating (TCG) detected the most number of ROIs which we visualize in Fig.~\ref{fig:pvalue}.

\textit{Control vs. Early-progression (Bottom row block of Table~\ref{table:neuroimaging}):}
We setup a more challenging chase where we generate samples which resemble subjects that show slower progression of the disease (i.e., lower rate of covariate change over time).
Such case is especially important in AD when early detection leads to effective preventions.
By following the same statistical analyses set as the Control vs. Progression setup, we first sampled $N_1=100$ and $N_2=100$ samples, but no significant ROIs were found in any of the models.
To improve the sensitivity, we generated $N_1=150$ and $N_2=150$ samples in all models and found several significant ROIs only with CRow{} (bottom rows of Table~\ref{table:neuroimaging}).

\textit{Generation assessment:} In Fig.~\ref{fig:trajectory}, we see the generated samples (right 3 columns) through time (top to bottom) in three views of the ROIs and compare them to the real training samples (left 3 columns).
We see that the generated samples have similar AV45 loads through the ROIs, and more importantly, the progression pattern across time (i.e., ROIs turning more red indicating amyloid accumulation) follows that of the real sequence as well.
Also, throughout the analyses setups, the significant ROIs often involved AD-related regions acknowledged in the neuroscience field, this implies that the generated longitudinal sequences consistently follow the underlying distribution of the real data which we may not have been able to make use of otherwise.

\section{Introduction}

Consider a dataset of {\em longitudinal or temporal sequences} of data samples $\{\mathbf{x}^t\}_{i=1}^N$ where each sample $\mathbf{x}_i$ comes with \textit{sequential covariates} $\{\mathbf{y}^t\}_{i=1}^N$, one
for each time point $t$.
In other words, we assume that for each sequential sample $i$, $\mathbf{x}^1_i,\cdots,\mathbf{x}^T_i=\{\mathbf{x}^t\}_i$,
the sequential covariates $\mathbf{y}^1_i,\cdots,\mathbf{y}^T_i=\{\mathbf{y}^t\}_i$ provide some pertinent auxiliary information associated with that sequential sample.
For example,
in a neuroimaging study, if the sequential samples correspond to several longitudinal image scans of a participant over multiple years, the sequential covariate associated with each time point may be an assessment of disease severity or some other clinical measurement.
If the sequential samples denote written musical notes of a rhythm (e.g., tabla or drum beats), the sequential labels may specify the speed or frequency of the beats in a musical arrangement.
If the sequential data corresponds to heart rate sensors when a participant is
watching a video, the sequential covariate may indicate the presence
of violence in the corresponding video segment.

Our high level goal is to design conditional generative models for such sequential data. In particular, we want a model
which provides us a type of flexibility that is highly desirable in this setting.
For instance, for a sample drawn from
the distribution after the generative model has been estimated,
we should be able to ``adjust" the sequential covariates, say at a time point $t$, dynamically to influence the expected future predictions after $t$ for {\em that} sample. It makes sense that
for a heart rate sequence, the appropriate sub-sequence should be
influenced by {\em when} the ``violence" stimulus was
introduced {\em as well as} the default heart rate pattern of the specific
sample (participant) \cite{akselrod1981power}.
Notice that
when $T=1$, this construction is similar to conditional generative models where the ``covariate" or condition $\mathbf{y}$ may simply denote an attribute that we may want to adjust for a sample: for example, increase the smile or age attribute for a face image sampled from the distribution as in \cite{kingma2018glow}.

We want our formulation to provide a modified set of $\mathbf{x}^t$s adaptively, if we adjust one or more sequential covariates $\mathbf{y}^t$s for that sample.
If we know some important clinical information at some point during the study (say, at $t=5$), this information should influence the future generation $\mathbf{x}^{t>5}$ conditioned both on this sequential covariate or event $\mathbf{y}^5$ as well as the past sequence of this sample $\mathbf{x}^{t<5}$.
This will require \textit{conditioning} on the corresponding sequential covariates at \textit{each} time point $t$ by accurately capturing the posterior distribution $p(\mathbf{x}^t | \mathbf{y}^t)$.
This type of \textit{conditional sequential generation}, in effect, requires a generative model for sequential data which can dynamically incorporate time-specific sequential covariates $\mathbf{y}^t$ of interest to adaptively modify sequences.

The above
setup models a number of applications in
in medical imaging, computer vision and engineering,
that may need generation of frame sequences conditioned on frame-level covariates.
In neuroimaging, many longitudinal studies focus on identifying disease trajectories \cite{alexander2002longitudinal,baddeley1991decline,landin2017disease}: for example, at what point in the future will the brain or specific regions in the brain exceed a threshold for brain atrophy?
The future trend is invariably a function of
clinical measurements that a participant provides at each visit {\em as well as} the past trend of the subject.
From a methodological standpoint, constructing a sequential generative model may appear feasible by appropriately augmenting the generation process using existing generative models.
For example, it seems that
one could simply concatenate the sequential measurements $\{\mathbf{x}^t\}$ as a single input for existing non-sequential conditional generative models
such as conditional GANs \cite{mirza2014conditional,isola2017image} and conditional variational autoencoders \cite{sohn2015learning,abbasnejad2017infinite}.

We find that for our application, an attractive alternative to  discriminator-generator based GANs,
is a family of neural networks called normalizing flow \cite{rippel2013high,rezende2015variational,dinh2016density,dinh2014nice}
which involve \textit{invertible networks} (i.e., reconstruct input from its output). What is particularly
relevant is that such formulations work well for
{\em conditionally} generating diverse samples with controllable degrees of freedom \cite{ardizzone2018analyzing} -- with an explicit mechanism to adjust the conditioning
variable (or covariate).
But the reader will notice that while these models, in principle, can be used to approximate the posterior probability given an input of any dimensions,
concatenating a series of sequential
inputs quickly blows up the size for these highly expressive models and quickly renders them impractical to run, even on high end GPU compute clusters.
Even if we optimistically assume computational feasibility, variable length sequences cannot easily be adapted to these innately non-sequential generative models, especially for those that extend beyond the training sequence length.
Also, data generated in this manner involve simply ``concatenated" sequential data and do not take into account the innate temporal relationships among the sequences, fundamental in the success of recurrent models.
These are the core issues we study here.

Given various potential downstream applications and the issues identified above with conditional sequential generation problem,
we seek a model which
(i) efficiently generates high dimensional sequence samples of variable lengths
(ii) with dynamic time-specific conditions reflecting upstream observations
(iii) with fast posterior probability estimation.
We tackle the foregoing issues by introducing an invertible recurrent neural network, {\bf CRow}, that includes {\bf recurrent sub-network} and {\bf temporal context gating}. These
modifications are critical in the following sense.
{\em Invertibility} lets us precisely estimate the distribution of $p(\mathbf{x}^t | \mathbf{y}^t)$ in a latent space.
Introducing {\em recurrent subnetworks} and {\em temporal context gating} enables
obtaining cues from previous time points $\mathbf{x}^{<t}$ to generate temporally sensible subsequent time points $\mathbf{x}^{\ge t}$.
Specifically, our {\bf contributions} are: \begin{inparaenum}[\bfseries (A)]
\item Our model generates conditional sequential samples $\{\mathbf{x}^t\}$ given sequential covariates $\{\mathbf{y}^t\}$ for $t=1,\dots,T$ time points where $T$ can be arbitrarily long. Specifically, we allow this by posing the task as a \textit{conditional sequence inverse problem} based on a conditional invertible neural network \cite{ardizzone2018analyzing}.
\item Assessing the quality of the generated samples may not be trivial for certain modalities (e.g., non-visual features). With the specialized capability of the normalizing flow construction, our model estimates the posterior probabilities $p(\mathbf{x}^t | \mathbf{y}^t)$ of the generated sequences at each time point for potential downstream analyses involving uncertainty.
\item We demonstrate an interesting practical application of our model in a longitudinal neuroimaging dataset. In an intuitive
manner, we show that the generated longitudinal brain pathology trajectories can lead to identifying specific regions in the brain which are statistically associated with the manifestation of Alzheimer's disease.
\end{inparaenum}

\section{Methods}
In this section, we describe our conditional sequence generation method called Conditional Recurrent Flow (\textit{CRow{}}).
We first describe a conditional invertible neural network (cINN) \cite{ardizzone2018analyzing} which is one component of our model. Then, we explain how to incorporate temporal context gating and discuss the settings where CRow{} can be useful.

\subsection{Conditional Sample Generation}
Naturally, an inverse problem can be posed as a sample generation procedure by sampling a latent variable $\mathbf{z}$ and inverse mapping it to $\mathbf{x} = f^{-1}(\mathbf{z})$.
The most critical concern is that we cannot specifically `choose' to generate an $\mathbf{x}$ of interest since a latent variable $\mathbf{z}$ does not provide any interpretable associations with $\mathbf{x}$.

In other words, estimating the \textit{conditional probability} $p(\mathbf{x} | \mathbf{y})$ is desirable since it represents an underlying phenomenon of the input $\mathbf{x} \in \mathbb{R}^d$ and covariate $\mathbf{y} \in \mathbb{R}^k$ (e.g., the probability of a specific brain imaging measure $\mathbf{x}$ of interest given a diagnosis $\mathbf{y}$).
In fact, when we cast this problem into a normalizing flow, the goal becomes to construct an invertible network $f$ which maps a given input $\mathbf{x} \in \mathbb{R}^d$ to its corresponding covariate/label $\mathbf{y} \in \mathbb{R}^l$ and its latent variable $\mathbf{z} \in \mathbb{R}^m$ such that $[\mathbf{y},\mathbf{z}] = f(\mathbf{x})$, and the mapping must have an inverse $\mathbf{x} = f^{-1}([\mathbf{y},\mathbf{z}])$ to be recovered.

Specifically, when a flow-based model jointly encodes latent and label information
(i.e., $[\mathbf{y},\mathbf{z}] = \mathbf{v} = f(\mathbf{x})$ via Eq.~\eqref{eq:coupling})
while ensuring that $p(\mathbf{y})$ and $p(\mathbf{z})$ are independent,
then the network becomes conditionally invertible.
Such network can be theoretically constructed through a bidirectional-type training \cite{ardizzone2018analyzing}, and this
allows a conditional sampling $\mathbf{x} = f^{-1}([\mathbf{y},\mathbf{z}])$ and the posterior estimation $p(\mathbf{x} | \mathbf{y})$.
This training process involves several losses: (1) $\mathcal{L}_\mathbf{Z}(p(\mathbf{y},\mathbf{z}), p(\mathbf{y})p(\mathbf{z}))$ enforces the independence between $p(\mathbf{y},\mathbf{z})$ and $p(\mathbf{y})p(\mathbf{z})$. (2) $\mathcal{L}_\mathbf{Y}(\mathbf{y}, \hat{\mathbf{y}})$ is the supervised label loss between our prediction $\hat{\mathbf{y}}$ and the ground truth $\mathbf{y}$. (3) $\mathcal{L}_\mathbf{X}(p(\mathbf{x}),p_\mathbf{X})$ improves the likelihood of the input $\mathbf{x}$ with respect to the prior $p_\mathbf{X}$. $\mathcal{L}_\mathbf{Z}$ and $\mathcal{L}_\mathbf{X}$ are based on a kernel-based moment matching measure called Maximum Mean Discrepancy (MMD) \cite{dziugaite2015training}. See supplement for more details.

In practice, $\mathbf{x}$ and $[\mathbf{y},\mathbf{z}]$ may not be of the same dimensions. To construct a square triangular Jacobian matrix, zero-padding both $\mathbf{x}$ and $[\mathbf{y},\mathbf{z}]$ can alleviate this issue while also increasing the intermediate subnetwork dimensions for higher expressive power.
Also, the forward mapping is essentially a prediction task that we encounter often in computer vision and machine learning, i.e., predicting $\mathbf{y} = f(\mathbf{x})$ or maximizing the likelihood $p(\mathbf{y} | \mathbf{x})$ without explicitly utilizing the latent $\mathbf{z}$.
On the other hand, the inverse process of deriving $\mathbf{x} = f^{-1}(\mathbf{y})$, allows a more scientifically based analysis of the underlying phenomena, e.g., the interaction between brain and observed cognitive function.

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.9\columnwidth]{figs/networks/our_network_forward.png}

    \caption{Our conditional sequence generation model. Only the forward map of a single block (two coupling layers) is shown for brevity. The inverse map involves a similar order of operations (analogous to Fig.~\ref{fig:coupling-layer-forward} and Fig.~\ref{fig:coupling-layer-inverse})}
    \label{fig:network}
\end{figure}

\subsection{Conditional Recurrent Flow}
The existing normalizing flow type networks cannot explicitly incorporate sequential data which are now increasingly becoming important in various applications.
Successful recurrent models such as gated recurrent unit (GRU) \cite{chung2014empirical,tang2015document} and Long short-term memory (LSTM) \cite{hochreiter1997long,sak2014long} explicitly focus on encoding the ``memory'' about the past time points and output proper state information for accurate sequential predictions given the past.
Similarly, generated sample sequences must also follow sequentially sensible patterns or trajectories resembling likely sequences by encoding appropriate temporal information for the subsequent time points.

In order to overcome the issues above, we introduce Conditional Recurrent flow (CRow{}) model for conditional sequence generation. Given a sequence of input/output pairs $\{\mathbf{u}^t,\mathbf{v}^t\}$ for $t=1,\dots,T$ time points, modeling the relationship between the variables across time needs to also account for the temporal characteristic of the sequence.
Variants of recurrent neural networks (RNN) such as GRU and LSTM have been showing success in numerous sequential problems, but they only enable forward mapping $\mathbf{u} \rightarrow \mathbf{v}$.
We are specifically interested in constructing an invertible network which is also recurrent such that given a sequence of inputs $\{\mathbf{u}^t\}$ (i.e., $\{\mathbf{x}^t\}$) and their outputs $\{\mathbf{v}^t\}$ (i.e., $\{\mathbf{y}^t\}$), we can successfully model the invertible relationship between those sequences.
We now show our conditional recurrent flow model called \textit{CRow{}} which allows the conditional mapping between the \textit{sequence} of input $\{\mathbf{u}^t\}$ and output $\{\mathbf{v}^t\}$ where $\{\mathbf{v}^t\} = \{\mathbf{y}^t,\mathbf{z}^t\}$ for posterior estimation and conditional sequence generation.

Without loss of generality, we can describe our model in terms of generic $\{\mathbf{u}^t\}$ and $\{\mathbf{v}^t\}$.
We follow the coupling block described in Eq.~\eqref{eq:coupling} and Eq.~\eqref{eq:inverse-coupling} to setup a normalizing flow type invertible model.
Then, we impose the recurrent nature to the model by allowing the model to learn and pass a hidden state $\mathbf{h}^t$ to the next time point through the recurrent subnetworks.
Specifically, we construct a \textit{recurrent subnetwork} $q$ which contains a recurrent network (i.e., GRU) internally.
This allows $q$ to take the previous hidden state $\mathbf{h}^{t-1}$ and output the next hidden state $\mathbf{h}^t$ as $[\mathbf{q}, \mathbf{h}^t] = q(\mathbf{u}, \mathbf{h}^{t-1})$ where $\mathbf{q}$ is an element-wise transformation vector derived from $\mathbf{u}$ analogous to the output of $s(\mathbf{u})$.
In previous coupling layers, two transformation vectors $\mathbf{s} = s(\cdot)$ and $\mathbf{r} = r(\cdot)$  were explicitly computed from two subnetworks for each layer.
For CRow{}, we follow the structure of Glow \cite{kingma2018glow} which computes a single vector $\mathbf{q} = q(\cdot)$ and split it as $[\mathbf{s}, \mathbf{r}] = \mathbf{q}$.
This allows us to use a single hidden state while concurrently learning $[\mathbf{s}, \mathbf{r}]$ which we denote as $\mathbf{s} = q_s(\cdot)$ and $\mathbf{r} = q_r(\cdot)$ to indicate the corresponding vectors.
Thus, at each $t$ with given $[\mathbf{u}^t_1, \mathbf{u}^t_2] = \mathbf{u}^t$ and $[\mathbf{v}^t_1, \mathbf{v}^t_2] = \mathbf{v}^t$ our model becomes as follows:
\begin{equation}
\small
\begin{aligned}\label{eq:rec-coupling}
    \mathbf{v}^t_1 &= \mathbf{u}^t_1 \otimes \exp (q_{s_2}(\mathbf{u}^t_2, \mathbf{h}^{t-1}_2)) + q_{r_2}(\mathbf{u}^t_2, \mathbf{h}^{t-1}_2) \\
    \mathbf{v}^t_2 &= \mathbf{u}^t_2 \otimes \exp (q_{s_1}(\mathbf{v}^t_1, \mathbf{h}^{t-1}_1)) + q_{r_1}(\mathbf{v}^t_1, \mathbf{h}^{t-1}_1)
\end{aligned}
\end{equation}
and the inverse is
\begin{equation}
\small
  \begin{aligned}\label{eq:inverse-rec-coupling}
    \mathbf{u}^t_2 &= (\mathbf{v}^t_2 - q_{r_1}(\mathbf{v}^t_1, \mathbf{h}^{t-1}_1)) \oslash \exp (q_{s_1}(\mathbf{v}^t_1, \mathbf{h}^{t-1}_1)) \\
    \mathbf{u}^t_1 &= (\mathbf{v}^t_1 - q_{r_2}(\mathbf{u}^t_2, \mathbf{h}^{t-1}_2)) \oslash \exp (q_{s_2}(\mathbf{u}^t_2, \mathbf{h}^{t-1}_2)).
\end{aligned}
\end{equation}
Note that the hidden states $\mathbf{h}^t_1$ and $\mathbf{h}^t_2$ generated from the RNN components of the subnetworks are implicitly used within the subnetwork architecture (i.e., inputs to additional fully connected layers) and also passed down to their corresponding RNN component in the next time point as shown in Fig.~\ref{fig:network}.

\subsubsection{Temporal Context Gating}\label{sec:tgc}
A standard (single) coupling layer transforms only a part of the input (i.e., $\mathbf{u}_1$ in Eq.~\eqref{eq:real-nvp}) by design for the following triangular Jacobian matrix:
\begin{equation}\label{eq:jacobian}
\small
    J_\mathbf{v} = \frac{\partial \mathbf{v}}{\partial \mathbf{u}} = \begin{vmatrix} \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_2} \\
    \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_2}
    \end{vmatrix}
    = \begin{vmatrix}
    I & 0 \\
    \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & diag(\exp s(\mathbf{u}_1))
    \end{vmatrix}
\end{equation}
thus $J_\mathbf{v} = \exp(\sum_i (s(\mathbf{u}_1))_i)$.
This is a result from Eq.~\eqref{eq:coupling} the element-wise operations on $\mathbf{u}_2$ for diagonal ${\partial \mathbf{v}_2}/{\partial \mathbf{u}_2}$  and (2) the bypassing of $\mathbf{u}_1$ for ${\partial \mathbf{v}_1}/{\partial \mathbf{u}_1} = I$ and ${\partial \mathbf{v}_1}/{\partial \mathbf{u}_2}  = 0$.
Ideally, transforming $\mathbf{u}_1$ would be beneficial as well but avoided by the coupling layer design since an effective transformation cannot be learned from neither $\mathbf{u}_1$ nor $\mathbf{u}_2$ directly.
In case of CRow{}, it incorporates a hidden state $\mathbf{h}^{t-1}$ from the previous time point which is \textit{not} a part of the variables which we model the relationship of (i.e., $\mathbf{u}$ and $\mathbf{v}$).
This is our recurrent information which adjust the mapping function $f(\cdot)$ to allow more accurate mapping depending on the previous sequences which is crucial for sequential modeling.

Specifically, we incorporate a \textit{temporal context gating} $f_{\textrm{TCG}}(\alpha^t, \mathbf{h}^{t-1})$ using the temporal information $\mathbf{h}^{t-1}$ on a given input $\alpha^t$ at $t$ as follows:
\begin{equation}
\small
\begin{aligned}\label{eq:cgate}
    f_{\textrm{TCG}}(\alpha^t, \mathbf{h}^{t-1}) &= \alpha^t \otimes cgate(\mathbf{h}^{t-1})   \quad \textrm{(forward)}\\
    f^{-1}_{\textrm{TCG}}(\alpha^t, \mathbf{h}^{t-1}) &= \alpha^t \oslash cgate(\mathbf{h}^{t-1})   \quad \textrm{(inverse)}
\end{aligned}
\end{equation}
where $cgate(\mathbf{h}^{t-1})$ can be any learnable function with a sigmoid function at the end.
This is analogous to the context gating \cite{miech2017learnable} in video analysis which scales the input $\alpha^t$ (since $cgate(\mathbf{h}^{t-1}) \in (0,1)$) based on some context, which in our setup is the previous time points.

In the context of $J_\mathbf{v}$ computation in Eq.~\eqref{eq:jacobian}, we observe that this `auxiliary' variable $\mathbf{h}^{t-1}$ could safely be used to transform $\mathbf{u}_1$ without altering the triangular nature of the Jacobian matrix by (1) learning an element-wise operation $cgate(\mathbf{h}^{t-1})$ on $\mathbf{u}_1$ for diagonal ${\partial \mathbf{v}_1}/{\partial \mathbf{u}_1}$ which (2) is not a function of $\mathbf{u}_2$ so ${\partial \mathbf{v}_1}/{\partial \mathbf{u}_2} = 0$.
Thus, we now have
\begin{equation}
\small
    J_\mathbf{v} = \begin{vmatrix} \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_1}{\partial \mathbf{u}_2} \\
    \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_2}
    \end{vmatrix}
    = \begin{vmatrix}
    diag(cgate(\mathbf{h}^t)) & 0 \\
    \frac{\partial \mathbf{v}_2}{\partial \mathbf{u}_1} & diag(\exp s(\mathbf{u}_1))
    \end{vmatrix}
\end{equation}
where $J_\mathbf{v} = [\prod_j cgate(\mathbf{h}^t)_j]*[\exp(\sum_i (s(\mathbf{u}_1))_i)]$.

As seen in Fig.~\ref{fig:network}, we place $f_{\textrm{TCG}}$ to the originally non-transformed variable of each \textit{layer} of a block (i.e., $\mathbf{u}_2$ in the bottom layer and $\mathbf{v}_1$ in the top layer).
We specifically chose a gating mechanism for conservative adjustments so that the original information is preserved to a large degree through simple but learnable `weighting'.
The full forward and inverse steps involving $f_{\textrm{TCG}}$ can easily be formulated by following Eq.~\eqref{eq:rec-coupling} and Eq.~\eqref{eq:inverse-rec-coupling} while respecting the order of operations seen in Fig.~\ref{fig:network}. See appendix for details.

\subsection{How do we use CRow{}?}
Before we demonstrate the results using CRow{} on the experiments, let us briefly describe how we can use CRow{} for conditional sequence generation and density estimation.
In essence, CRow{} aims to model an invertible mapping $[\{\mathbf{y}^t\},\{\mathbf{z}^t\}] = f(\{\mathbf{x}^t\})$ between sequential/longitudinal measures $\{\mathbf{x}^t\}$ and their corresponding observations $\{\mathbf{y}^t\}$ with $\{\mathbf{z}^t\}$ encoding the latent information across $t=1,\dots,T$ time points.
Once we train $f(\cdot)$, we can perform the following exemplary tasks:

\textbf{(1) Conditional sequence generation:} Given a series of observations of interest $\{\mathbf{y}^t\}$, we can sample $\{\mathbf{z}^t\}$ (each independently from a standard normal distribution) to generate $\{\mathbf{x}^t\} = f^{-1}([\{\mathbf{y}^t\},\{\mathbf{z}^t\}])$. The advantage comes from how $\{\mathbf{y}^t\}$ can be flexibly constructed (either seen or unseen from the data) such as an arbitrary disease progression over time.
Then, we randomly generate corresponding measurements $\{\mathbf{x}^t\}$ to observe the corresponding longitudinal measurements for both quantitative and qualitative downstream analyses. Since the model is recurrent, the sequence length can be extended beyond the training data to model the future trajectory.

\textbf{(2) Sequential density estimation:} Conversely, given  $\{\mathbf{x}^t\}$, we can predict $\{\mathbf{y}^t\}$, and more importantly, estimate the density $p_\mathbf{X}(\{\mathbf{x}^t\})$ at each $t$.
When $\{\mathbf{x}^t\}$ is generated from $\{\mathbf{y}^t\}$, the estimated density can indicate the `integrity' of the generated sample (i.e., low $p_\mathbf{X}$ implies that sequence is perhaps less common with respect to $\{\mathbf{y}^t\}$).

In the following section,
we demonstrate several experiments that CRow{} is able to  precisely perform these tasks described above.

\section{Preliminary: Invertible Neural Networks}
We first describe an \textit{invertible neural network} (INN) which inverts an output back to its input for solving inverse problems (i.e., $\mathbf{z} = f(\mathbf{x}) \Leftrightarrow \mathbf{x} = f^{-1}(\mathbf{z})$).
This becomes the building block of our method; thus, before we present our main model, let us briefly describe a specific type of invertible structure which was originally specialized for density estimation with neural network models.

\subsection{Normalizing Flow}
Estimating the density $p_\mathbf{X}(\mathbf{x})$ of sample $\mathbf{x}$ is a classical statistical problem in various fields including computer vision and machine learning in, e.g., uncertainty estimation \cite{gal2015bayesian,gal2016dropout}.
For tractable computation throughout the network, Bayesian adaptations are popular \cite{ranganath2015deep,fortunato2017bayesian,papamakarios2016fast,kingma2015variational,kendall2017uncertainties}, but these methods make assumptions on the prior distributions (e.g., exponential families).

A \textit{normalizing flow} \cite{rippel2013high,rezende2015variational}, first learns a function $f(\cdot)$ which maps a sample $\mathbf{x}$ to $\mathbf{z} = f(\mathbf{x})$ where is from a standard normal distribution $\mathbf{Z}$.
Then, with a change of variables formula, we estimate
\begin{equation*}
\small
p_\mathbf{X}(\mathbf{x}) = p_\mathbf{Z}(\mathbf{z}) / |J_\mathbf{X}|, \quad J_\mathbf{X} = \left| \frac{\partial [\mathbf{x} = f^{-1}(\mathbf{z})]}{\partial \mathbf{z}} \right|
\end{equation*}
where $J_\mathbf{X}$ is a Jacobian determinant.
Thus, $f(\cdot)$ must be invertible, i.e., $\mathbf{x} = f^{-1}(\mathbf{z})$, and to use a neural network as $f(\cdot)$, a coupling layer structure was introduced in Real-NVP \cite{dinh2014nice,dinh2016density} for an easy inversion and efficient $J_\mathbf{X}$ computation as follows.

\begin{figure}[!t]
    \centering
    \begin{subfigure}[t]{0.49\columnwidth}
        \centering
        \includegraphics[width=0.95\columnwidth]{figs/networks/real-nvp-forward.png}
        \caption{\footnotesize Forward map (Eq.~\eqref{eq:real-nvp})}
        \label{fig:coupling-layer-forward}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.49\columnwidth}
        \centering
        \includegraphics[width=0.95\columnwidth]{figs/networks/real-nvp-inverse.png}
        \caption{\footnotesize Inverse map (Eq.~\eqref{eq:inverse-real-nvp})}
        \label{fig:coupling-layer-inverse}
    \end{subfigure}
    \caption{Coupling layer in normalizing flow. Note the change of operation orders: $\mathbf{u} \rightarrow \mathbf{v}$ in forward and $\mathbf{v} \rightarrow \mathbf{u}$ in inverse.}
    \label{fig:coupling-layer}
\end{figure}

Without loss of generality, in the context of network structures, we use an input $\mathbf{u} \in \mathbb{R}^d$ and an output $\mathbf{v} \in \mathbb{R}^d$. First, we split $\mathbf{u}$ into $\mathbf{u}_1 \in \mathbb{R}^{d_1}$ and $\mathbf{u}_2 \in \mathbb{R}^{d_2}$ where $d = d_1 + d_2$. Then, we forward map $\mathbf{u}_1$ and $\mathbf{u}_2$ to $\mathbf{v}_1$ and $\mathbf{v}_2$ respectively (Fig.~\ref{fig:coupling-layer-forward}):
\begin{equation}\label{eq:real-nvp}
\small
    \mathbf{v}_1 = \mathbf{u}_1, \quad \mathbf{v}_2 = \mathbf{u}_2 \otimes \exp (s(\mathbf{v}_1)) + r(\mathbf{v}_1)
\end{equation}
where $s, r$ are independent functions (i.e., subnetworks), and $\otimes$ and $+$ are element-wise product and addition respectively.
A straightforward arithmetic allows an exact inverse from $\mathbf{v}$ to $\mathbf{u}$ (Fig.~\ref{fig:coupling-layer-inverse}):
\begin{equation}\label{eq:inverse-real-nvp}
\small
    \mathbf{u}_1 = \mathbf{v}_1, \quad \mathbf{u}_2 = (\mathbf{v}_2 - r(\mathbf{v}_1)) \oslash \exp(-s(\mathbf{v}_1))
\end{equation}
where the subnetworks are \textit{identical} to those used in the forward process in Eq.~\eqref{eq:real-nvp}, and $\oslash$ and $-$ are element-wise division and subtraction respectively.
Not that the subnetworks are \textit{not} explicitly inverted, thus any arbitrarily complex network can be utilized.
Also, the Jacobian matrix $\partial \mathbf{v} / \partial \mathbf{u}$ is triangular so its determinant $J_\mathbf{v}$ is just the product of diagonal entries (i.e., $\prod_i \exp(-s(\mathbf{v}_1))_i)$ which is extremely easy to compute
(we will discuss this further in Sec.~\ref{sec:tgc}).

Then, to apply a transform on the ``bypassed'' $\mathbf{u}_1$ as well, a \textit{coupling block} (consisting of two complementary coupling layers) is usually constructed:
\begin{equation}
\small
\begin{aligned}\label{eq:coupling}
    \mathbf{v}_1 &= \mathbf{u}_1 \otimes \exp (s_2(\mathbf{u}_2)) + r_2(\mathbf{u}_2) \\
    \mathbf{v}_2 &= \mathbf{u}_2 \otimes \exp (s_1(\mathbf{v}_1)) + r_1(\mathbf{v}_1)
\end{aligned}
\end{equation}
and its inverse
\begin{equation}
\small
\begin{aligned}\label{eq:inverse-coupling}
    \mathbf{u}_2 &= (\mathbf{v}_2 - r_1(\mathbf{v}_1)) \oslash \exp(-s_1(\mathbf{v}_1)) \\
    \mathbf{u}_1 &= (\mathbf{v}_1 - r_2(\mathbf{u}_2)) \oslash \exp (-s_2(\mathbf{u}_2)).
\end{aligned}
\end{equation}
Such a series of transformations allow a more complex mapping with a chain of efficient Jacobian determinant computations, i.e., $\det(AB) = \det(A)\det(B)$. More details are included in the appendix.

\section{Goal}
\begin{itemize}
	\item (1) Invertible (2) sequential (3) conditional model
	\item Given:
\begin{enumerate}
	\item Input sequence $X = x^{(1)},\dots,x^{(T)}$ (e.g., connectivity sequence, video, etc.)
	\item Corresponding covariates $Y = y^{(1)},\dots,y^{(T)}$ (e.g., age, gender, cognition, action label, class label, etc.)
\end{enumerate}
	\item Do:
	\begin{enumerate}
		\item Derive its sequential latent variables $Z = z^{(1)},\dots,z^{(T)}$
		\item With a $Z$ derived from $X$, change the covariates (at each $t$ or for all $t$) then generate a new sequence $X'$ (e.g., same subject, new trajectory conditioned on arbitrary cognition scores)
		\item If $Z$ is sampled such that $z^{(t)} \sim N(0,1)$ (while keeping $Y$), generate similar sequences $X'$.
		\item \textit{Exact invertible} process with scale/location operations.
		\item Then, for whatever generated $X'$, estimate the sequential densities $P_\mathcal{X}(X') = P_\mathcal{X}(x'^{(1)}), \dots, P_\mathcal{X}(x'^{(T)})$.
		\item Easy computation of Jacobian determinant for fast density estimation.
	\end{enumerate}
\end{itemize}

\section{Normal (Conditional) Inversion (Dinh et al., Ardizzone et al.)}
The goal is to approximate $p(x|y,z)$ where $y$ is an observable measurement (e.g., covariates, classes, etc.) and $z$ is a latent variable. Regardless of how $x$ is sampled, if we have an invertible function $f_\theta(x) = [y,z]$ such that $x = f_\theta^{-1}(y, z)$, we can sample $z \sim P_\mathcal{Z}(z)$ and exactly derive $x$:
\begin{equation}
x = g_\theta(y,z).
\end{equation}

An invertible architecture acts as an invertible function $f_\theta=g_\theta^{-1}$. Notice the network parameters $\theta$ is the same for both $f$ and $g$ implying that this is a single network trained in a bi-directional manner. Both $y$ and $z$ are computed with this network without two separate networks:
\begin{equation}
[y,z] = f_\theta(x) = [f_\theta^y(x), f_\theta^z(x)] = g_\theta^{-1}(x).
\end{equation}
For dimensionality differences, padding seems to be useful.

Then, there are mainly two cases to consider: (1) given $x$ and $y$, compute $z=f_\theta^z(x)$ and estimate $P_\mathcal{X}(x)$ and (2) sample $z$ and estimate $P_\mathcal{X}(x)$ with the $y$ of interest:
\begin{align}
P_\mathcal{X}(x) &= P_\mathcal{Z}(z)|\det(J^{-1})| \\
&=  P_\mathcal{Z}(f_\theta^z(x))\left|\det \left( \frac{\partial z}{\partial x} \right) \right| =  P_\mathcal{Z}(f_\theta^z(x))\left|\det \left( \frac{\partial f_\theta^z(x)}{\partial x} \right) \right| \quad \textrm{(given $x$)}\\
&= P_\mathcal{Z}(z)\left|\det \left( \frac{\partial x}{\partial [y,z]} \right)^{-1} \right| = P_\mathcal{Z}(z)\left|\det \left( \frac{\partial g_\theta(y,z)}{\partial [y,z]} \right)^{-1} \right| \quad \textrm{(sample $z$)}
\end{align}
Regardless of the case, $|J|$ can be computed solely based on the scale function $s$. The following is when there is only one block (no alternation):
\begin{equation}
\det \left( \frac{\partial f_\theta(x)}{\partial x} \right)= \exp \left[ \sum_k s_k(u_1) \right]
\end{equation}
for all $k$ entries of the first partition $u_1$ of $x$. With the alternation (coupling), use the fact that
\begin{equation}
\det(AB) = \det(A) \det(B).
\end{equation}
The inversion also works
\begin{equation}
(f_1 \circ f_2)^{-1} = f_1^{-1} \circ f_2^{-1}
\end{equation}

\section{Conditional Sequential Inversion?}
Extend the above formulations to sequential setting. We want to approximate $p(x^{(j)} | y^{(j)}, z^{(j)})$ for each time point $j = 1, \dots, J$. In other words, we need an invertible function $f_\theta(x^{(j)}) = [ y^{(j)}, z^{(j)}]$ such that
\begin{equation}
x^{(j)} = g_\theta(y^{(j)}, z^{(j)})
\end{equation}
is possible. But this does not utilize any sequential aspect of the data so all the time points are independent from each other except for the fact that the same model parameters $\theta$ are used. A more ideal setting would be to somehow use the previous sequence information $z^{(j-1)}$:
\begin{equation}
x^{(j)} = g_\theta(y^{(j)}, z^{(j)}, z^{(j-1)})
\end{equation}
where $z^{(j-1)}$ is the latent variable for the previous time point. Note that depending on how this is derived, it may contain knowledge about just the previous $j-1$ (i.e., each $z^{(j)}$ is derived independently), or it may be contain all the previous sequence information (i.e., each $z^{(j)}$ is sequentially derived).

How should we incorporate the latent information from other time points? We let the intermediate neural network components to incorporate this latent information. In other words, the scaling and translation functions can be neural networks that can \textit{explicitly} incorporate latent representations
\begin{equation}
v_1^{(j)} = u_1^{(j)} \otimes \exp(s_2(u_2^{(j)}, z^{(j-1)}) + t_2(u_2^{(j)}, z^{(j-1)})
\end{equation}
which is for the first block $u_1$. For the first time point $j=1$, $z^{(0)}$ can be some random/fixed initial latent variable which is commonly used in other sequential models.

Now, the new function $f_\theta(x^{(j)})$ can be represented similarly with the inclusion of a latent variable as a new argument:
\begin{equation}
[y^{(j)},z^{(j)}] = f_\theta(x^{(j)},z^{(j-1)}) = [f_\theta^y(x^{(j)},z^{(j-1)}), f_\theta^z(x^{(j)},z^{(j-1)})] = g_\theta^{-1}(x^{(j)},z^{(j-1)}).
\end{equation}
The inversion is trivial since even with the new networks with $z^{(j-1)}$, we consistently use the network to perform the same forward process. For instance, suppose we are given $X$ (and some $Y$). Once we sequentially generate $[y^{(j)},z^{(j)}]$ to construct $Z$, we can still perfectly reconstruct $X$ by generating $x^{(j)}$ sequentially with the same $Z$. This process is still guaranteed as long as the networks are used in the same order.

How about the Jacobian? Now it also depends on $z^{(j-1)}$ which makes it ``conditional'' on the previous sequences. Since the Jacobian determinant is merely a volume ``ratio'' between two distributions transformed by a function of interest, this function need not be static. The previous latent information is used as a ``function parameter'', no different from $\theta$. In that sense, it may be more appropriate to express $f_\theta^z(x^{(j)},z^{(j-1)})$ as $f_{\theta,z^{(j-1)}}^z(x^{(j)})$ to be explicit about how the function only maps $x^{(j)}$ to $z^{(j)}$.
\begin{align}
P_\mathcal{X}(x^{(j)}) &= P_\mathcal{Z}(z^{(j)})|\det(J^{-1})| \\
&=  P_\mathcal{Z}(f_{\theta,z^{(j-1)}}^z(x^{(j)}))\left|\det \left( \frac{\partial z^{(j)}}{\partial x^{(j)}} \right) \right| \\
&=  P_\mathcal{Z}(f_{\theta,z^{(j-1)}}^z(x^{(j)}))\left|\det \left( \frac{\partial f_{\theta,z^{(j-1)}}^z(x^{(j)})}{\partial x^{(j)}} \right) \right| \quad \textrm{(given $x^{(j)}$)}\\
&= P_\mathcal{Z}(z^{(j)})\left|\det \left( \frac{\partial x^{(j)}}{\partial [y^{(j)},z^{(j)}]} \right)^{-1} \right| \\
&= P_\mathcal{Z}(z^{(j)})\left|\det \left( \frac{\partial g_{\theta,z^{(j-1)}}(y^{(j)},z^{(j)})}{\partial [y^{(j)},z^{(j)}]} \right)^{-1} \right| \quad \textrm{(sample $z^{(j)}$)}
\end{align}

\subsection{Possible Networks}
Any network model that naturally make use of latent variables should work. Although the full model resembles a recurrent-type model, depending on how much we want the previous latent variables to be informative of the past, it may not even need to be RNN models. If we were to use a RNN model for $s$ and $t$, GRU seems reasonable since the cell actually only outputs the latent representations ($h$ which is like $z$ we want), which is exactly what we want. At least on Moving MNIST, GRU gets the job done. This is one place where we can be creative.

\section{Architecture}
Variables at time $j$ and partitions $k=1,2$:
\begin{enumerate}
	\item \textbf{Input}: $x^{(j)} \in \mathbb{R}^d$
	\item \textbf{Latent variable}: $z^{(j)} \in \mathbb{R}^{d_z}$
	\item \textbf{Conditional variable}: $y^{(j)} \in \mathbb{R}^{d_y}$ where $d_x = d_y + d_z$
	\item \textbf{Partitioned inputs}: $u_k^{(j)}\in \mathbb{R}^{d_{u_k}}$ where $d_x = d_{u_1} + d_{u_2}$
	\item \textbf{Partitioned outputs}: $v_k^{(j)}\in \mathbb{R}^{d_{u_k}}$
	\item \textbf{Scale functions}: $s_1(u_1^{(j)}, z^{(j-1)}): d_{u_1} \times d_z \rightarrow d_{u_2}$ and $s_2(u_2^{(j)}, z^{(j-1)}): d_{u_2} \times d_z \rightarrow d_{u_1}$ (notice the cross-mapping from $u_1$ to $u_2$, vice versa)
	\item \textbf{Translation functions}: $t_1(u_1^{(j)}, z^{(j-1)}): d_{u_1} \times d_z \rightarrow d_{u_2}$ and $t_2(u_2^{(j)}, z^{(j-1)}): d_{u_2} \times d_z \rightarrow d_{u_1}$ (notice the cross-mapping from $u_1$ to $u_2$, vice versa)
\end{enumerate}

\subsection{Forward Mapping}
\begin{equation}
v_1^{(j)} = u_1^{(j)} \otimes \exp(s_2(u_2^{(j)}, z^{(j-1)}) + t_2(u_2^{(j)}, z^{(j-1)})
\end{equation}
\begin{equation}
v_2^{(j)} = u_2^{(j)} \otimes \exp(s_1(v_1^{(j)}, z^{(j-1)}) + t_1(v_1^{(j)}, z^{(j-1)})
\end{equation}

\subsection{Inverse Mapping}
Note the covariates $y^{(j)}$ are part of $v^{(j)} = [v_1^{(j)}, v_2^{(j)}]$.
\begin{equation}
u_2^{(j)} = (v_2^{(j)} - t_1(v_1^{(j)}, z^{(j-1)})) \otimes \exp(-s_1(v_1^{(j)}, z^{(j-1)})
\end{equation}
\begin{equation}
u_1^{(j)} = (v_1^{(j)} - t_2(u_2^{(j)}, z^{(j-1)})) \otimes \exp(-s_2(u_2^{(j)}, z^{(j-1)})
\end{equation}

\setlength{\textfloatsep}{5pt
\begin{algorithm}[!t]
\caption{\label{alg:rprop} Conditional Invertible RNN (Post-Training)}
\begin{algorithmic}
\STATE \textbf{Given:}
\STATE 1) Sequence $X$
\STATE 2) Observed variables $Y$ (Optional)
\STATE 3) Sampled $Z$ (Optional)
\FOR{$j=1,2,\dots T$}
    \STATE $[y^{(j)},z^{(j)}] = f_\theta(x^{(j)},z^{(j-1)})$
    \STATE ehh....
\ENDFOR
\end{algorithmic}
\end{algorithm}

\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{figs/fig1.png}
\caption{An example for $j=1$ to $j=2$ with one coupling.}
\end{figure}

redpajama_set_name: RedPajamaArXiv


Benitoite is an extremely rare barium titanium silicate mineral named after San Benito County, California - the location where it was first found and described in 1907. There it occurs in a host rock of blue schist where the benitoite crystals formed in fractures from hydrothermal fluids. Benitoite is known to occur in just a few other locations worldwide. The Dallas Gem Mine in San Benito County is the only place in the world where gem-quality benitoite is found and where benitoite is found in specimen-quality crystals.
When cut as a gemstone, benitoite has an appearance and optical properties that are similar to sapphire. Most specimens are blue to violetish-blue, although a few rare orange specimens are known.


text:
A Review of Dying, by René Belletto
By Ryan W. Bradley
"It was here that I was dying," appears on the first page of René Belletto's novel Dying. And in the end this is really nothing more than a thesis statement for what is more meditation than novel. While reading Dying, I was often reminded of another work that seems to challenge genre conventions: Minotaur, by Israeli novelist Benjamin Tammuz. Dying mixes and matches its genre influences throughout. Though elements of adventure and spy narratives are prevalent, the foundation of the story is romance, much like Tammuz' masterpiece. But instead of presenting a great novel of love and desire as Tammuz did, Belletto indulges more in philosophy. Dying reads less as a novel than a journal of fictional thoughts exploring a theme.
The Immortality Machine: A review of Coda by René Belletto
By J.P. Smith
Paris is murder in August. It's the month when everyone leaves town and abandons the city to the tourists. Hotels fill with an assortment of American accents (the Germans have all gone to England); waiters, at least at the restaurants that remain open, spend far too long explaining the difference between service compris and service non compris. It's a hot month, a dead month, the Seine gets punky, and every man and woman who's leaving on this first day of August—meaning pretty much everyone—in fact on this first page of Coda—is right now sitting in traffic, anticipating their arrival at their vacation home in the Midi or some prefab bungalow bought for a small fortune in the Auvergne, next to the neighbors with the goats and loud children. To come to Paris in August is to come to a city nakedly out of sorts. But in the latest novel by René Belletto to be translated into English, it's where we've just arrived.

redpajama_set_name: RedPajamaCommonCrawl


text:
Above-Pastor Paul's Radio and Tape Ministry
Pastor Paul ministered to a congregation on Friday evenings and Sunday mornings. He recorded his messages. He had a Radio Ministry, the 'Up-Look Broadcast', on KUXL -AM 1570- that aired his recorded messages Monday-Friday from 1:30-2:00pm in 30 minute segments. He also had a Tape Ministry providing the duplicated recorded messages to individuals, churches, African and Prison Ministries.
In an interview Pastor Paul was asked if he felt all the work he was doing within his mission distributing food and feeding the poor was a hindrance to ministering the Word of God on Sunday mornings, he said, "On the contrary my experiences during the week are the basis for sharing on Sunday...I minister what I see. The Word isn't alive until it's lived."
The reason for recording Pastor Paul's Messages was to be able to 'spread the gospel' beyond the church congregation by distributing his recorded messages. Don and Sue Hastings who became church members in 1975-76 were also professional musicians. They lead the church music ministry group 'Dove' that included their sister Linda and brother in-law Dean Kleven, and Deborah Schaeffer and Steve Arnopoulos, Pastor Paul's daughter and son. They were able to put together the sound equipment needed to record the messages during church services professionally creating excellent master tapes that were duplicated and distributed to church members after church services. And later they were supplied by the Prison Ministry to the Stillwater Prison library and to African congregations and individuals in Africa through the African Tape Ministry.
The 'Up-Look Radio Broadcast' was started when the congregation moved from the little church in Brooklyn Park to Mikro Kodesh in North Minneapolis. Mick Saunders, a longtime church member and friend, used the master tapes to produce the broadcast that aired on KUXL -AM1570- Monday-Friday from 1:30-2:00 pm weekly. He would add an introduction and conclusion to the daily message that was reduced to 30 minute segments. It was originally recorded at the previous Friday and Sunday services. Pastor Paul had the largest listening radio audience from the Twin Cities. Mick did a great job!
The scriptures says,
'Go to the highways and the by-ways and preach unto to all creation.'
So he did!
Above-Pastor Paul with Mick at Sunday Brunch
Pastor Paul and Mick Saunders break for Mission Sunday Brunch. Mick produced the 'Up-Look Broadcast' Radio Ministry that aired on KUXL-AM 1570-radio station Monday-Friday from 1:30-2:00pm weekdays. The broadcast had one of the largest listening radio audiences in the Twin Cities.
Above-Pastor Paul's Mission Historic Programs and Services Sign
This is a historic sign showing all programs and services. Historic Programs are credited with substantial mission growth and the growth in Grocery Shelf participation. The programs and services include: Disciples Ministry Church Mission, Worship Services; Outreach Services; Hot Meals, Grocery Shelf, Clothing Closet, Bread Bags, Bread Run Street Distribution, Radio Ministry on KUXL AM570. Before closing its door in 2012, Pastor Paul's Mission focused on Grocery Shelf Distribution to a boundary free Minnesota.


Thanks Jeanie. Now I'm waiting for spring to get back there!
Yvonne, This is also such a beautiful painting.
Great work and a truly beautiful location, you captured it well!
Thanks, I can't wait to get back there!


Wench by Dolen Perkins-Valdez – This is the story of four slave women who spend their summers at Tawana House, a real life resort in Ohio in the 1850s. Ohio was a free state but southern slaveowning men would vacation here and bring their slaves. The story explores the four differing reactions of these slave mistresses as they interact with free blacks, northern whites, and their masters.
This story started slow for me but by the end it was very compelling. The many aspects of thr cruelty of slave owners is discussed. I thought it was interesting that this was a real place that closed because the northern whites quit coming in part because they were uncomfortable with having slaves there. It then became a church school and eventually Wilberforce University, the country's oldest private African-American college.
Maybe This Time by Jennifer Crusie- Andie is getting married. But first she wants to get closure with her ex. He asks for a final favor. He will pay her a lot to take care of some orphaned relatives for a month until he can figure out what to do with them. They've driven off three nannies already. What Andie doesn't know is that she is walking into a house controlled by some very territorial ghosts.
This is what you expect from Jennifer Crusie- a funny fast read with good characters and a fast moving story.
A Cup of Friendship by Deborah Rodriguez – This is a novel by the author of KABUL BEAUTY SCHOOL. Sunny, an American, owns a coffee shop in Kabul. This is the story of the family that she gains there. Yazmina is a pregnant widow. Halajan owns the building and is hiding a secret love from her very traditional son. Isabel is a British journalist and Candace is a wealthy American who left her husband for an Afghan man.
This book is a great slice of life about what it is like to live in Kabul today.
I love the look of your winterized blog! Your reads sounds great, hope you enjoy them. Sorry to hear about the loss of your bird, that's never an easy thing.
Wench is somewhere in my TBR pile…. I need to pull it out and read it soon!


Recommended at least 2 months prior to move out of dorms. Call to schedule an appointment.
Money-Welcome to Virtual Finance - good personal financial information on the Virtual Finance website. Access Virtual Finance via the AF Portal (requires Common Access Card), then click the Life & Fitness tab , then Money-Welcome to Virtual Finance. Covers a variety of financial topics.


Get an alert with the newest ads for "shoes jimmy choo" in Toronto (GTA).
JIMMY CHOO SHOES 6 1/2 - 7 - LOUIS VUITTON BAGS..
Size 9 jimmy choo , Burberry shoe size nine .
Authentic jimmy choo patent pumps!
Overall in great condition....just a little mark in the upper corner. Shown in photo.


text:
It's Official: America Now Has a REAL Queen Melania
It's Official: America Now Has a REAL Queen and Her Name is Melania
In case you hadn't noticed, First Lady Melania Trump is hot.
The Left pretended to fawn over Michelle Obama, but we know the deal. Even the fake news media can't make up for Michelle Obama's lack of true style.
Melania has style oozing from every part of her body. But she has something Michelle Obama just didn't have: class
I'm not hammering the former First Lady, as her lack of class wasn't from trying. Class can be learned, and Michelle eventually acquired a smidgen. However, Melania appears to have been born with it.
Trending: Now We Know Why John Durham Delayed Report
In her first overseas jaunt as First Lady, you can see how the world views America's new First Lady: like a queen.
Look at how France's First Lady gushes over Melania, as Melania allows her to take it all in.
https://twitter.com/SylviaCoppola/status/867941059745239040?ref_src=twsrc%5Etfw
I'd like to be the first on record that I know of to anoint Melania, Queen Melania.
I suggest conservatives refer to her as Queen Melania just to piss off leftists.
Twitter certainly noted how Queen Melania compared to the wives of other heads of state. Most of the women didn't even bother to look at the camera, lest they be turned in to pumpkins.
https://twitter.com/chrisk2000/status/867858885163364353?ref_src=twsrc%5Etfw
Here is how BizPac Review described the event,
Later that day, the NATO first ladies visited Belgium's Queen Mathilde at the Royal Castle of Laeken, where Melania slayed in a glamorous off-the-shoulder, black lace gown.
The event provided a hilarious unplanned photo-op of Turkish first lady Emine Erdogan giving what appeared to be a side-eye to the statuesque Melania
While Melania opted to change outfits, French first lady Brigitte Macron wore the same mini-dress she wore hours earlier as she posed for a group photo with the NATO spouses.
I didn't check, but I'm betting the American press has said little about Melania's visit.
To do so would mean they would have to recognize Melania's class. Inevitably this would lead to the obvious comparisons between Queen Melania and Michelle.
Remember when Sophie Theallet said she wouldn't design a dress for then-regular Melania? Here's what I wrote at the time:
How will Melania handle the loss of Michelle Obama's fashion designer?
Can you believe that "fashion designer" Sophie Theallet considers not designing dresses for Melania Trump a slam? Talk about a pre-emptive strike. As if Melania Trump would hire that hack to dress her.
Has this chick actually seen the crap wardrobe she's made for Michelle Obama? One of the perks of becoming First Lady is not to look like the homeless.
As Fox News reports:
"Designer Sophie Theallet isn't a fan of President-elect Donald Trump and has announced she's refusing to dress the next first lady, Melania Trump.
"As one who celebrates and strives for diversity, individual freedom, and respect for all lifestyles, I will not participate in dressing or associating in any way with the next First Lady," Theallet wrote in an open letter she posted to Twitter on Thursday. "The rhetoric of racism, sexism, and xenophobia unleashed by her husband's presidential campaign are incompatible with the shared values we live by."
Theallet, 52, is a French-born designer who grew to prominence by dressing current first lady Michelle Obama on a number of occasions."
If Michelle Obama can get a designer prominence, what do you think Melania could do for one's career?!
For Melania to use Michelle designer would be a BIG, HUGE, no YUUUGGE step down, fashionably speaking.
Melania has upped the class at the White House.
Even the Left knows it.


Are you searching for home insurance coverage in Oxford, PA? Enter your Zip Code to compare the best options available nearby. Comparing multiple quotes is the best way to save money on your homeowners coverage in Chester county.
Have you been asked by your mortgage lender to get homeowners insurance? Did you just purchase your first home and understand the value of insuring it but don't know what type of coverage to buy? If so, here is some information to help you get the Oxford, Pennsylvania homeowners insurance you need.
If you are sensation a little bit anxious about receiving a home owner's insurance policies policy, you are not by itself. There are millions of other property owners out there that truly feel just like you as this is extremely crucial protection that is essential to shield your property and house from reduction. Make positive you are starting up with guidelines like these ahead of you start purchasing.
Make confident that you have a document of every little thing that is protected below your property insurance coverage. Make a thorough record, and just take images of all valuables. It also helps to preserve receipts for large ticket items. There is free of charge software online that will enable you to create a home inventory, area by space. If you need to make a declare, having all this info on hand will assist to speed up the procedure.
After buying your homeowner's insurance policy, go about your residence and just take photos of your belongings so you have a visible stock. Keep these pictures in a fireproof secure or at a relative's residence. These photos will support the insurance policy business doc your promises, and aid you get your cash more rapidly.
To keep your protection up to day, be positive to assessment your homeowner's policy each calendar year. Let your insurer know of changes in your house and residence that may possibly assist hold your rates down. For instance, if you have changed a shake roof with some thing far more fireproof, like composite shingles, you could get a top quality reduction.
If you can, spend off your property and you can preserve income. A individual who owns their property without having a home loan is viewed by insurance companies as being much more likely to correctly care for their home. As a result, numerous insurers give considerable premium reductions as a outcome. When you finally spend your mortgage, get in touch with your insurance policy companies to commence preserving.
The world wide web becomes a valuable tool when purchasing for the ideal house insurance coverage plan at the ideal possible price. You will be in a position to compare different coverage with various companies to locate the a single that will in shape your requirements the greatest. Be certain to check out the organization before signing on with them although. The greatest fee does not constantly suggest the best plan.
Think about increasing the deductible on your home insurance coverage coverage. A higher deductible on your insurance coverage plan can drastically reduced your annual house insurance coverage rates. However, by boosting the deductible, your property insurance firm will no for a longer time pay out for small statements, such as broken window repair, leaky pipe mend and slight wind and flood harm repairs.
At minimum once a calendar year assess the value of your house owners insurance coverage to that of other businesses to see in which the financial savings are. You may very properly locate another firm offering much much better rates than your personal or a price reduction for signing up with them. If you believe you can get considerable savings with yet another insurer, contact them up and explain to them you may possibly be fascinated and they may possibly present you with even far better numbers to receive you as a client.
Document all of your valuables and keep the images or movies of the things that you want coated below your residence insurance policies, in a fireproof lockbox. This will safeguard your files and make submitting a claim for the missing or destroyed products with your residence insurance firm, less complicated and quicker.
If you want to make positive that you are acquiring the ideal direction in searching for house owner's insurance policies, you have to commence with crucial methods and excellent ideas, which you can acquire proper from this report. You never have to lose out on a excellent coverage if you are utilizing the guidance in this article to get the proper policy when you need to have it.