text:
\section{Introduction}
Decision trees (DTs) and tree ensembles are widely used in practice, particularly for applications that require few parameters \cite{zharmagambetov2020smaller, kumar2017resource, zhu2020resot, shoaran2018energy, tanno2019adaptive}, fast inference \cite{carreira2018alternating, mathy2015boundary, zhu2021closed}, and good interpretability \cite{silva2020optimization, lundberg2017unified}. In a DT, the internal and leaf nodes are organized in a binary structure, with internal nodes defining the routing function and leaf nodes predicting the class label. Although DTs are easy to train by recursively splitting leaf nodes,  the tree structure can be suboptimal for the following reasons: (1) DTs can grow exponentially large as the depth of the tree increases. Yet, the root-leaf path can be short even for large DTs,  limiting the predictive power.
(2) In a DT, the nodes are not shared across different paths, reducing the efficiency of the model.

Decision trees are similar to neural networks (NNs) in that both models are composed of basic units. A possible way to enhance the performance of DTs or NNs is to replace the basic units with more powerful models. For instance, ``Network in Network'' builds micro NNs with complex structures within local receptive fields to achieve state-of-the-art performances on image recognition tasks \cite{lin2013network}. As for DTs, previous work replaced the axis-aligned splits with logistic regression or linear support vector machines to construct oblique trees \cite{carreira2018alternating, zharmagambetov2020smaller, norouzi2015efficient, kontschieder2015deep, zhu2020resot, hazimeh2020tree}. The work in \cite{tanno2019adaptive} further incorporates convolution operations into DTs for improved performance on image recognition tasks, while \cite{zharmagambetov2020smaller} replaces the leaf predictors with linear regression to improve the regression performance. Unlike the greedy training algorithms used for axis-aligned trees (e.g., Classification and Regression Trees or CART \cite{steinberg2009cart}), oblique trees are generally trained by gradient-based  \cite{kontschieder2015deep,zhu2020resot,hazimeh2020tree} or alternating  \cite{carreira2018alternating, zharmagambetov2020smaller} optimization algorithms.

Inspired by the concepts of Network in Network \cite{lin2013network} and oblique trees \cite{carreira2018alternating, norouzi2015efficient}, we propose a novel model, Tree in Tree (TnT), to recursively replace the internal and leaf nodes with micro decision trees. In contrast to a conventional tree structure, the nodes in a TnT form a Directed Acyclic Graph (DAG) to address the aforementioned limitations and construct a more efficient model. Unlike previous oblique trees that were optimized on a predefined tree structure \cite{tanno2019adaptive, zharmagambetov2020smaller}, TnT can learn graph connections from scratch. The major contributions of this work are as follows: \textit{(1) We extend decision trees to decision graphs and propose a scalable algorithm to construct large decision graphs. (2) We show that the proposed algorithm outperforms existing decision trees/graphs, either as a stand-alone classifier or base estimator in an ensemble,  under the same model complexity constraints. (3) Rather than relying on a predefined graph/tree structure, the proposed algorithm is capable of learning graph connections from scratch (i.e., starting from a single leaf node) and offers a fully interpretable decision process.} We provide a Python implementation of the proposed TnT decision graph at \url{https://github.com/BingzhaoZhu/TnTDecisionGraph}.

\setlength{\columnsep}{0.05\textwidth}
\begin{wrapfigure}{R}{0.55\textwidth}
\vspace{-2mm}
\begin{minipage}{0.55\textwidth}
  \begin{algorithm}[H]
    \SetAlgoLined
    $G \gets \text{initialize graph with a leaf node}$\;
    \For{$i\gets1$ \KwTo $N$}{
        \For{\text{each leaf node ($l_i$)} $\in G$}{
        Find the maximum gain ($g_i$) if we split $l_i$ \;
        }
        \For {$\text{each pair of leaf nodes ($l_i, l_j$)} \in G$}{
        Record gain ($g_{i,j}$) if we merge $l_i$ and $l_j$\;}
        Split/merge nodes to maximize gain\;
    }
    \scriptsize{Note: The split operation has a model complexity penalty ($C$) for creating an internal node.}
    \caption{Naive decision graph (NDG) \cite{oliver1992decision}}
  \end{algorithm}
\end{minipage}
\vspace{-3mm}
\end{wrapfigure}

\section{Related work}
\label{related}

Decision graph (DG) is a generalization of the conventional decision tree algorithm, extending the tree structure to a directed acyclic graph \cite{oliver1992decision, sudo2018efficient}. Despite similarity in using a sequential inference scheme, training and optimizing DGs is more challenging due to the large search space for the graph structure. The work in \cite{oliver1992decision} and \cite{Zighed2007} proposed a greedy algorithm to train DGs by tentatively joining pairs of leaf nodes at each training step (NDG, Algorithm 1).  \cite{benbouzid2012fast} constructed the DG as a Markov decision process, where base estimators were cascaded in the form of a data-dependent AdaBoost ensemble. \cite{platt1999large} combined multiple binary classifiers to construct a DAG for efficient multi-class classification.
While \cite{benbouzid2012fast} and \cite{platt1999large} constructed DAGs with specific settings (e.g., Adaboost  \cite{benbouzid2012fast} and multi-class classification \cite{platt1999large}), these methods do not provide a general extension of decision trees. \cite{shotton2013decision} improved NDG by jointly optimizing  split nodes and their connections to  next-layer nodes. However, simultaneously learning the split and branching structure has no exact solution and relies on search-based algorithms to reach local minimum.
Alternatively, in this work, we revisit the concept of decision graphs by exploiting recent advances in non-greedy tree optimization algorithms \cite{carreira2018alternating, norouzi2015efficient, hazimeh2020tree, lin2020generalized}. Our proposed Tree in Tree algorithm can construct DGs as a more accurate and efficient alternative to the widely-used decision trees, both as stand-alone classifiers and as weak learners in the ensembles.

Conventional decision tree learning algorithms such as CART \cite{steinberg2009cart} and its variations follow a greedy top-down growing scheme. Recent work has focused on optimizing the structure of the tree  \cite{lin2020generalized, zhu2020scalable, hu2019optimal}. However, constructing an optimal binary DT is NP-hard \cite{laurent1976constructing} and optimal trees are not scalable to large datasets with many samples and features \cite{lin2020generalized, zhu2020scalable, hu2019optimal}. Recent studies have further developed scalable algorithms for non-greedy decision tree optimization, with no guarantee on tree optimality \cite{carreira2018alternating,norouzi2015efficient,zharmagambetov2020smaller,zhu2020resot,kontschieder2015deep,hazimeh2020tree}. Such scalable approaches can be categorized into two groups: tree alternating optimization (TAO) \cite{carreira2018alternating,zharmagambetov2020smaller} and gradient-based optimization \cite{norouzi2015efficient, zhu2020resot,kontschieder2015deep,hazimeh2020tree}.

TAO decomposes the tree optimization problem into a set of reduced problems imposed at the node levels. The work in \cite{carreira2018alternating} applied the alternating optimization to both axis-aligned trees and sparse oblique trees. Later, \cite{zharmagambetov2020smaller} extended TAO to regression tasks and ensemble methods. Unlike TAO, gradient-based optimization requires a differentiable objective function, which can be obtained by different methods. For example, \cite{norouzi2015efficient} derived a convex-concave upper bound of the empirical loss. \cite{kontschieder2015deep} and \cite{zhu2020resot} considered a soft (i.e., probabilistic) split at the internal nodes and formulated a global objective function. The activation function for soft splits was refined in \cite{hazimeh2020tree} to enable conditional inference and parameter update. Both TAO and gradient-based optimization operate on a predefined tree structure and optimize the parameters of the internal nodes.

The proposed Tree in Tree algorithm aims to optimize the graph/tree structure by growing micro decision trees inside current nodes. Compared to the greedy top-down tree induction \cite{steinberg2009cart}, Tree in Tree solves a reduced optimization problem at each node, which is enabled via non-greedy tree alternating optimization techniques \cite{carreira2018alternating}. Compared to NDG, TnT employs a non-greedy process to construct decision graphs, which leads to an improved classification performance (discussed in later sections). Compared to axis-aligned decision trees (e.g., TAO \cite{carreira2018alternating, zharmagambetov2020smaller}, CART \cite{steinberg2009cart}), TnT extends the tree structure to a more accurate and compact directed acyclic graph, in which nodes are shared across multiple paths.

\vspace{-0mm}
\section{Methods} \vspace{-0mm}
In this work, we consider a classification task with input and output spaces denoted by $ \mathcal{X} \subset \mathbb{R}^{D}$ and $\mathcal{Y} = \{1,...,K\}$, respectively. Similar to conventional decision trees, a decision graph classifier $G$ consists of internal nodes and leaf nodes. Each internal node is assigned a binary split function $s(\cdot ; \theta): \mathcal{X} \rightarrow[left\_child,right\_child]$ parametrized by $\theta$, which defines the routing function of a graph. For axis-aligned splits, $\theta$ indicates a feature index and a threshold.
The terminal nodes (with no children) are named leaf nodes and indicate the class labels.

\vspace{-2mm}
\subsection{Decision graph}\vspace{-2mm}
As an extension to the tree structure, decision graphs organize the nodes into a more generic directed acyclic graph. In this work, we limit our discussion to axis-aligned binary DTs/DGs in which each internal node compares a feature value to a threshold to select one of the two child nodes. Similar to the sequential inference process in DTs, the test samples in a DG start from the root and successively select a path at the internal nodes until a leaf node is reached. The main differences between binary DTs and DGs are the following: (1) In DTs, each child node has one parent node. However, DGs allow multiple parent nodes to share the same child node. Therefore, DG can combine the nodes with similar behaviors (e.g., similar split functions) to reduce model complexity. (2) In binary DTs, the number of leaf nodes is always greater than the internal nodes by one. In DGs, however, $\# Leaves \leq \# Internals + 1 $, since multiple internal nodes can share the same leaf node. Furthermore, there exists a unique path to reach each leaf node in a tree structure, which does not hold within DGs. (3) The model complexity of a DT is often quantified by the number of internal or leaf nodes. However, we can post-process a DG by merging the leaf nodes with the same class label. As a result, DGs have a minimum leaf node count equal to the number of classes. Therefore, we use the number of splits (i.e., internal nodes) to quantify the  model complexity of a DG.

\begin{wrapfigure}{r}{0cm}
  \centering
  \raisebox{0pt}[\dimexpr\height-2.5\baselineskip\relax]{
  \includegraphics[width=7.1cm]{f1.png}}
  \caption{(a) The growing phase of TnT. The micro decision tree (in dashed box) replaces an internal node (dashed circle). Compared to a single node, the substitute micro tree can provide a more powerful split function. (b) The merging phase of TnT. We merge the fitted micro tree into the current structure to create a directed acyclic graph.}
  \vspace{-3mm}
  \label{f1}
\end{wrapfigure}

\vspace{-2mm}
\subsection{Tree in Tree}\vspace{-2mm}
We propose a novel algorithm named Tree in Tree as a scalable method to construct large decision graphs. Conventional DT training algorithms (e.g., CART) are greedy and recursively split the leaf nodes to grow a deep structure, without optimizing the previously learned split functions. \textit{The key difference between the proposed TnT model and conventional approaches lies in the optimization of the internal nodes. TnT fits new decision trees in place of the internal/leaf nodes and employs such micro DTs to construct a directed acyclic graph.} Overall, the proposed TnT model is a novel extension to the conventional decision trees and generates accurate predictions by routing samples through a directed acyclic graph.

Figure \ref{f1} shows the high-level procedure for training a decision graph with the proposed TnT algorithm. Assuming a starting decision graph (e.g., a decision tree or a single leaf node), our goal is to grow a larger model with improved predictive power. In the growing phase of TnT (Fig. \ref{f1}(a)), we replace a node (dashed circle) with a micro decision tree with multiple splits to enable more accurate decision boundaries. In the merging phase (Fig. \ref{f1}(b)), the micro decision tree is merged into the starting model to construct a TnT decision graph, in which a child node (node 2) may have multiple parent nodes (node 4 and 5).

\vspace{-2mm}
\paragraph{Growing the graph from internal nodes} We consider the training of a decision graph as an optimization problem with the aim of minimizing the loss function on the training data:
\begin{gather}
\label{eq1}
    \min \sum_{x,y \in \mathbb{X},\mathbb{Y}} L\left(y, G(x; \Theta\right)).
\end{gather}
TnT grows the decision graph $G(\cdot; \Theta)$ from an arbitrary internal node $n_i \in G$ with the split function $s(\cdot; \theta_i)$. $\theta_i$ denotes the trainable parameters of $n_i$ including a feature index and a threshold for axis-aligned splits. The overall goal is to replace $n_i$ with a decision tree $t_i$ and minimize the loss function as indicated in \eqref{eq1}. All other nodes remain unchanged as we train $t_i$.

Let us consider a subset of samples ($x_{subset}\in \mathbb{X}_{subset}, y_{subset}\in \mathbb{Y}_{subset}$) that is sensitive to the split function $s(\cdot; \theta_i)$, as defined by the following expression:
\begin{gather}
\label{eq2}
    G_{n_i\to left}(x_{subset}; \Theta \backslash \theta_i) \neq G_{n_i\to right}(x_{subset}; \Theta \backslash \theta_i),
\end{gather}
where $\Theta \backslash \theta_i$ denotes the parameters of all nodes in $G$ excluding $n_i$. Growing the graph from $n_i$ does not change $\Theta \backslash \theta_i$ since all other nodes are fixed as we solve the reduced optimization problem at $n_i$.
$G_{n_i\to left}$ sends the samples to the left child at $n_i$ (i.e., $s(\cdot;\theta_i)\to left\_child$) while $G_{n_i\to right}$  routes the samples to the right child at $n_i$. With $\Theta \backslash \theta_i$ being fixed, the output of decision graph only depends on $\theta_i$ (i.e., $s(\cdot;\theta_i)$)
\begin{gather}
\label{eq2.5}
G(x; \Theta)=
    \begin{cases}
        G_{n_i\to left}(x; \Theta \backslash \theta_i)  & \text{if } s(x;\theta_i)\to left\_child\\
        G_{n_i\to right}(x; \Theta \backslash \theta_i) & \text{if } s(x;\theta_i)\to right\_child.
    \end{cases}
\end{gather}
Since $L(y, G_{n_i\to left}(x; \Theta \backslash \theta_i)) \neq L(y, G_{n_i\to right}(x; \Theta \backslash \theta_i))$ only if the inequality \eqref{eq2} holds, we can solve the reduced optimization problem at node $n_i$ based on the subset ($\mathbb{X}_{subset}, \mathbb{Y}_{subset}$) instead of using the entire training set:
\begin{gather}
\label{eq3}
    \min_{\theta_i} \sum_{\substack{x\in\mathbb{X}_{subset}\\
                  y\in\mathbb{Y}_{subset}}} L(y, G(x; \Theta)).
\end{gather}
Having Equation \eqref{eq2.5}, the optimization problem \eqref{eq3} has a closed-form solution as follows:
\begin{gather}
\label{eq4}
t_{i}^{*}(x) \coloneqq
    \begin{cases}
      left\_child & \text{if } L(y, G_{n_i\to left}(x; \Theta \backslash \theta_i))<L(y, G_{n_i\to right}(x; \Theta \backslash \theta_i))\\
      right\_child & \text{if } L(y, G_{n_i\to right}(x; \Theta \backslash \theta_i))<L(y, G_{n_i\to left}(x; \Theta \backslash \theta_i)).
    \end{cases}
\end{gather}
Equation \eqref{eq4} defines the optimal split function at the internal node $n_i$ which is used to fit the micro decision tree $t_i$. With other nodes being fixed, we show that the overall loss function of $G$ can be minimized by pursuing the optimal spilt function at an arbitrary internal node $n_i$. Rather than using a simple axis-aligned split, the proposed TnT algorithm learns a complexity-constrained decision tree to better approximate the optimal split function (Equation \eqref{eq4}).

\vspace{-2mm}
\paragraph{Growing the graph from leaf nodes} Growing from the leaf nodes is a standard practice in greedy training algorithms, where we recursively split the leaf nodes to achieve a deeper tree with a better fit on the training data \cite{steinberg2009cart}. In TnT, we replace the leaf predictors with decision trees. Let $G(\cdot; \Theta)$ be a decision graph and $n_l \in G$ an arbitrary leaf node with a constant class label $l(\cdot;\theta_l)=c$. Our goal is to minimize the overall loss function $L\left(\mathbb{Y}, G(\mathbb{X}; \Theta\right))$ by replacing the leaf predictor $l(\cdot;\theta_l)$ with a micro decision tree $t_l(x)$.

Consider the subset of samples ($\mathbb{X}_{subset}, \mathbb{Y}_{subset}$) that visit the leaf node $n_l$. Minimization of the loss function (\ref{eq1}) can be expressed as
\begin{gather}
\label{eq5}
    \min_{\theta_l} \sum_{\substack{x\in\mathbb{X}_{subset}\\
                  y\in\mathbb{Y}_{subset}}}
                  L\left(y, t_l(x \right)),
\end{gather}
where the minimum is simply achieved at $t_l^*(x)\coloneqq y$ for $x,y \in \mathbb{X}_{subset}, \mathbb{Y}_{subset}$ (i.e., the ideal leaf predictor). We build a decision tree to approximate the ideal leaf predictor

\begin{algorithm}[b]
  \SetAlgoLined
  \KwData{Training set $\mathcal{X},\mathcal{Y}$}
  \KwResult{TnT decision graph $G$ fitted on the training set}
  \{$infer(n, \mathcal{X})$ denotes the forward inference of data $\mathcal{X}$ starting from node $n$\}\;
  \{Nodes are visited in the breadth-first order\}\;
  $G \gets \text{initialize graph with a leaf node}$\;
  \For{$i_1\gets1$ \KwTo $N_1$}{
    \For{$i_2\gets1$ \KwTo $N_2$}{
        \For{$\text{each node ($n_i$)} \in G$}{
            Samples that visit $n_i$: $\mathcal{X}_i,\mathcal{Y}_i \subset \mathcal{X},\mathcal{Y}$\;
            \uIf{$n_i$ is an internal node}{
                $\mathcal{Y}_{i,left} \gets infer(n_i.left\_child, \mathcal{X}_i)$\;
                $\mathcal{Y}_{i,right} \gets infer(n_i.right\_child, \mathcal{X}_i)$\;
                $index\_left \gets (\mathcal{Y}_{i}=\mathcal{Y}_{i,left} \textbf{ and } \mathcal{Y}_{i}\neq \mathcal{Y}_{i,right})$ \;
                $index\_right \gets (\mathcal{Y}_{i}=\mathcal{Y}_{i,right} \textbf{ and } \mathcal{Y}_{i}\neq \mathcal{Y}_{i,left})$ \;
                $\mathcal{X}_{subset}, \mathcal{Y}_{subset} \gets$ copy samples from $\mathcal{X}_{i},\mathcal{Y}_{i}$ at $(index\_left \textbf{ or } index\_right)$\;
                $\mathcal{Y}_{subset}[index\_left] \gets 0$,     $\mathcal{Y}_{subset}[index\_right] \gets 1$\;
            }
            \uElseIf{$n_i$ is a leaf node}{
                $\mathcal{X}_{subset} \gets \mathcal{X}_i$,    $\mathcal{Y}_{subset} \gets \mathcal{Y}_i$\;
                }
            Grow a micro tree $t_i$ on subset $\mathcal{X}_{subset}, \mathcal{Y}_{subset}$ in place of $n_i$\;
    }}
    Merge $t_i$ into the current decision graph $G$ for all nodes ($n_i \in G$)
    }
  \label{algorithm2}
  \caption{Tree in Tree (TnT)}
\end{algorithm}

\vspace{-2mm}
\subsection{Learning procedure} \vspace{-2mm}
Unlike the learning procedures in \cite{carreira2018alternating, zharmagambetov2020smaller} which require a predefined tree structure, our proposed TnT algorithm grows a decision graph from a single leaf node. The training of TnT decision graphs is an iterative process that follows a \textit{grow-merge-grow- $\cdots$-merge} alternation. Algorithm 2 shows the pseudocode to train a TnT decision graph. Lines 9-14 find the subset of data samples $\mathcal{X}_{subset},\mathcal{Y}_{subset}$ that is sensitive to the internal split functions or leaf predictors at each node, and grow micro decision trees. In the internal nodes, $\mathcal{Y}_{subset}$ represents binary labels for the left or right child (i.e., not the label of the training set). Line 17 grows micro decision trees according to the growing phase of the TnT. Line 18 merges the trees into the graph structure. In Algorithm 2, $N_1$ is the number of merging phases that micro trees are merged into the graph. $N_2$ is the number of rounds to grow and optimize micro trees, similar to the number of iterations in the tree alternating optimization algorithm \cite{carreira2018alternating, zharmagambetov2020smaller}.

\vspace{-2mm}
\paragraph{Regularization} Regularization is critical to limit model complexity and prevent overfitting of a decision tree and it is similarly required for TnT decision graphs. In the growing phase of a TnT (either from internal or leaf nodes), the subsets of samples $\mathcal{X}_{subset},\mathcal{Y}_{subset}$ at different nodes may have various sizes. Therefore, we need a robust regularization technique to operate across all nodes of the TnT and to train the micro decision trees without overfitting on small subsets. In this work, we propose to use the sample-weighted cost complexity pruning approach \cite{bradford1998pruning, kiran2017cost}. We prune micro decision trees by minimizing $R(t_i)+C_i|t_i|$, where $R(t_i)$ is the misclassification measurement and $|t_i|$ denotes the tree complexity. We calculate $R(t_i)$ using Gini impurity and measure $|t_i|$ by counting the number of splits \cite{steinberg2009cart}. $C_i$ is the sample-weighted regularization coefficient calculated by
\begin{gather}
\label{eq5}
    C_i = C \frac{\#\mathcal{X}}{\#\mathcal{X}_{subset,i}},
\end{gather}
where $\#\mathcal{X}_{subset,i}$ is the sample count of subset at node $n_i$. $C$ is a hyperparameter of the TnT and is used to control the pruning strength and tune the model complexity ($\#$ splits). For a smaller subset, we need to apply a stronger cost complexity pruning to prevent overfitting.

\begin{figure}[h]
  \centering
  \includegraphics[width=13.5cm]{f2.png}
  \vspace{-3mm}
  \caption{Comparison of DT and TnT decision graph on synthetic data; (a) A toy classification task with desired axis-aligned boundaries. $x_1, x_2$ and $t_{1}-t_{4}$ denote two features and four thresholds, respectively. Different markers represent binary class labels. (b) A decision tree requires at least six splits to classify the data. (c) A TnT decision graph only requires four binary splits on the same task.}
  \vspace{-2mm}
  \label{f2}
\end{figure}

\vspace{-2mm}
\paragraph{Fine-tune and post pruning} The TnT decision graphs are compatible with Tree Alternating Optimization (TAO \cite{carreira2018alternating}), previously proposed to optimize decision trees. We used TAO to fine-tune the TnT decision graphs, which led to slight improvements in classification accuracy. The pseudocode for  TnT fine-tune algorithm is provided in the supplementary materials. A post pruning process is further applied to TnT decision graphs to remove the dead nodes. A node is pruned if no training samples travel through that node. Post pruning can result in a more compact decision graph and reduce the number of splits without affecting the training accuracy.
\vspace{-2mm}
\paragraph{Time complexity} Compared to decision trees, decision graphs offer an enriched model structure, which increases the complexity of learning the graph structure. Previous work constructed decision graphs by tentatively merging two leaf nodes at each training step, with a time complexity of $O(N_{l}^{2})$, where $N_{l}$ is the number of leaf nodes \cite{oliver1992decision}. Since the proposed TnT algorithm generates new splits by growing micro decision trees inside the nodes, the dataset is initially sorted in $O(mklog(m))$ for $m$ samples and $k$ features. The time complexity for creating a new split depends on the dataset (i.e., $O(mk)$) and not on the size of the graph. As the graph grows larger, the  TnT algorithm  optimizes each node for $N_1*N_2$ times in the worst case (Algorithm 2). Since $N_1$ and $N_2$ are hyperparameters that were fixed in this work ($N_1=2, N_2=5$, the choice of $N_1$ and $N_2$ will be discussed in the following section), TnT exhibits a linear time complexity to the number of nodes, $O(nmk+mklog(m))$ with $n$ being the number of nodes. Testing our Python implementation on an Intel i7-9700 CPU, it took 325.3 seconds to build a TnT of 1k splits on the MNIST dataset (60k samples, 784 features, 10 classes).

\vspace{-2mm}
\paragraph{Synthetic data} We first construct a synthetic classification dataset to show the potential benefits of TnT over conventional decision tree algorithms (e.g., CART). Figure. \ref{f2}(a) visualizes the two-dimensional data distribution with one class on the corners and the other class elsewhere. To achieve optimal decision boundaries, a conventional decision tree requires six splits (Fig. \ref{f2}(b)), whereas TnT only requires four splits to generate the same decision boundaries (Fig. \ref{f2}(c)). \textit{By sharing nodes among different decision paths in a graph, TnT enables a more compact model with fewer splits compared to a conventional DT.}

\vspace{-0mm}
\section{Experiments: TnT as a stand-alone classifier}\vspace{-0mm}
We test the TnT decision graph as a stand-alone classifier and benchmark it against several state-of-the-art decision tree/graph algorithms with axis-aligned splits, including classification and regression trees (CART \cite{steinberg2009cart}), tree alternating optimization (TAO \cite{carreira2018alternating}), and the naive decision graph (NDG \cite{oliver1992decision}).
We did not include  decision jungles \cite{shotton2013decision} in our comparison since no implementation was provided by the authors. We also implemented the TnT algorithm in two different settings: with or without fine-tuning. We observed that the proposed TnT algorithm consistently achieves a superior performance under similar complexity constraints on multiple datasets.

\begin{figure}[h]
  \centering
  \includegraphics[width=14cm]{f4.png}
  \caption{(a) The number of splits as a function of the root-leaf path length. The standard deviation across different samples is shown by shaded areas. (b) The number of splits vs. regularization coefficient $C$. (c, d) Test performance using different hyperparameter settings on the MNIST dataset. The default setting ($N_1=2, N_2=5$) is plotted in both figures for comparison.}
  \label{f3}
\end{figure}

In the worst-case scenario, the number of nodes increases exponentially with the depth of a tree, which prevents DTs from growing very deep. However, this limitation does not apply to TnT decision graphs. Figure \ref{f3}(a) illustrates the average length of the root-leaf path as a function of model complexity for TnT and CART. With 1000 splits, the average decision depth of the best-first CART is 12.3, whereas the TnT decision graph has a mean depth of 27.3. In the best-first decision tree induction, we add the best split in each step to maximize the objective \cite{shi2007best}. \textit{Therefore, TnT can achieve a much ``deeper'' model without significantly increasing the number of splits.} The regularization coefficient $C$ is used to control the complexity of decision graphs in TnT. The number of splits decreases as we increase the pruning strength $C$ (Fig. \ref{f3}(b)). Figures \ref{f3}(c, d) compare the effect of different hyperparameter settings ($N_1, N_2$). \textit{We note that the proposed TnT decision graph is a superset of decision trees and that TnT can reduce to a DT learning algorithm under certain conditions.} With $N_1=1$, Algorithm 2 replaces a single leaf node with a decision tree, which is equivalent to training a CART with cost complexity pruning. In general, higher values of $N_1$ and $N_2$ can lead to a better classification performance. In the following experiments, we set the hyperparameters as $N_1=2, N_2=5$. A marginal improvement in classification performance can be obtained by increasing $N_1$ and $N_2$, at the cost of increased training time.

\begin{figure}
  \centering
  \includegraphics[width=14cm]{f5.png}

  \caption{Model comparison in terms of train and test accuracy on multiple classification tasks. The following axis-aligned decision trees/graphs are included: \textbf{TnT (this work):} We implement the proposed TnT decision graph at various complexity levels. Hyperparameters are fixed at $N_1=2, N_2=5$ on all tasks. \textbf{TnT (fine-tuned):} The alternating optimization algorithm is used to fine-tune the TnT. \textbf{TAO:} The tree alternating optimization algorithm is applied to axis-aligned decision trees \cite{zharmagambetov2019experimental}. \textbf{CART:} Classification and regression trees trained in a best-first manner to assess the optimal tree structure under certain complexity constraint \cite{steinberg2009cart,shi2007best}. \textbf{NDG:} The naive decision graph trained with Algorithm 1 \cite{oliver1992decision}. The complexity penalty is fixed at $C=3e-4$ on all tasks. Dataset statistics are indicated on top of each figure with  the following format ($\#$ Train/Test samples * $\#$ Features, $\#$~Classes). }
  \vspace{-3mm}
  \label{f4}
\end{figure}

\begin{table}[h]
  \caption{Comparison of TnT and CART at  optimal split count (\#S, determined by cross-validation). Mean test accuracy (±standard deviation) are calculated on 5 independent trials.}
  \label{table0}
  \centering
  \scalebox{0.83}{
  \begin{tabular}{lllllllll}
    \toprule
    \multirow{2}{*}{model} & \multicolumn{2}{c}{MNIST} & \multicolumn{2}{c}{Connect-4} & \multicolumn{2}{c}{Letter} &  \multicolumn{2}{c}{Optical recognition} \\ \cmidrule{2-9}
    & accuracy & \#S & accuracy & \#S & accuracy & \#S & accuracy & \#S \\
    \midrule
    TnT  & \textbf{90.87$\pm$0.31} & \textbf{600} & \textbf{78.85$\pm$0.46} & \textbf{864} & \textbf{86.62$\pm$0.02} & \textbf{1.2k} & \textbf{86.32$\pm$0.24} & \textbf{174}  \\
    CART & 88.59$\pm$0.14 & 1.1k & 77.23$\pm$0.01 & 931 & 86.26$\pm$0.15 & 1.3k & 85.56$\pm$0.46 & 193 \\
    \midrule
    \multirow{2}{*}{model} & \multicolumn{2}{c}{Pendigits} & \multicolumn{2}{c}{Protein} & \multicolumn{2}{c}{SenseIT} & \multicolumn{2}{c}{USPS}\\ \cmidrule{2-9}
    & accuracy & \#S & accuracy & \#S & accuracy & \#S & accuracy & \#S \\
    \midrule
    TnT  & \textbf{92.61$\pm$0.53} & \textbf{125} & \textbf{57.26} & \textbf{69} & \textbf{80.48$\pm$0.42} & \textbf{198} & \textbf{88.76$\pm$1.36} & \textbf{31} \\
    CART & 91.74$\pm$0.13 & 166 & 55.30 & 76 & 79.40& 345& 87.35$\pm$0.15 & 109 \\
    \bottomrule
  \end{tabular}
  }
\end{table}

Figure \ref{f4} compares the proposed TnT decision graphs with axis-aligned decision trees/graphs previously reported. We include the following datasets: MNIST, Connect-4, Letter, Optical reconstruction, Pendigits, Protein, SenseIT, and USPS from the UCI machine learning repository \cite{UC} and LIBSVM Dataset \cite{LIBSVM} under Creative Commons Attribution-Share Alike 3.0 license. The statistics of datasets including the number of train/test instances, number of attributes, and number of classes are shown in Fig. \ref{f4}. If a separate test set is not available for some tasks, we randomly partition 33\% of the entire data as test set. For all models, we repeat the training procedure five times with different random seeds. The mean classification accuracy is plotted in Fig. \ref{f4} with shaded area indicating the standard deviation across trials. The proposed Tree in Tree (TnT) algorithm outperforms axis-aligned decision trees such as TAO \cite{carreira2018alternating, zharmagambetov2019experimental} and CART \cite{steinberg2009cart}, as well as NDG which is also based on axis-aligned decision graphs \cite{oliver1992decision}. We also present the results for TnT(fine-tuned), which employs alternating optimization to fine-tune the TnT and slightly improve the classification performance.

The node-sharing mechanism in TnT effectively regularizes the growth of the graph and can increase test performance compared to greedy training of trees (e.g., CART). To show the reduction of variance in TnT decision graphs, we removed the complexity constraints on both models and selected the hyperparameters that achieve the highest cross-validation accuracy on the training set. Table \ref{table0} compares the proposed TnT method with greedy training of trees. Overall, TnT achieves a better test accuracy with reduced model complexity compared to CART.

\paragraph{Visualization} Similar to decision trees, TnT decision graphs enjoy a fully interpretable and visualizable decision process. Figures \ref{f5}(a-c) visualize the TnT decision graphs with 20, 129, and 1046 splits, respectively. We use different node colors to indicate the dominant class labels. A node will have a dominant class if most samples at that node belong to the same class. We show the nodes in blue if class labels are mixed (i.e., no class label contributes to greater than 50\% of the samples visiting that node). As the graph grows larger, TnT performs better on the MNIST dataset, achieving improved classification accuracy on both training and testing sets.

\begin{figure}[h]
  \centering
  \includegraphics[width=13cm]{f3.png}
  \vspace{-2mm}
  \caption{Visualization of TnT decision graphs at various complexity levels. (a) TnT with 20 internal nodes and 16 leaf nodes (train/test accuracy: 70.41\%/71.75\% on MNIST classification task). (b) 129 internals and 75 leaves (train/test accuracy: 85.54\%/85.49\%). (c) 1046 internals and 630 leaves (train/test accuracy: 96.04\%/90.56\%). Different node colors represent dominant class labels (more than 50\% of samples belong to the same class). Nodes are shown in blue if no dominant class is found.}
  \vspace{-3mm}
  \label{f5}
\end{figure}

\vspace{-0mm}
\section{Experiments: TnT in the ensemble}\vspace{-0mm}
Decision trees are widely used as base estimators in ensemble methods such as bagging and boosting. Random Forests apply a bagging technique to decision trees to reduce variance \cite{breiman2001random}, in which each base estimator is trained using a randomly drawn subset of data  with replacement \cite{breiman1996bagging}. As opposed to bagging, boosting is used as a bias reduction technique where base estimators are incrementally added to the ensemble to correct the previously misclassified samples. Popular implementations of the boosting methods include AdaBoost \cite{freund1997decision} and gradient boosting \cite{chen2016xgboost,ke2017lightgbm}. Both AdaBoost and bagging use classifiers as base estimators, whereas the gradient boosting methods require regressors \cite{chen2016xgboost, ke2017lightgbm}. Although we argue that the proposed TnT algorithm can be applied to regression tasks with a slight modification in the objectives, it is beyond the scope of this paper to demonstrate TnTs as regressors.

Here, we use the TnT decision graphs as base estimators in the bagging (TnT-bagging) and AdaBoost (TnT-AdaBoost) ensembles. \textit{Our goal is to replace decision trees with the proposed TnT classifiers in ensemble methods and compare the performance under various model complexity constraints.} The ensemble methods are implemented using the scikit-learn library in Python (under the 3-Clause BSD license) \cite{scikit-learn}. We change the ensemble complexity by tuning the number of base estimators ($\#$E) and the total number of splits (i.e., internal nodes, $\#$S).
For Random Forest  with 100 trees, we remove the limit on $\#$S. Thus, the trees are allowed to grow as large as possible to better fit on training data.
Note that  TnT  has additional hyperparameters such as $N_1$ and $N_2$. We set the hyperparameters as $N_1=2, N_2=5$ throughout the experiments so that the TnT and tree ensembles share a similar hyperparameter exploration space.

\begin{table}[h]
  \caption{Comparison of TnT-based ensembles with conventional random forest and AdaBoost. Mean train and test accuracy ($\pm$ standard deviation) are calculated across 5 independent trials. We tune the ensemble size ($\#$E, the number of base estimators) and split count ($\#$S) to change the complexity of the ensemble. Dataset statistics are given in the format: Dataset name ($\#$ Train/Test samples * $\#$ Features, $\#$ Classes). Six additional datasets are included in the supplementary materials.}
  \label{table1}
  \centering
  \scalebox{0.83}{
  \begin{tabular}{lllllllllll}
    \toprule
    model & & \#E & \#S & train &  test & & \#E & \#S & train &  test\\
    \midrule
    TnT-bagging & \multirow{14}{*}{ \rotatebox{90}{MNIST (60k/10k*784, 10)}}
      & 5 &  4.8k  & \textbf{97.46$\pm$0.16} & \textbf{93.65$\pm$0.24} &
     \multirow{14}{*}{ \rotatebox{90}{Connect-4 (45.3k/22.3k*126, 3)}}
      & 5 &  4.6k  & \textbf{84.42$\pm$0.19} & \textbf{80.61$\pm$0.18}\\
     Random Forest & & 5 & 4.8k & 96.55$\pm$0.36 & 92.31$\pm$0.57  &
      & 5 & 4.6k & 83.60$\pm$0.12 & 79.21$\pm$0.19  \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 5 &  640  & \textbf{90.26} & 88.38 &
     & 5 &  450  & \textbf{77.75$\pm$0.16} & \textbf{77.39$\pm$0.19}   \\
     AdaBoost & & 5 &  640 & 89.75 & \textbf{88.61} &
     & 5 &  450 & 77.28 & 76.74   \\ \cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging& & 10 &  9.6k  & \textbf{98.28$\pm$0.06} & \textbf{94.92$\pm$0.20} &
     & 10 &  9.2k  & \textbf{85.11$\pm$0.05} & \textbf{81.44$\pm$0.14}  \\
     Random Forest & & 10 & 9.6k & 97.44$\pm$0.18 & 93.64$\pm$0.38 &
     & 10 & 9.2k & 84.21$\pm$0.12 & 79.85$\pm$0.20   \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 10 &  1.4k  & \textbf{95.09$\pm$0.09} & \textbf{92.36$\pm$0.13} &
     & 10 &  940  & \textbf{80.10$\pm$0.23} & \textbf{78.94$\pm$0.29} \\
     AdaBoost & & 10 &  1.4k & 94.28 & 91.49 &
     & 10 &  940 & 79.69 & 78.37  \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 20 &  19.2k  & \textbf{98.64$\pm$0.06} & \textbf{95.57$\pm$0.14} &
     & 20 &  18.3k  & \textbf{85.66$\pm$0.12} & \textbf{81.93$\pm$0.13}  \\
     Random Forest & & 20 & 19.2k & 97.90$\pm$0.12 & 94.36$\pm$0.19 &
     & 20 & 18.3k & 84.57$\pm$0.08 & 80.39$\pm$0.09\\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 20 &  2.9k  & \textbf{98.03$\pm$0.11} & \textbf{94.49$\pm$0.21} &
     & 20 &  1.8k  & 82.46$\pm$0.41 & 80.53$\pm$0.50 \\
     AdaBoost & & 20 &  2.9k & 97.70 & 94.04 &
     & 20 &  1.8k & \textbf{82.77} & \textbf{81.14}\\ \cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 100 &  111k  & 99.09$\pm$0.03 & \textbf{96.11$\pm$0.09} &
     & 100 &  143k  & 88.44$\pm$0.07 & \textbf{82.84$\pm$0.02}  \\
     Random Forest & & 100 & 292k & \textbf{100} & 95.72$\pm$0.17 &
     & 100 & 718k & \textbf{100} & 82.33$\pm$0.10\\
    \bottomrule
  \end{tabular}
  }
\end{table}

Table 2 compares the performance of TnT ensembles with that of decision tree ensembles on two datasets. A complete comparison table on eight datasets is included in the supplementary materials. Since the bagging method can effectively reduce variance, we use large models (i.e., TnTs/decision trees with many splits) as the base estimator. On the contrary, TnTs/decision trees with few splits are used in the AdaBoost ensemble, given that boosting can decrease the bias error. According to Table 1, TnT-bagging is almost strictly better than Random Forest under the same model complexity constraints, indicating that TnT decision graphs outperform decision trees as base estimators. TnT-AdaBoost also outperforms AdaBoost in most cases, showing the advantage of TnT over decision trees. However, we observe a few exceptions in the TnT-AdaBoost vs. AdaBoost comparison, as weak learners with high bias (e.g., decision stumps) are also suitable for boosting ensembles. Overall, the TnT ensembles (TnT-bagging, TnT-AdaBoost) achieve a higher classification accuracy compared to decision trees when used in similar ensemble methods (Random Forest, AdaBoost).

\vspace{-0mm}
\section{Discussions}\vspace{-0mm}
\paragraph{Broader impact} Recently, the machine learning community has seen different variations of decision trees \cite{carreira2018alternating, zharmagambetov2020smaller, norouzi2015efficient, kontschieder2015deep, zhu2020resot, hazimeh2020tree,tanno2019adaptive}. In this paper, we present the TnT decision graph as a more accurate and efficient alternative to the conventional axis-aligned decision tree. However, the core idea of TnT (i.e., growing micro trees inside nodes) is generic and compatible with many existing algorithms. For example, linear-combination (oblique) splits can be easily incorporated into the proposed TnT framework. Specifically, we can grow oblique decision trees inside the nodes to construct an oblique TnT decision graph (Line 17 of Algorithm 2). In addition to oblique TnTs, the proposed TnT framework is also compatible with regression tasks. As suggested in \cite{zharmagambetov2020smaller}, we may grow decision tree regressors (rather than DT classifiers) inside the leaf nodes to construct TnT regressors, which remains as our future work. Overall, our results show the benefits of extending the tree structure to directed acyclic graphs, which may inspire other novel tree-structured models in the future.

\vspace{-2mm}
\paragraph{Limitations}
The proposed TnT decision graph is scalable to large datasets and has a linear time complexity to the number of nodes in the graph. However, the training of TnT is considerably slower than CART. The current TnT algorithm is implemented in Python. It takes about 5 minutes to construct a TnT decision graph with $\sim$1k splits on the MNIST classification task (train/test accuracy: 95.9\%/90.4\%). Training a CART with the same number of splits requires 12.6 seconds (train/test accuracy: 93.6\%/88.3\%). TnT has a natural disadvantage in terms of training time since each node is optimized multiple times (in this work $N_1*N_2=10$),  similar to other non-greedy tree optimization algorithms (e.g., 1-4 minutes for TAO \cite{carreira2018alternating}). The Python implementation may also contribute to the slow training, and we expect that the training time would significantly improve with an implementation in C. The longer training time prevents us from constructing large TnT-Adaboost ensembles that follow a sequential training process, where the training time increases linearly to the number of base estimators. We also observe that TnT decision graphs have longer decision paths compared to CART (Figure \ref{f3}(a)), which may raise a concern on increased inference time. Since TnT uses a pruning factor ($C$) to control the model complexity, we can not precisely control the number of nodes as in CART. In the experiments, we searched over a range of $C$ values to meet the model complexity constraints.

\vspace{-2mm}
\paragraph{Parallel implementation}
Algorithm 2 presents a sequential algorithm to construct TnT decision graphs by visiting the nodes in the breadth-first order. However, it is also possible to concurrently grow micro decision trees inside multiple nodes, which could lead to a parallel implementation of TnT. Specifically, only those nodes in the graph that are non-descendant of each other can be optimized in parallel. Parallel optimization is not applicable to the nodes on the same decision path, since the parent node optimization may alter the samples
visiting the child node. The parallel optimization of non-descendant nodes follows the separability condition of TAO \cite{carreira2018alternating,zharmagambetov2020smaller}. The separability condition also holds for the proposed TnT decision graph, enabling a parallel implementation.

\vspace{-0mm}
\section{Conclusion}\vspace{-0mm}
In this paper, we propose the Tree in Tree decision graph as an effective alternative to the widely used decision trees. Starting from a single leaf node, the TnT algorithm recursively grows decision trees to construct decision graphs, extending the tree structure to a more generic directed acyclic graph. We show that the TnT decision graph outperforms the axis-aligned decision trees on a number of benchmark datasets. We also incorporate TnT decision graphs into popular ensemble methods such as bagging and AdaBoost, and show that in practice, the ensembles could also benefit from using TnTs as base estimators. Our results suggest the use of decision graphs rather than conventional decision trees to achieve superior classification performance, which may potentially inspire other novel tree-structured models in the future.

\begin{ack}
We thank Dr Masoud Farivar from Google for his valuable feedback and comments on this manuscript. This work was partially supported by a Google faculty research award in machine learning.
\end{ack}

\medskip

\small

\bibliographystyle{unsrtnat}

\section{Pseudocode to fine-tune TnT decision graph}
We propoed the TnT algorithm to construct a decision graph from scratch. The TnT decision graph can be further fine-tuned using alternating optimization \cite{carreira2018alternating}. As opposed to TnT, TnT fine-tuning requires a predefined graph structure as input. A comparison between TnT and TnT(fine-tuned) is presented in Fig. 4, where TnT(fine-tuned) slightly improves both train and test accuracy. Algorithm A.1 shows the pseudocode to fine-tune TnT. Similar to Algorithm 2 in the main text, the TnT fine-tune algorithm also computes the subset $\mathbb{X}_{subset}, \mathbb{Y}_{subset}$ at each node. The hyperparameter $N$ is the number of rounds for TnT fine-tune and we fix $N=5$ for all experiments in Fig. 4.

\begin{algorithm}[H]
  \SetAlgoLined
  \KwData{Training set $\mathcal{X},\mathcal{Y}$}
  \KwIn{TnT decision graph $G$}
  \KwResult{TnT decision graph $G^\prime$ fine-tuned on $\mathcal{X},\mathcal{Y}$}
  \{$infer(n, \mathcal{X})$ denotes the forward inference of data $\mathcal{X}$ starting from node $n$\}\;
  \{Nodes are visited in the breadth-first order\}\;
    \For{$i\gets1$ \KwTo $N$}{
        \For{$\text{each node ($n_i$)} \in G$}{
            Samples that visit $n_i$: $\mathcal{X}_i,\mathcal{Y}_i \subset \mathcal{X},\mathcal{Y}$\;
            \uIf{$n_i$ is an internal node}{
                $\mathcal{Y}_{i,left} \gets infer(n_i.left\_child, \mathcal{X}_i)$\;
                $\mathcal{Y}_{i,right} \gets infer(n_i.right\_child, \mathcal{X}_i)$\;
                $index\_left \gets (\mathcal{Y}_{i}=\mathcal{Y}_{i,left} \textbf{ and } \mathcal{Y}_{i}\neq \mathcal{Y}_{i,right})$ \;
                $index\_right \gets (\mathcal{Y}_{i}=\mathcal{Y}_{i,right} \textbf{ and } \mathcal{Y}_{i}\neq \mathcal{Y}_{i,left})$ \;
                $\mathcal{X}_{subset}, \mathcal{Y}_{subset} \gets$ copy samples from $\mathcal{X}_{i},\mathcal{Y}_{i}$ at $index\_left \textbf{ or } index\_right$\;
                $\mathcal{Y}_{subset}[index\_left] \gets 0$,     $\mathcal{Y}_{subset}[index\_right] \gets 1$\;
                Update the split function of $n_i$ based on $\mathcal{X}_{subset}, \mathcal{Y}_{subset}$ \;
            }
            \uElseIf{$n_i$ is a leaf node}{
                $\mathcal{X}_{subset} \gets \mathcal{X}_i$,    $\mathcal{Y}_{subset} \gets \mathcal{Y}_i$\;
                Label the leaf $n_i$ as the dominant class of $\mathcal{Y}_{subset}$\;
            }
        }
    }
  \label{algorithm2}
  \caption{Tree in Tree fine-tune}
\end{algorithm}

\section{Hyperparameters of TnT}
The TnT algorithm has three hyperparameters. $N_1$ is the number of merging phases where we merge micro trees into the graph. $N_2$ is the number of rounds to grow and optimize micro trees. The choice of $N_1$ and $N_2$ reflects the trade-off between training time and classification performance. We empirically set $N_1=2, N_2=5$ for all experiments in this work. $C$ is the cost complexity pruning coefficient to tune the complexity of TnT decision graphs \cite{bradford1998pruning, kiran2017cost}. With greater $C$, TnT tends to have fewer splits. For example, Fig. 5 in the main text visualizes various model complexities with 20, 129 and 1046 splits, which is achieved with $C=1e-2$, $C=1e-3$ and $C=1e-4$, respectively.

Figure 4 in the main text plots the classification performance as a function of model complexity. We tuned $C$ to change the number of splits. For each dataset, we sampled 30 values of $C$ which are equally spaced on a log scale. The maximum and minimum values of $C$ are summarized in Table B.1.
\begin{table}[h]
    \centering
    \caption{The maximum and minimum values of $C$ on different datasets.}
    \scalebox{0.92}{
    \begin{tabular}{c|cccccccc}
    \toprule
         Dataset& MNIST & Connect-4 & Letter& Optical recognition& Pendigits& Protein & SenseIT & USPS \\ \midrule
         $C_{min}$ & 1e-4 & 6e-5 & 5e-5 & 3e-4 & 5e-4 & 8e-4 & 3e-4 & 8e-4\\
         $C_{max}$ & 5e-2 & 1e-2 & 2e-2 & 6e-2 & 1e-1 & 1e-2 & 1e-2 & 3e-2\\
         \bottomrule
    \end{tabular}
    }
\end{table}

In addition to using TnTs as stand-alone classifiers, we combine TnT decision graphs with ensemble methods and present TnT-bagging and TnT-AdaBoost. Additional hyperparameters are introduced to TnT-bagging and TnT-AdaBoost by the ensemble methods. In this work, we tuned the number of base estimators and the total number of splits to change the ensemble complexity. For the bagging ensemble, we randomly draw samples from the training set with replacement to train each base estimator. We set \textit{max$\_$samples} to 1.0 and \textit{bootstrap$\_$features=False} for both Random Forest and TnT-bagging. For the AdaBoost ensemble, we used the \textit{``SAMME''} algorithm with a learning rate of 1.0 to build both AdaBoost and TnT-AdaBoost. Both ensemble methods were implemented using the scikit-learn library in Python \cite{scikit-learn}.

\section{Comparison of TnT and DT ensembles}
Table C.1 is similar to Table 2 in the main text but includes additional datasets. A summary on model comparison is given in the last two rows. The results show that both bagging and AdaBoost ensembles benefit from using the TnT as a base estimator.

\begin{table}[!htbp]
  \caption{Comparison of TnT ensembles with random forest and AdaBoost. Mean train and test accuracy ($\pm$ standard deviation) is calculated across 5 independent trials. We tune the ensemble size ($\#$E, the number of base estimators) and split count ($\#$S) to change the complexity of the ensembles. Dataset statistics are given in the format: Dataset name ($\#$ Train/Test samples * $\#$ Features, $\#$ Classes). }
  \label{table1}
  \centering
  \scalebox{0.75}{
  \begin{tabular}{lllllllllll}
    \toprule
    model & & \#E & \#S & train &  test & & \#E & \#S & train &  test\\
    \midrule
    TnT-bagging & \multirow{14}{*}{ \rotatebox{90}{MNIST (60k/10k*784, 10)}}
      & 5 &  4.8k  & \textbf{97.46$\pm$0.16} & \textbf{93.65$\pm$0.24} &
     \multirow{14}{*}{ \rotatebox{90}{Connect-4 (45.3k/22.3k*126, 3)}}
      & 5 &  4.6k  & \textbf{84.42$\pm$0.19} & \textbf{80.61$\pm$0.18}\\
     Random Forest & & 5 & 4.8k & 96.55$\pm$0.36 & 92.31$\pm$0.57  &
      & 5 & 4.6k & 83.60$\pm$0.12 & 79.21$\pm$0.19  \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 5 &  640  & \textbf{90.26} & 88.38 &
     & 5 &  450  & \textbf{77.75$\pm$0.16} & \textbf{77.39$\pm$0.19}   \\
     AdaBoost & & 5 &  640 & 89.75 & \textbf{88.61} &
     & 5 &  450 & 77.28 & 76.74   \\ \cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging& & 10 &  9.6k  & \textbf{98.28$\pm$0.06} & \textbf{94.92$\pm$0.20} &
     & 10 &  9.2k  & \textbf{85.11$\pm$0.05} & \textbf{81.44$\pm$0.14}  \\
     Random Forest & & 10 & 9.6k & 97.44$\pm$0.18 & 93.64$\pm$0.38 &
     & 10 & 9.2k & 84.21$\pm$0.12 & 79.85$\pm$0.20   \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 10 &  1.4k  & \textbf{95.09$\pm$0.09} & \textbf{92.36$\pm$0.13} &
     & 10 &  940  & \textbf{80.10$\pm$0.23} & \textbf{78.94$\pm$0.29} \\
     AdaBoost & & 10 &  1.4k & 94.28 & 91.49 &
     & 10 &  940 & 79.69 & 78.37  \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 20 &  19.2k  & \textbf{98.64$\pm$0.06} & \textbf{95.57$\pm$0.14} &
     & 20 &  18.3k  & \textbf{85.66$\pm$0.12} & \textbf{81.93$\pm$0.13}  \\
     Random Forest & & 20 & 19.2k & 97.90$\pm$0.12 & 94.36$\pm$0.19 &
     & 20 & 18.3k & 84.57$\pm$0.08 & 80.39$\pm$0.09\\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 20 &  2.9k  & \textbf{98.03$\pm$0.11} & \textbf{94.49$\pm$0.21} &
     & 20 &  1.8k  & 82.46$\pm$0.41 & 80.53$\pm$0.50 \\
     AdaBoost & & 20 &  2.9k & 97.70 & 94.04 &
     & 20 &  1.8k & \textbf{82.77} & \textbf{81.14}\\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 100 &  111k  & 99.09$\pm$0.03 & \textbf{96.11$\pm$0.09} &
     & 100 &  143k  & 88.44$\pm$0.07 & \textbf{82.84$\pm$0.02}  \\
     Random Forest & & 100 & 292k & \textbf{100} & 95.72$\pm$0.17 &
     & 100 & 718k & \textbf{100} & 82.33$\pm$0.10\\

    \midrule
    TnT-bagging & \multirow{14}{*}{ \rotatebox{90}{Letter (13.4k/6.6k*16, 26)}}
      & 5 &  5.3k  & 98.08$\pm$0.12 & \textbf{89.97$\pm$0.37} &
     \multirow{14}{*}{ \rotatebox{90}{Optical recognition (3.8k/1.8k*64, 10)}}
      & 5 &  890  & \textbf{99.48} & 90.45$\pm$1.24\\
     Random Forest & & 5 & 5.3k & \textbf{98.16$\pm$0.11} & 89.93$\pm$0.25  &
      & 5 & 890 & 99.38$\pm$0.11 & \textbf{90.46$\pm$0.91} \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 5 &  440  & \textbf{74.51$\pm$0.83} & \textbf{73.58$\pm$0.63} &
     & 5 &  200  & \textbf{96.74$\pm$0.29} & \textbf{88.31$\pm$0.61}   \\
     AdaBoost & & 5 &  440 &  73.40 & 71.38 &
     & 5 &  200 & 96.73 & 87.87  \\ \cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging& & 10 &  10.6k  & \textbf{99.16$\pm$0.10} & \textbf{92.35$\pm$0.15} &
     & 10 &  1.8k  & \textbf{99.83} & \textbf{92.41$\pm$0.51}  \\
     Random Forest & & 10 & 10.6k & 99.10$\pm$0.08 & 91.92$\pm$0.33 &
     & 10 & 1.8k & 99.79$\pm$0.10 & 92.23$\pm$0.37   \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 10 & 900  & \textbf{82.90$\pm$0.38} & \textbf{80.02$\pm$0.33} &
     & 10 &  420  & \textbf{99.81$\pm$0.06} &92.87$\pm$0.65 \\
     AdaBoost & & 10 & 900 & 81.10 & 78.09 &
     & 10 &  420 & 99.58 & \textbf{92.92$\pm$0.02} \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 20 &  21.3k  & \textbf{99.57$\pm$0.04} & \textbf{93.35$\pm$0.19} &
     & 20 &  3.6k  & \textbf{99.91} & \textbf{92.93$\pm$0.41}  \\
     Random Forest & & 20 & 21.3k & 99.33$\pm$0.03 & 92.85$\pm$0.21 &
     & 20 & 3.6k & 99.84$\pm$0.06 & 92.78$\pm$0.23 \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 20 &  1.8k  & \textbf{90.89$\pm$0.67} & \textbf{85.33$\pm$0.56} &
     & 20 &  820  & \textbf{99.99$\pm$0.01} & \textbf{94.52$\pm$0.55} \\
     AdaBoost & & 20 & 1.8k & 89.84 & 84.75 &
     & 20 &  820 & 99.97 &94.50$\pm$0.02 \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 100 &  108k  & 99.78$\pm$0.02 & \textbf{94.37$\pm$0.03} &
     & 100 &  18k  & 99.93$\pm$0.03 & \textbf{93.62$\pm$0.17}  \\
     Random Forest & & 100 & 136k & \textbf{100} & 94.29$\pm$0.07 &
     & 100 & 19k & \textbf{100} & 93.37$\pm$0.24\\

    \midrule
    TnT-bagging & \multirow{14}{*}{ \rotatebox{90}{Pendigits (7.5k/3.5k*16, 10)}}
      & 5 &  570  & \textbf{99.32$\pm$0.11} & \textbf{94.12$\pm$0.27} &
     \multirow{14}{*}{ \rotatebox{90}{Protein (11.9k/5.9k*357, 3)}}
      & 5 &  1.4k  & \textbf{77.05$\pm$0.58} & \textbf{59.59$\pm$0.62} \\
     Random Forest & & 5 & 570 & 98.86$\pm$0.12 & 92.77$\pm$0.41 &
      & 5 & 1.4k & 77.30$\pm$0.53 & 59.67$\pm$0.33  \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 5 &  200  & \textbf{98.53$\pm$0.14} & \textbf{93.24$\pm$0.62} &
     & 5 &  140  & \textbf{63.99} & \textbf{59.29}  \\
     AdaBoost & & 5 &  200 & 97.66 & 92.31 &
     & 5 &  140 & 62.43 & 58.45   \\ \cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging& & 10 &  1.1k  & \textbf{99.54$\pm$0.10} & \textbf{94.81$\pm$0.19} &
     & 10 &  2.7k  & 80.87$\pm$0.40 & \textbf{62.75$\pm$0.25}  \\
     Random Forest & & 10 & 1.1k & 99.01$\pm$0.13 & 93.47$\pm$0.33 &
     & 10 & 2.7k & \textbf{80.88$\pm$0.28} & 62.60$\pm$0.33   \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 10 &  410  & 99.52$\pm$0.22 & \textbf{94.83$\pm$0.21} &
     & 10 &  270  & \textbf{67.47} & \textbf{61.16} \\
     AdaBoost & & 10 &  410 & \textbf{99.65} & 94.75$\pm$0.02 &
     & 10 &  270 & 66.76 & 60.92  \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 20 &  2.3k  & \textbf{99.61$\pm$0.05} & \textbf{95.48$\pm$0.16} &
     & 20 &  5.4k  & \textbf{83.20$\pm$0.47} & \textbf{64.44$\pm$0.44}  \\
     Random Forest & & 20 & 2.3k & 99.16$\pm$0.10 & 93.71$\pm$0.24 &
     & 20 & 5.4k & 82.82$\pm$0.24 & 64.06$\pm$0.20 \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 20 & 820  & 100 & 96.35$\pm$0.30 &
     & 20 &  580  & \textbf{73.15} & 62.92 \\
     AdaBoost & & 20 &  820 & 100 & \textbf{96.63} &
     & 20 &  580 &  72.03 & \textbf{64.03} \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 100 &  11k  & 99.69$\pm$0.04 & \textbf{95.69$\pm$0.16} &
     & 100 &  0.3k  & 86.71$\pm$0.21 & \textbf{66.63$\pm$0.30}  \\
     Random Forest & & 100 & 20k & \textbf{100} & 95.31$\pm$0.22 &
     & 100 & 1.5k & \textbf{100} & 66.34$\pm$0.09\\

    \midrule
    TnT-bagging & \multirow{14}{*}{ \rotatebox{90}{SenseIT (78.8k/19.7k*100, 3)}}
      & 5 &  910  & \textbf{83.92$\pm$0.12} & \textbf{82.27$\pm$0.12} &
     \multirow{14}{*}{ \rotatebox{90}{USPS (7.3k/2k*256, 2)}}
      & 5 &  540  & \textbf{98.44$\pm$0.13} & \textbf{91.29$\pm$0.34}\\
     Random Forest & & 5 & 910 & 83.06$\pm$0.18  & 80.95$\pm$0.31  &
      & 5 & 540 & 97.27$\pm$0.16 & 90.06$\pm$0.39  \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 5 &  110  & \textbf{77.98} & \textbf{77.47} &
     & 5 &  160  & \textbf{99.07} & \textbf{91.73}  \\
     AdaBoost & & 5 &  110 & 77.83 & 77.03 &
     & 5 &  160 & 97.63 & 90.53   \\ \cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging& & 10 &  1.8k  & \textbf{84.52$\pm$0.08} & \textbf{82.87$\pm$0.20} &
     & 10 &  1.1k  & \textbf{98.75$\pm$0.06} & \textbf{91.90$\pm$0.16}  \\
     Random Forest & & 10 & 1.8k & 83.48$\pm$0.18 & 81.41$\pm$0.22 &
     & 10 & 1.1k & 97.85$\pm$0.19 & 90.53$\pm$0.26   \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 10 &  170  & \textbf{79.06} & \textbf{78.46} &
     & 10 &  350  & \textbf{100} & \textbf{92.83} \\
     AdaBoost & & 10 &  170 &  78.82 & 78.21 &
     & 10 &  350 & 99.95 & 92.50$\pm$0.40  \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 20 &  3.6k  & \textbf{84.88$\pm$0.03} & \textbf{83.19$\pm$0.13} &
     & 20 &  2.2k  & \textbf{99.20$\pm$0.09} & \textbf{92.72$\pm$0.39} \\
     Random Forest & & 20 & 3.6k & 83.77$\pm$0.15 & 81.64$\pm$0.19 &
     & 20 & 2.2k & 98.16$\pm$0.04 & 91.29$\pm$0.42  \\\cmidrule{3-6} \cmidrule{8-11}
     TnT-AdaBoost & & 20 &  280  & \textbf{80.00} & 79.18 &
     & 20 &  740  & 100 & \textbf{94.37} \\
     AdaBoost & & 20 &  280 & 79.96 & \textbf{79.19} &
     & 20 &  740 & 100 & 94.03$\pm$0.25 \\\cmidrule{3-6} \cmidrule{8-11}

     TnT-bagging & & 100 &  116k  & 90.92$\pm$0.02 & \textbf{84.09$\pm$0.09} &
     & 100 &  11k  & 99.29$\pm$0.05 & \textbf{93.18$\pm$0.28}  \\
     Random Forest & & 100 & 590k & \textbf{99.98} & 83.83$\pm$0.11 &
     & 100 & 24k & \textbf{100} & 92.67$\pm$0.28\\

     \midrule
     \multirow{2}{*}{Summary} & \multicolumn{3}{c}{\textbf{TnT-bagging wins}} & \textbf{test accuracy: 31} & & \multicolumn{3}{c}{Random Forest wins}  & test accuracy: 1\\ \cmidrule{2-11}
     &\multicolumn{3}{c}{\textbf{TnT-AdaBoost wins}}  & \textbf{test accuracy: 18} & & \multicolumn{3}{c}{AdaBoost wins}  & test accuracy: 6\\
    \bottomrule
  \end{tabular}
  }
\end{table}

\bibliographystyle{unsrtnat}

redpajama_set_name: RedPajamaArXiv


text:
Massive Sam Darnold what-if could drastically change the Jets' future
What if Sam Darnold appears in those last two games, as he did Sunday in Los Angeles?
What if the Jets' vicious third-year quarterback is as confident and decisive as he was in the shocking 23-20 victory over the Rams?
What if Darnold throws 300 yards and three touchdowns and doesn't turn the ball over and the Jets beat the 10-4 Browns in their home final Sunday at MetLife Stadium?
What if he follows it up by going up to Foxborough, Mass., Next week for the season finale and getting Bill Belichick's Patriots defense to see ghosts?
What if Darnold's 2020 resume is limited by three consecutive strong, winning performances – even though they came at the end of an otherwise miserable 3-13 season?
They would have been three meaningless victories for the Jets, but would they be meaningless performances by Darnold?
Before Ramsen's anger, the assumption that the jets were destined to finish 0-16 was guaranteeing the overall overall draw choice and the chance to pick Clemson quarterback Trevor Lawrence.
With Lawrence, the supposed "generational talent" staring them in the face in April, the Jets wanted to decide what to do with Darnold. They would move on. Darnold would be traded for what the Jets could get for a third-year quarterback with a 12-24 record and 44 turnovers.
But what now?
Barring an unexpected stumble in the next two games (against the Bears and Colts) by the 1-13 Jaguars, who by virtue of the Jets win in LA and Jacksonville, who own tiebreaker for No. 1 pick due to the strength of the schedule, they get Lawrence .
Corey Sipkin
Now the Jets need to ask themselves this question when this season is over: Is it better to build around Darnold (who they've done a bad job over the last three years) or roll the dice on a college quarterback named Trevor Lawrence ?
Ohio State's Justin Fields looked ordinary Saturday against Northwestern and lost his stock. BYU's Zach Wilson looked dynamic in a pointless bowling game against a pedestrian UCF team on Tuesday night. Is Florida's Kyle Trask the answer? North Dakota State's Trey Lance?
What if Darnold shines this Sunday and next?
Does it change the narrative, or will it simply be dismissed as garbage time?
The Jets, remember, finished last season 6-2 after a 1-7 start. What turned out to be when the 2020 season began? Fool's gold.
"Every game is important for evaluation in the NFL," former Jets general manager Mike Tannenbaum told The Post on Wednesday. "Look, they have to make some important decisions in the offseason – especially from the quarterback. The big question that many people have about Sam is revenue. Right now, Sam has 44 turnovers since being in the NFL. It is the sixth most of any quarterback this season.
"It's not unlike Daniel Jones," Tannenbaum continued, referring to the Giants' second-year quarterback with turnover issues. "When the young quarterbacks do not turn the ball around, they both give their team a fair chance to win. We have seen it consistently for both teams. "
Strange stats over Darnold, though: Of his 36 career starts, in the 14 he hasn't thrown a bout, the Jets are only 3-11.
It's a result of too many bad teams around Darnold, and that raises the question: Have the Jets failed Darnold more than Darnold has failed the Jets?
The answer to this question is: Yes.
However, that does not exempt Darnold from the fact that he has not been special, has not elevated the players around him that a No. 3 overall draft should choose.
Sunday in LA, however, Darnold did. His quick and correct decision making was the difference in the game. Was it a flash in the pan, a push on the radar? Or a sign that he's starting to figure it out?
"I thought Sam looked really good on Sunday against the rams," said Tannenbaum. "He was crucial and accurate. It was a very encouraging achievement. "
It was only the fifth game this season that he had all three of his best receivers – Jamison Crowder, Breshad Perriman and Denzel Mims – on the field at the same time.
Head coach Adam Gase was asked on Wednesday what the "next step" is for Darnold.
"Just keep playing games like that," Gase said, referring to the Rams game.
Asked why it has not yet clicked on Darnold, three years into his career, Gase said: "A little bit has to do with who is around him and the consistency there. You see some of these quarterbacks play with the same receivers and play with the same skilled guys. It would be a good start for him – to have some connection with the list. "
The question, of course, is what list will Darnold have in 2021?
Tags Change, Darnold, drastically, football, future, Jets, massive, new york jetfly, Sam, sam darnold, sport, whatif

redpajama_set_name: RedPajamaCommonCrawl


text:
National Underwriter Acquires INSUR'ED
September 23, 2004 at 08:00 PM Share & Print
NU Online News Service, Sept. 23, 2004, 5:48 p.m. EDT
The National Underwriter Company, a unit of Highline Media L.L.C., announced today that it has acquired Insur'Ed Inc., a nationwide insurance training organization.[@@]
Terms of the transaction were not disclosed. Insur'Ed specializes in life-health and property-casualty pre-licensing, continuing education, sales and product training for banks, insurance companies and brokerage firms.
The National Underwriter Company, based in Erlanger, Ky., is the nation's leading insurance publishing, information services, and training content provider for the insurance industry.
Highline Media was formed in October 2003 by Spire Capital Partners L.P. and Andrew L. Goodenough, its president and chief executive officer.
Highline's intent is to use NUCO as a base from which to build a multi-market, multi-product publishing, training and information services business, including a combination of magazine, data, newsletter, conference, trade show, advanced-level training materials, and Web site content delivered in print, electronic, and live event form.
NIA Acquires Hamilton Green
Bart Shachnow, the founder and president of Insur'Ed, will be joining National Underwriter as executive director, reporting to Jim Keefe, vice president and managing director of the Professional Publishing Division.
"The Insur'Ed acquisition furthers our reach into the continuing education market and provides a platform for growth in the pre-licensing market," Keefe says. "We are absolutely delighted to have Bart and the Insur'Ed team join us. The company and its people are an excellent compliment to our core strength."
Shachnow says, "As a customer of National Underwriter products, training materials and publications for over 15 years, I believe NUCO's products, services, and resources are uniquely suited to enhancing the skill sets, expertise, and productivity of our ultimate end users in the financial industry.
"I look forward to working with NUCO to develop its pre-licensing, training and continuing education businesses." Shachnow also will be responsible for growing NUCO's corporate relationships.
Shachnow is the author of several insurance textbooks, including the Plain Language Guides to life and health insurance pre-licensing preparation, also acquired by NUCO.
The Insur'Ed staff will be relocating to NUCO's Hoboken offices in October.


text:
Scholarship & Innovation
M.Ed. in Educational Leadership
San Diego Teacher Residency
New School Creation Fellowship
Professional Learning Home
PBL Design Camp
Customized Residencies
PBL Leadership Academy
Research Center Home
Care Network
College Access Network
Protocol Library
Change Ideas
IRB Process
Financial Aid & Funding
Institutional Reporting
OUR RESPONSE TO RACIAL INJUSTICE
Stacey Lopaz
Director, New School Creation Fellowship
slopaz@hightechhigh.org
"In my role, I have the opportunity to support innovative and disruptive leaders around the country who are opening new schools rooted in equity and deeper learning. I am inspired constantly by their passion, tenacity, and willingness to take risks in service of changing the status quo in the communities they serve."
Stacey's passion for progressive education came from her time spent as an elementary school teacher in New Orleans, as part of her work with Teach for America from 2002 to 2004. Since then, Stacey has been constantly engaged in the study and practice of project-based learning as a vehicle for equity, across all grades. Stacey has been the founding teacher and director at two different HTH schools, K-8.
Stacey has had a long history working with the High Tech High GSE. From 2007 to 2010, she was one of the founding members of High Tech High's Teacher Leadership and School Leadership graduate programs, earning a Master's degree in School Leadership. Since then, Stacey has been part of the adjunct faculty at the GSE, facilitating the Design and Leadership for Deeper Learning course.
Much of Stacey's work in education has centered around supporting school transformation. Stacey spent two years working as a School Development Coach for New Tech Network, where she worked with several elementary schools around the country, helping them reimagine the scope of their work. She currently works as the Co-Director of the GSE's New School Creation Fellowship.
Racial Injustice Statement
Funding & Financial Aid
HTH Graduate School of Education
2150 Cushing Road
© High Tech High Graduate School of Education 2008 - 2020. All rights reserved.


Hi everyone! I hope you had a reflective, enjoyable Halloween/Samhain, and a happy Bonfire Night if you're in the UK. Jon and I went out with friends to see fireworks and eat hot dogs on Bonfire Night. We didn't do anything for it last year, so this was my first time celebrating!
And during the first six days of this month, I was happily reliving the memories of getting my visa and coming home to England in 2016.
I can't believe it's been TWO YEARS already! Those memories are so fresh in my mind that I feel my heart start to pound in the anticipation I'd felt then.
November 1st – That night, I received the email from UKVI saying my passport and documents had returned from processing and were ready for pickup.
November 2nd – I went on what felt like the longest subway ride of my life to get downtown to the UKVI office. I was terrified of what might lie inside that big envelope. But, of course, I'd been approved and there was a beautiful visa inside my passport. I got home and Jon and I booked my flight to England.
November 3rd – Reducing my life to two suitcases and a backpack was surprisingly difficult and stressful! Many panic attacks were had!
November 4th – I did some Important Things such as close my bank account, finish packing and organize everything else into storage, checking in, printing off my flight itinerary, and saying goodbye to my beloved friends.
November 5th – I got an Uber down to Pearson International and boarded my evening flight to London.
November 6th – I landed at Gatwick in the early morning, waited for Jon to come meet me, and then we took the bus to Leicester. After we fetched Jon's car we drove to Coalville, had a celebratory dinner together, then went home to Ravenstone.
Memories have sweetened and life moves ever on. I don't know what the future holds, but I can say right now that I would be happy living the rest of my life in the UK. So here's to many more years here in my beautiful home!
This entry was posted in Life in England and tagged art, england, immigration.


text:
Stockton police officer charged with elder theft, forgery, credit card fraud
20131016 — Three new officers were sworn into the Stockton Police Department by Chief Jones on Wednesday, October 16. They also announced the recruitment of two new police officers in training. Chief Jones can be seen here with Ny Tran.Transmission Reference: REC1310161756383127
Stockton police officer Ny Tran wore a gray suit during a short hearing, in which he was the defendant.
The officer is charged with elder theft, grand larceny, identity theft, fraudulent use of a credit card and three counts of forgery. Tran was arrested last week but released on bail.
He remains on administrative leave.
Tran's attorney, Chris Walsh, asked San Joaquin County Judge Ronald Northup to object to news outlets showing his client's face. The lawyer said he got discovery on Monday and couldn't see through it.
Northup rejected Walsh's request.
A protection order was issued and given to the judge. Tran must obey the order and stay away from the victim, whose name was kept confidential during the court hearing, and continue the investigation pending trial.
"We're going to look closely at the evidence. Obviously, if he did something wrong, they're going to be responsible for it," Walsh said. "In the meantime, I would like to ask people not to prejudge this case and let the trial run smoothly. He is presumed innocent until proven guilty. Even though he is a police officer, he has a right to be."
District Attorney Tori Verber Salazar held a press conference after the hearing to discuss the allegations.
There is one victim in the case who is under guardianship and in his early eighties. The victim suffers "from some form of disability," said Verber Salazar.
"My office… received a tip about the elder fraud line from this victim's state-appointed conservator," she said. "The trustee finds checks written to the defendant in large sums of money, in addition to frequent credit card payments totaling nearly $50,000."
Verber Salazar said her office has been investigating Tran for the past few months. On Thursday, the San Joaquin County District Attorney's office filed a criminal complaint.
Tran has been with the Stockton Police Department since October 2013.
"Like you, I am at my wits end. Like you, I am disappointed," said Verber Salazar.
Verber Salazar urged the community to come forward if they have experienced something similar. She said the office "received information" this weekend about other possible police investigations.
"At this time, my office has opened the elder fraud tip line in case new victims come forward," she said.
Tran is scheduled to appear in court on December 13.
Record reporter Angelaydet Rocha covers community news in Stockton and San Joaquin County. She can be reached at [email protected] or on Twitter @AngelaydetRocha. Support local news, subscribe to The Stockton Record at https://www.recordnet.com/subscribenow.
This article originally appeared on The Record: Officer Tran will appear in court on Monday
Canada captain Sophie de Goede up for World Rugby Women's 15s Player of the Year & More Trending News
A mobile game that adds a pretty girl to strategy, has established itself as a popular genre
AMD Ryzen 7000 CPU prices slashed even further
Technology-driven hedge fund Two Sigma extends lease...
Community in shock as Greensburg police chief's...
Arraignment date set for officers charged with fatally...
NFC Championship Game: San Francisco 49ers v...
BoilerUpload – Quick Wrap – #1 Purdue 77 vs Michigan...


text:
Immunophenotyping of sheep paraffin-embedded peripheral lymph nodes
Melzi, E., Rocchi, M. S., Entrican, G., Caporale, M. and Palmarini, M. (2018) Immunophenotyping of sheep paraffin-embedded peripheral lymph nodes. Frontiers in Immunology, 9, 2892. (doi: 10.3389/fimmu.2018.02892) (PMID:30619264) (PMCID:PMC6297804)
174157.pdf - Published Version
Sheep are not only a major livestock species globally, they are also an important large animal model for biomedical research and have contributed to our understanding of the ontogeny and architecture of the mammalian immune system. In this study, we applied immunohistochemistry and multicolor immunofluorescence in fixed and paraffin-embedded lymph nodes to phenotype the key populations of antigen presenting cells, lymphocytes and stromal cells that orchestrate the host adaptive immune response. We used an extensive panel of antibodies directed against markers associated with dendritic cells (MHC class II, CD83 and CD208), macrophages (CD11b, CD163 and CD169), stromal cells (CNA.42, S100 and CD83), and lymphocytes (CD3, Pax5, CD4, CD8). Using different methods of tissue fixation and antigen retrieval, we provide a detailed immunophenotyping of sheep lymph nodes including the identification of potential subpopulations of antigen presenting cells and stromal cells. By characterizing cells expressing combinations of these markers in the context of their morphology and location within the lymph node architecture, we provide valuable new tools to investigate the structure, activation and regulation of the sheep immune system in health and disease.
This study was funded by the Wellcome Trust (Investigator Award # 206369/Z/17/Z) and in part by a W.B. Martin Scholarship. Animal experiments conducted at the Istituto Zooprofilattico Sperimentale dell'Abruzzo e Molise G. Caporale were funded by the Italian Ministry of Health. GE and MSR are funded by the Scottish Government Rural and Environment Science and Analytical Services (RESAS). GE is also funded under European Union's Horizon 2020 research and innovation program under grant agreement No. 731014 (VetBioNet).
Rocchi, Dr Mara and Melzi, Miss Eleonora and Palmarini, Professor Massimo and Entrican, Professor Gary and Caporale, Dr Marco
Melzi, E., Rocchi, M. S., Entrican, G., Caporale, M., and Palmarini, M.
College of Medical Veterinary and Life Sciences > School of Infection & Immunity
College of Medical Veterinary and Life Sciences > School of Infection & Immunity > Centre for Virus Research
Frontiers in Immunology
Copyright © 2018 Melzi, Rocchi, Entrican, Caporale and Palmarini
First published in Frontiers in Immunology 9:2892
Reproduced under a Creative Commons License
301049 0 Host determinants of disease outcomes in arboviral infections Massimo Palmarini Wellcome Trust (WELLCOTR) 206369/Z/17/Z III - Centre for Virus Research
Ms Jacqui Brannan


I love the labs that I use. You can't see it but I have a big grin on my face. No, not the nice kind. The kind like you can't tell if I'm serious or not. Consistency consistency that's the priority. In my 27 years I found that all labs are very consistent. There will always be a job that takes weeks to get. There will always be a job that arrives incorrect or has a defect. There will always be credits I have to chase after. Private or public owned it does not matter. Higher price lower price it does not matter. I get the same results all the way around. I like to deal with privately owned labs. You may think there's a tradeoff from lower price compared to higher priced lenses. I don't believe in that. Lenses are lenses and should be exact to Rx and free of surface anomalies. Lens material and coating quality/durability is the subject of another blog. Thankfully these consistencies are few and far between, a least with the labs I use, otherwise I move onto another lab. That being said, I received a call from Peter at Identity Optical. I always have an ear for motivated individuals seeking prosperity. It was his lucky day. I'm sure he wasn't expecting a grilling that day. If you want my business you better have knowledge equal or more than mine, know your products, know your pricing and be quick about it. I'm a stickler, perfectionist and short on time. He was quick and knowledgeable and gained my business. Communication is key and Peter scores well there too. So far, excellent products and service from Identity Optical. Incoming sales calls can be annoying but they're worth the opportunity of finding a diamond in the rough. It's worth checking out these guys.


text:
Latest GE facility lights up Middle East and Africa
Since the 1890's, from Thomas Alva Edison's invention of the very first incandescent lightbulb, to the invention of the Light Emitting Diode, to the mass marketing of LEDs in all the brightness and colours we see today, GE has been at the forefront of innovation.
Now, that innovation is illuminating the Middle East and Africa, with the opening of the first GE Lighting distribution center in the region. Located in Dubai's Jebel Ali Free Zone, covering 1,375 square meters, the GE Lighting center will serve as a strategic hub for the company to meet the growing need for intelligent and energy-efficient lighting solutions in the UAE as well as strengthen exports to key markets across the Middle East and Africa.
The distribution center will have over 2,000 products, bringing one of the largest showcases of GE Lighting products beside its headquarters in Budapest, Hungary. All products will be shipped in from Hungary and the other manufacturing centers of the company around the world.
In the UAE, GE Lighting has been associated with several key initiatives including nearly 70 per cent of street lighting in Abu Dhabi, as well as lighting solutions for several landmark projects across the country.
In fact, with a range that extends beyond mainstream applications such as home, office, shop or street lighting, and into showbiz applications, aviation and even lighting for medical applications that destroys bacteria, you could say Lighting is one of GE's brightest stars.
Importance of desalination in the Middle East
Working for GE in the Middle East
Healthymagination: Addressing the Middle East's healthcare challenges


BIN_INPUT mod provides input implementation which extracts bytes from bin vals.
BIN_INPUT.new makes an input which extracts bytes from Bin.
Bin must be a bin.
The result input does not cause IO errors.
.close method of the result input does nothing, thus you don't have to call .close for the input.


Industrialization, globalization, and liberal policies of government about foreign direct investment in recent years have generated tough competition between companies to maximize profits. In order to survive in highly competitive market companies needs to reduce overhead cost, pay back maximum return to investors and attract potential buyers as well as to retain them. Healthy finance is the key for growth of any individual or organization. To compete with peer group companies, in industry, one needs to put oneself up ahead of others. All this created needs of eligible finance professionals to manage finances, minimize losses and maximizes profits.
There are many paths in academics following which one can become part of finance industry. Students can pursue graduation and post graduation in finance related disciplines like accounts, commerce, business, economics, statistics etc and further start work in finance and investment sector. Another option is to obtain technical professional qualification in finance like MBA Finance or Obtain CFA Certification etc to be part of Finance industry. Entry into finance field is open to both commerce and non commerce stream students.
To start work in industry in core financial capacity like Financial Manager etc one needs at least post graduate degree in finance. MBA Finance is one such course which helps you in getting job as Financial Manager in a company. Students with post graduate degree in MBA Finance i.e. Financial Management can get work opportunities with government and public sector companies as finance officer, risk manager, insurance manager, treasures, manager finance & accounts, to name a few. Though, designations may vary according to company profile. Companies also hire financial managers as consultants who advise senior managers on various business issues. Government also receives advice from financial experts in various issues related to finance and investment.


66 ft. x 130 ft.
Located at the end of a tranquil cul-de-sac, this charming home is situated on a private and sunny 8,580 sq.ft. lot and boasts large principal rooms with 5 bedrooms & 5 bathrooms. Recent renovations on the main level yield a true chefs kitchen with gas range, double wall oven, quality cabinets and quartz counters, tons of natural light and stunning spa-like bathrooms. Enjoy the unbeatable privacy this property offers and the breathtaking mountain views. Only steps to all shopping and amenities in classic Kerrisdale & very close proximity to some of Vancouver's top rated public & private schools, UBC and Downtown. Currently rented & earns over $7K/mnth income, but redevelopment and/or expansion is allowed with city's approval.


text:
Günter Schultze ist der Name folgender Personen:

* Günter Schultze-Dewitz (* 1928), deutscher Forst- und Holzwissenschaftler
 Günter Siegmund-Schultze († 1993), deutscher Jurist, Anwalt und Fachautor

Siehe auch:
 Günther Schultze

redpajama_set_name: RedPajamaWikipedia


The anas latest investigative video on the corrupt activities within the Ghana Football Association (GFA) has left so many Ghanaians shocked.
The video which was premiered at the Accra International Conference Centre has uncovered the many rots in the Ghana football administration.
It emerged that Communications Director of the Ghana Football Association, Ibrahim Saani Daara accepted money to fix a player in the Black Stars.
The money we now know came from agents of Anas' Tiger Eye PI to influence the selection of a player into the Black Stars not knowing their real identity.
In the investigative video put out by Anas and his team on corruption into football in Ghana, the FA spokesperson refused to touch the money being offered him but asked a colleague standing on the side to accept it on his behalf.


text:
Q: WPF listbox binding selected only updates items in view I've been running into a strange issue with a listbox within a UserControl with WPF and MVVM. The listbox has its items bound via a ViewModel and an ObservableCollection<T>, the style property IsSelected is bound to a property Selected inside the individual items.
[...]
<ListBox ItemsSource="{Binding Path=SQLObjects}" FontFamily="Consolas" Margin="5,0,5,5" DisplayMemberPath="Name" SelectionMode="Extended" SelectedItem="{Binding Path=SelectedSQLObject}">
    <ListBox.Resources>
        <Style TargetType="{x:Type ListBoxItem}" BasedOn="{StaticResource MetroListBoxItem}">
            <Setter Property="IsSelected" Value="{Binding Selected, Mode=TwoWay}" />
        </Style>
    </ListBox.Resources>
</ListBox>
[...]

When a user selects a single or multiple items (with ctrl or shift) the properties are updated as expected. But selecting all items with ctrl + a only updates the property Selected of the items visible inside the scroll area of the listbox.

in the screenshot all items are selected with ctrl + a but only the items currently visible have their property updated. When scrolling down in the listbox without changing the selection, all items are updated as they come into view.
Could this be caused by MahApps.Metro?
The ViewModel is bound to the UserControls Datacontext inside the xaml in the main window.
[...]
<TabItem Header="Trigger">
    <userControls:ObjectControl>
        <userControls:ObjectControl.DataContext>
            <viewmodel:TriggerObjectViewModel DialogCoordinator="{x:Static local:MainWindow.Coordinator}" />
        </userControls:ObjectControl.DataContext>
    </userControls:ObjectControl>
</TabItem>
[...]

All other TabItems use the same usercontrol and have the same issue.
I would like to not use a workaround (e.g. scrolling trought the whole list when the user selects items outside of the current view), is there any kind of property I could set for WPF to update all items inside a collection, regardless if they are visible or not?

A: I suspect this is due to virtualization. The UI items don't exist until they are scrolled into view.
Try turning off virtualization on your list control:
<ListBox ItemsSource="{Binding Path=SQLObjects}"
         VirtualizingPanel.IsVirtualizing="False">

redpajama_set_name: RedPajamaStackExchange


text:
Facebook Acquires Simple File-sharing Service Drop.io
By Ben Parr 2010-10-29 18:28:31 UTC
Simple file-sharing service Drop.io has been acquired by Facebook.
"Today, we're proud to announce that we've struck a deal with Facebook," the company said in an announcement on its blog. "What this means is that Facebook has bought most of Drop.io's technology and assets, and Sam Lessin is moving to Facebook."
Drop.io was led by Sam Lessin, a former consultant for Bain & Company. He attended Harvard University (2001-2005) and even knew Mark Zuckerberg during the early days of Facebook. Now he will take up a new role at the world's largest social network.
As a result of the acquisition, Drop.io will be shutting down; paid accounts will be discontinued on December 15. This is yet another talent acquisition for Facebook; the data on Drop.io will be deleted by the end of the year.
We've included Drop.io's full announcement below. Let us know what you think of the acquisition in the comments:
"Here at drop.io, we've had a great three years helping people privately share pictures, video, audio, documents, andmore. It's been a thrill to watch the first, thousandth, millionth, and ten-millionth drop be created.
Today, we're proud to announce that we've struck a deal with Facebook. What this means is that Facebook has bought most of drop.io's technology and assets, and Sam Lessin is moving to Facebook.
In the coming weeks, we'll be winding down the drop.io service. As of this week, people will no longer be able to create new free drops, but you'll be able to download content from existing drops until Dec. 15. Paid user accounts will still be available through Dec. 15 and paid users will be able to continue using the service normally. After Dec. 15, paid accounts will be discontinued as well.
Please download your information before Dec. 15 – we plan to delete it after that time. No user data or content will be transferred to Facebook, and we'll send out e-mails to everyone to remind them about the service closing.
Other drop.io services like Presslift, our Yahoo! Mail Application, and our APIs will stay online for a longer period before the company winds down. We'll announce more details on that soon. However, starting immediately, we'll no longer be selling new premium accounts for those services.
To our loyal users and fans, we cannotthank you enough for your patronage, support, and feedback (good and bad) over the years.
Warmest regards, The drop.io Team"
Topics: acquisition, drop.io, Facebook, Social Media


text:
Marsh View Cottage, self catering holiday cottage in Camber Sands, Rye, East Sussex - the accommodation
MARSH VIEW COTTAGE
What is included in your stay
Weekend breaks in Camber Sands, Rye
Our welcome baskets
Availability/rates/booking information
Places to visit/local information
Camber Sands beach
Local food and drink
Romney Marsh and local area
Romney Marsh is the collective name for three marshes – Walland Marsh, Romney Marsh and Denge Marsh. These marshes stretch from East Sussex into Kent, covering around 100 square miles.
The marsh you can see directly behind Marsh View Cottage is Walland Marsh. Part of the bigger Romney Marsh, Walland Marsh has been designated SSSI status [Site of Special Scientific Interest] as it is an important habitat for a variety of wildlife and rare and protected plants and wildlife.
Some of these include the protected great crested newt as well as common newts [often found basking on the path leading to our front door!], marsh frogs, stoats, badgers,rabbits, hares, Sussex emerald moth, a variety of birds, Emperor dragonfly, marsh mallow moth, to name just a few.
Romney Marsh is sometimes referred to as 'The Fifth Continent'. It was dubbed this in the 19th century by Thomas Ingoldsby, an author, who wrote – ''The world, according to the best geographers, is divided into Europe, Asia, America, Africa and Romney Marsh.''
Romney Marsh, Rye and Camber were the birthplace of smuggling in southern England. This land, reclaimed from the sea following a great storm in the 13th century, made good grazing land for sheep. The export of their wool was highly taxed and poorly policed. Wool smuggling fluctuated according to the laws of the land and market forces at the time. During the 15th century, wool prices in this country fell, and there was temptation to smuggle overseas for more profit. By the 17th century, this had reached a huge scale, and Romney Marsh and the local area, including Rye and Camber Sands [the dunes made a good hiding place] were at the very heart of the wool smuggling trade. Soon, the goods being smuggled in and out of the country, to nearby mainland Europe, expanded to include tobacco, brandy, tea, lace, silk, and more.
There are 14 ancient churches on Romney Marsh. There used to be around eight more, but these have now disappeared over the passing centuries. This is a high number of churches in a comparatively small area. Many of these churches were themselves involved with the smuggling trade. Under cover of darkness, the smugglers would take their contraband to at least four of these churches, where an obliging cleric would hide the goods in the church vaults and other places that were unlikely to be searched. The church in Ivychurch is said to have a secret passage leading to the inn next door.
The churches are interesting to visit. We haven't visited them all ourselves yet. If you only have time to visit one, we would recommend Thomas a Becket in Fairfield. This ancient little church is surrounded by sheep and the water courses that cover the marsh which serve to help protect against flooding. The village of Fairfield, which it served, has long since disappeared, making the church isolated. On the notice board is a note explaining where you can get the key from to have a look inside.
Sue - co-owner of Marsh View Cottage, Camber Sands, Rye,East Sussex