\section{Introduction}
Deep learning suffers a serious performance drop under the distribution shift~\cite{ben2006analysis}. Unsupervised domain adaptation (UDA) is proposed to adapt a source model to a new unlabeled target domain. Most UDA research considers the adaptation from single or multiple sources to a single target (STDA). However, in reality, the target domain can be diverse and include various styles and textures, and the distribution of each class also varies from each target. These steer us to consider a practical yet challenging setting termed as \textit{blended targets domain adaptation (BTDA)}: 1)Adaptation is conducted from one single source to multiple targets. 2)Neither domain labels nor class labels are available on targets and the model should perform well on each target. 3)Label distributions of different targets can be different (label shift). In the following, we first present the analysis of BTDA, and discuss limitations of current methods due to these essential issues. Finally, we discuss that domain labels are not \textit{directly} necessary for BTDA and propose the category-oriented mutual conditional domain adaptation (MCDA), which also generalizes to common settings.

There are two practical issues for distributional alignment in BTDA: 1)Diverse styles and textures of blended targets. 2)Label shift of various targets. These induced our key observation that categorical feature space in BTDA is hybrid and unstructured as shown in Figure~\ref{btda_hybrid_feat}. Features of different classes in the blended targets are pervasive and do not form a well-clustered structure. To analyze it, we conduct t-SNE for feature space under BTDA in the left. Besides, we also uniformly sample and calculate K nearest neighbors (KNN) of each class center under STDA and BTDA. The result in the middle shows that the number of samples within the same class in STDA is more than that of BTDA. This indicates that the cluster structure of BTDA is not well formed compared to STDA which corresponds to the hybrid categorical feature space in t-SNE visualization.
This \textit{weakens} the cluster assumption~\cite{chapelle2005semi} that serves as the necessary condition of many adaptation methods~\cite{tachet2020domain,shu2018dirt,tang2020unsupervised,yang2021exploiting}. Further, it motivates our analytical perspective from both categorical distribution shift and biased classifier for BTDA.

Current UDA methods yield sub-optimal performance in BTDA due to these. Concretely, methods based on the \textit{covariate shift} assumption and aligning marginal distributions~\cite{tzeng2017adversarial,ganin2015unsupervised,shen2018wasserstein} are inevitable to increase the joint error of optimal hypotheses under the label shift~\cite{wu2019domain,tachet2020domain}. BTDA worsens the situation with diverse target domains and the more serious imbalance and label shift issues. Some theories further propose conditional alignment formulation to avoid the joint error issue. ~\cite{tachet2020domain,jiang2020implicit} implicitly align conditional distribution by aligning reweighted marginal distributions that still needs cluster assumption. Other methods use the target pseudo labels to model and align conditional distributions through class centroids~\cite{pan2019transferrable,tanwisuth2021prototype,singh2021clda}, task-oriented classifiers~\cite{zhang2019bridging,saito2019semi}, and the conditional discriminator~\cite{long2018conditional}. These effective STDA methods produce limited performance under the hybrid feature space in BTDA. Centroid and general adversarial methods may not model distributions well. The biased classifier and clustering labeling algorithm also generate noisy labels in this situation.

\begin{figure*}[t]
\centering
\includegraphics[width=0.93\textwidth]{aaai23_main.pdf}
\caption{Left: t-SNE for hybrid categorical feature space of BTDA where features of various classes are pervasive and unstructured. The color indicates the domain and the digit indicates the class. Middle: the sample rate of the same class for each class center's K nearest neighbors. All data are collected from Office-Home (ResNet-50). Right: BTDA distribution shift where features are unstructured and the classifier is biased.}
\label{btda_hybrid_feat}
\end{figure*}

Recent multi-target domain adaptation (MTDA) methods produce impressive results by generally inferring or utilizing the domain level information and then conducting STDA methods. Some methods train separated models for each target which is not efficient in practice~\cite{saporta2021multi,isobe2021multi,nguyen2021unsupervised,saporta2021multi}. Other methods utilize graph neural networks with co-teaching~\cite{roy2021curriculum}, disentanglement methods~\cite{gholami2020unsupervised} and meta-clustering~\cite{chen2019blending}. These require domain labels and lack consideration for the imbalance and the hybrid target feature space in BTDA.

In this paper, we address two intrinsic issues of BTDA: 1)Domain labels. 2)Hybrid categorical feature space. First, our analysis shows that domain labels are not \textit{directly} necessary for BTDA only if the categorical distributions of various domains are sufficiently aligned even facing the imbalance and the label shift. However, categorical alignment requires labels. The hybrid categorical feature space in BTDA raises practical issues in modeling categorical distributions and producing reliable pseudo labels. Considering these, we design techniques to explicitly model and align categorical distributions $P(Z|Y)$ of various domains and simultaneously correct the biased classifier $P(Y|Z)$ among diverse targets to enhance pseudo labels.

Practically, this motivates two designs on $P(Z|Y)$ and $P(Y|Z)$. Firstly, for modeling and aligning $P(Z|Y)$, current methods such as prototype~\cite{pan2019transferrable,tanwisuth2021prototype} and kernel methods~\cite{wang2020rethink} inferiorly model conditional distributions of unstructured data features in the hybrid feature space  in Figure~\ref{btda_hybrid_feat}. Leveraging the distribution modeling ability of GAN~\cite{arora2017generalization,goodfellow2014generative}, we propose an uncertainty-guided categorical domain discriminator. We encode categorical distributions within the same semantic space to \textit{explicitly} model and \textit{directly} align $P(Z|Y)$ of various domains. Since the discriminator is supervised with source and noisy target labels, we adopt uncertainty to guide it to gradually learn and align categorical distributions. Secondly, to correct the biased classifier for reliable pseudo labels during adaptation, we first adopt balanced sampling on the source data and then utilize the low-level features in convolution neural networks (CNN) to augment the source features with diverse target styles to
reduce domain dependent information and balance the classifier training on target classes. Our method shows that one single labeled source can still be augmented with multiple targets to rectify the classifier during adaptation by leveraging the prior of low-level features in CNN.

In summary, our contributions are as follows: 1)We demonstrate that the adaptation can be well-achieved without domain labels in BTDA if categorical distributions are sufficiently aligned even facing the imbalance and label shift.
2)We propose the mutual conditional alignment to directly minimize conditional distributions and simultaneously correct the biased classifier.
3)Practically, to address the hybrid feature space of BTDA, we design an uncertainty-guided categorical domain discriminator to explicitly model and align categorical distributions, and utilize low-level features to mitigates the bias of classifier on blended targets.
Our method achieves the state-of-the-art in BTDA even compared with methods using domain labels, especially under the label shift, and in STDA with DomainNet.

\section{Related Works}
\noindent\textbf{Single target UDA (STDA):} STDA is a typical setting that adapts  single or multiple sources into one target. Generally, the research includes four categories. One branch minimizes the explicit statistical distance such as Maximum Mean Discrepancy (MMD) to mitigate the domain distribution shift~\cite{long2015learning,long2017deep,officehome,tzeng2014deep,shen2018wasserstein,lee2019sliced,xu2019wasserstein,montesuma2021wasserstein}. The second branch leverages the adversarial training to implicitly minimize the domain discrepancy through GAN~\cite{ganin2015unsupervised,tzeng2017adversarial,zhang2019bridging} or entropy minimization~\cite{pan2020unsupervised,vu2019advent}. The third one utilizes the self-training with the target pseudo labels to train the source model~\cite{liu2021cycle,french2017self}. The fourth one utilizes image translation techniques to mitigate the semantic irrelevant gap~\cite{sankaranarayanan2018generate,roy2021trigan,kim2020learning,yang2020label}. However, these methods produce limited performance in BTDA. The serious imbalance and label shift issues in blended targets cause a serious incremental error of classifiers~\cite{wu2019domain}, and the hybrid target feature space also yields noisy pseudo labels and calibration issues~\cite{mei2020instance}, which deteriorates the self-training and conditional alignment methods.

\noindent\textbf{Multi-target UDA (MTDA):} transfers the knowledge from a single source to multiple targets. MTDA is recently studied in both classification~\cite{gholami2020unsupervised,nguyen2021unsupervised,chen2019blending,roy2021curriculum,yang2020heterogeneous} and semantic segmentation ~\cite{saporta2021multi,isobe2021multi}. One common approach is to disentangle domain information from multiple targets by adversarial learning and adapt each target with a separated network~\cite{saporta2021multi,gholami2020unsupervised}. AMEAN~\cite{chen2019blending} first clusters blended targets into sub-clusters and adapts the source with each cluster. CGCT~\cite{roy2021curriculum} use graph convolution network (GCN) for feature aggregation, and use GCN classifier and source classifier for co-teaching. Differently, our method does not require any domain label and conducts BTDA in an united network which is scalable and efficient. Besides, our model considers the hybrid categorical feature space and is robust under the imbalance and label shift in BTDA.

\section{Methodology}
We present our analysis of BTDA and discuss the proposed mutual conditional domain adaptation (MCDA) framework including: explicit categorical adversarial alignment, uncertainty-guided discriminative adversarial training, and low-level feature manipulation for the classifier correction.

\noindent\textbf{Notation.} Let us denote the input-output space $\mathcal{X} \times \mathcal{Y}$ where $\mathcal{X}$ represents the image space and $\mathcal{Y}$ represents the label space. The labeled source domain is denoted as $\mathcal{S}=\{x_i^s, y_i^s \}^{|S|}_{i=1}$ and each unlabeled target is denoted as $\mathcal{T}_j=\{x_i^{t_j} \}^{|T_j|}_{i=1}$. Both source and target domains are i.i.d sampled from some distribution $P_{\mathcal{S}}(X, Y)$ and $P_{\mathcal{T}_j}(X, Y)$. For the model, we denote the feature extractor $g: \mathcal{X} \to \mathcal{Z}$ and the classifier $h: \mathcal{Z} \to \mathcal{Y}$. The error rate of the model on source $\mathcal{S}$ and target $\mathcal{T}_j$ are $\epsilon_S$ an $\epsilon_{T_j}$, and the blended target error rate is evaluated as $\epsilon_T = \frac{1}{K} \sum_{j} \epsilon_{T_j}$.

MCDA is a unified framework that adapts a single source to the blended targets such that the model performs well on each single target even facing the label distribution shift across various targets. i.e., $P_{\mathcal{S}}(Y) \neq P_{\mathcal{T}_j}(Y)$ and $ P_{\mathcal{T}_j}(Y) \neq P_{\mathcal{T}_m}(Y)$.
As proved in \cite{tachet2020domain}, minimizing marginal distribution shift of $P_{\mathcal{S}}(X)$ and $P_{\mathcal{T}}(X)$ can arbitrarily increase the target error $\epsilon_T$ due to the label shift, which fails the adaptation. The situation becomes worse in BTDA since each target can have a different $P_{\mathcal{T}_m}(Y)$. In that, we are interested to have a bound such that each term of it can be independently minimized as much as possible, and that is better irrelevant with domain labels.

\noindent\textbf{Blended error decomposition theorem.} Inspired by the generalized label shift (GLS) theorem in
\cite{tachet2020domain}, we intend to align the conditional distributions of each class $P(Z|Y)$ within each target to the same class in the source domain on the feature space $\mathcal{Z}$.
\begin{theorem}
For any classifier $\hat{Y}=(h\circ g)(X)$, the blended target error rate is
\begin{small}
\begin{equation}
\begin{split}
\| \epsilon_S  - \frac{1}{K} \sum_{j}^K \epsilon_{T_j} \| \leq & \frac{1}{K} \sum_{j}^K \| P_{\mathcal{S}}(Y) - P_{\mathcal{T}_j}(Y) \|_1 BER_{P_{\mathcal{S}}}(\hat{Y}\|Y) \\
& + 2(c-1)\Delta_{BTCE}(\hat{Y}).
\end{split}
\end{equation}
\label{BER}
\begin{equation}
BER_{P_{\mathcal{S}}}(\hat{Y}\|Y) = \max_{j \in [k]} P_{\mathcal{S}}(\hat{Y} \neq Y | Y = j)
\end{equation}
\begin{equation}
\begin{split}
\Delta_{BTCE}(\hat{Y}) = & \frac{1}{K} \sum_{j}^K \max_{y \neq y^\prime \in \mathcal{Y}^2}| P_{\mathcal{S}}(\hat{Y} \neq Y | Y=y)  \\
& - P_{\mathcal{T}_j}(\hat{Y} \neq Y | Y=y)|
\end{split}
\end{equation}
\end{small}
\end{theorem}
where $\| P_{\mathcal{S}}(Y) - P_{\mathcal{T}_j}(Y) \|_1$ represents the $L_1$ distance of label distributions between the source and each target and is a constant only depending on the data, $BER_{P_{\mathcal{S}}}(\hat{Y}\|Y)$ is the classification performance only related with the source domain. $\Delta_{BTCE}(\hat{Y})$ measures the conditional distribution discrepancy of each class between the source and each target. In this sense, we only need to minimize the $\Delta_{BTCE}(\hat{Y})$, which is equivalent to minimize the discrepancy between $P_{\mathcal{S}}(Z | Y=y)$ and $P_{\mathcal{T}_j}(Z | Y=y)$.

\noindent\textbf{Key differences.} First, different from \cite{tachet2020domain}, we argue that the theorem 3.3: clustering structure assumption in \cite{tachet2020domain} is a strong assumption in BTDA because each target has a different cluster structure $Z_{\mathcal{T}_j}$ in feature space under the pretrained feature extractor $g_{\mathcal{S}}$, which induces hybrid categorical feature space and different decision boundaries for different target $\mathcal{T}_j$ as illustrated in Figure~\ref{btda_hybrid_feat}. When blended together, the cluster of class $a$ in $\mathcal{T}_m$ may overlap with the cluster of class $b$ in $\mathcal{T}_n$. Consequently, the sufficient condition for GLS may not hold. Thus, calculating class ratios and aligning reweighted marginal distributions in \cite{tachet2020domain} do not induce GLS to align the semantic conditional distributions. Second, when the number of classes $ | \mathcal{Y} |$ is large, solving a quadratic problem to find class ratios require $O(|\mathcal{Y}|^3)$ time which is not efficient and accurate. We do not calculate class ratios. Finally, we do not make any assumption to satisfy GLS. Instead, we adopt the general bound in equation \ref{BER} and design model to directly minimize the conditional JS divergence $\mathcal{D}_{JS}(P_{\mathcal{S}}(Z | Y=y) \| P_{\mathcal{T}_j}(Z | Y=y)) $ to enforce it. However, aligning conditional distribution requires accurate pseudo labels. This motivates us to develop a mutual conditional alignment system to align $P(Z|Y)$ and $P(Y|Z)$ simultaneously. Besides, since we only use the class label, the domain label is unnecessary, which suits the BTDA setting.

\begin{figure*}[t]
\centering
\includegraphics[width=0.95\textwidth]{main2.pdf}
\caption{The framework of MCDA. The source data utilizes balanced sampling for training the categorical discriminator and is augmented with blended target styles to train the classifier. The target data is randomly sampled, and the predicted pseudo labels with low uncertainty are converted to one-hot labels to train the categorical domain discriminator.}
\label{main_fig}
\end{figure*}

\subsection{Explicit categorical adversarial alignment}
Our motivation is to \textit{explicitly} model and \textit{directly} align the categorical JS divergence $\mathcal{D}_{JS}(P_{\mathcal{S}}(Z|Y=y) \| P_{\mathcal{T}_j}(Z|Y=y)) $ between the source and each target under the hybrid feature space. Current categorical alignment methods utilizing task-classifiers, prototypes, and conditional discriminator~\cite{zhang2019bridging,saito2019semi,long2018conditional} may not represent conditional distributions well in the hybrid categorical feature space in BTDA.

Leveraging the distribution modeling ability of GAN, we intend to encode categorical distributions of various domains into the same semantic space to explicitly model categorical distributions for optimization. Inspired by DANN~\cite{ganin2015unsupervised}, we augment the last layer of a general domain discriminator $D$ into the number of classes $k$, and each logit of such a categorical domain discriminator $D^k$ is followed by a sigmoid function to predict the probability of a feature belonging to the source or target domain conditional on the corresponding class. Each logit behaves as a single GAN to minimize the discrepancy of JS divergence of a specific class $P_\mathcal{S}(Z | Y=y)$ and $P_{\mathcal{T}_j}(Z | Y=y)$. To make each logit corresponds to one class in $D^k$, we feed one feature $g(x_i)$ into $D^k$ and get the prediction $d_i \in R^k$. Then we use the corresponding \textit{one-hot label} $y_i \in \{0,1\}^k$ to only activate the corresponding logit to compute adversarial loss by $y_i\cdot d_i$. To achieve this, we use pseudo target labels and design a strategy to make the categorical adversarial alignment and pseudo label refinement reinforce each other. Then we formulate the optimization as follows
\begin{small}
\begin{equation}
\setlength{\abovedisplayskip}{2pt}
\setlength{\belowdisplayskip}{2pt}
\begin{split}
\mathcal{L}_{adv}(g,D^k) & = \frac{1}{n_s} \sum_{i=1}^{n_s} y_i \cdot log[D^k(g(x_i^s))] + \\
    & \frac{1}{n_t} \sum_{j=1}^{n_t} \bar{y}_j \cdot log[1-D^k(g(x_j^t))],
\end{split}
\end{equation}
\end{small}
where $y_i$ represent the one-hot true labels of the source and $\bar{y}_j$ represent the mix of soft and one-hot pseudo labels of target. We discuss this in detail in the next section.

\subsection{Uncertainty guided discriminative adversarial training}
To train a discriminative categorical domain discriminator $D^k$, we require the one-hot true labels of source and blended targets. However, since the initial target labels are noisy, we design the uncertainty-guided training strategy for our categorical domain discriminator. We start with soft target labels and then gradually covert soft target labels with low uncertainty into one-hot encoding as training goes by. We use the entropy as the metric of the uncertainty of each sample, and select the samples based on a threshold $\gamma$.
\begin{small}
\begin{equation}
\setlength{\abovedisplayskip}{2pt}
\setlength{\belowdisplayskip}{2pt}
\begin{split}
H(x_j) = - \sum_{k=1}^K \hat{y}_{j,k} \cdot log(\hat{y}_{j,k})
\end{split}
\end{equation}
\end{small}
\begin{small}
\begin{equation}
\setlength{\abovedisplayskip}{2pt}
\setlength{\belowdisplayskip}{2pt}
\bar{y}_j=\left\{
\begin{aligned}
& \{0,1\}^k,     & if H(x_j) < \gamma \\
&  \hat{y}_j,  & otherwise \\
\end{aligned}
\right.
\end{equation}
\end{small}
In the early stage, the entropy of soft pseudo labels on the target domain is large so that each digit of $D^k$ is assigned with similar probability mass. $D^k$ cannot discriminate different classes and behaves as the general discriminator $D$ in DANN since all logits share the same semantics. As the training goes, the entropy of target pseudo labels will decrease, and the labels will become more discriminative owing to the distribution alignment. At the same time, the discriminative target labels will also train $D^k$ to distinguish different categories and further align the categorical distributions, which forms a mutually reinforced process.

\subsection{Source-only balanced adversarial training}
We expect our model to be robust under the label shift across various domains such as in a case shown in Figure~\ref{btda_lmt}. Equation \ref{BER} indicates that the label shift only influences the classification error $BER_{P_{\mathcal{S}}}$ but does not influence the major distribution discrepancy $\Delta_{BTCE}$. It indicates that only if $\Delta_{BTCE}$ is small enough, the model is robust to label shift even in the blended domains.

So, we focus on training $D^k$ since the class imbalance will lead to bias to the training of $D^k$. Hence, $D^k$ cannot distinguish different classes and align distributions biased towards the majority classes, which will ruin the categorical distribution alignment. To train a balanced $D^k$, we propose to only conduct balanced sampling for the source domain rather than on both domains as in \cite{jiang2020implicit} for two reasons: 1)We only have true labels on the source domain, balanced sampling based on the hard target pseudo labels may introduce errors and bias because initially target pseudo labels are inaccurate. Filtering out confident target pseudo labels may rule out some classes, which exacerbates the class imbalance issue. 2)With conventional double-side balanced sampling, target pseudo labels are only updated every epoch. Instead, mixed target pseudo labels can be updated online with the distribution alignment, which is more beneficial for adaptation. We demonstrate the robustness and efficiency in the Experiments section.

\subsection{Low level feature for classifier correction}
We intend to correct the biased classifier $P(Y|Z)$ from a single source to blended targets during the adaptation process. This improves the pseudo label accuracy on blended targets during the adaptation process and further facilitates the training of our categorical domain discriminator. Inspired by the research on low-level features on CNN, we utilize the low-level features of CNN that mainly represent the style and background of images to project blended target styles into the source for correcting the classifier. Denoting the low-level feature maps $z \in \mathbb{R}^{D\times H \times W}$ where $D$ represents the channel and $H,W$ represents the spatial size, leveraging the AdaIN, we have augmented features $z^{st}$ with source content and target style as below:
\begin{small}
\begin{equation}
\mu_t = \frac{1}{HW} \sum_{h=1}^H \sum_{w=1}^W z^t \hspace{2mm}
\end{equation}
\begin{equation}
\begin{split}
\sigma_t = \sqrt{\frac{1}{HW} \sum_{h=1}^H \sum_{w=1}^W (z^t - \mu_t)^2+\epsilon}
\end{split}
\end{equation}
\begin{equation}
\begin{split}
z^{st} = AdaIN(z^s, z^t) = \sigma_t (\frac{z^s - \mu_s}{\sigma_s}) + \mu_t
\end{split}
\end{equation}
\end{small}
Compared with previous image translation methods, our method does not need to generate specific images, which is efficient in practice. Besides, considering the diversity and imbalance of blended targets, our method achieves the correction on two sides: 1)Since the source is evenly resampled on class, the augmented feature $z^{st}$ with source content is balanced on semantic classes, which forms a balanced classifier for inference. 2)The augmented $z^{st}$ with diverse target styles mitigate the domain irrelevant information. This regularizes the hybrid categorical feature space in BTDA and make the cluster assumption more practical.

\noindent\textbf{Overall Objective:} Eventually, the final loss function consists of the categorical adversarial loss and the classification loss of various domains. Note that $h^{\prime}$ indicates the networks excluding the shallow layers.
\begin{small}
\begin{equation}
\setlength{\abovedisplayskip}{2pt}
\setlength{\belowdisplayskip}{2pt}
\begin{split}
\mathcal{L}_{cls}(g,h) & = \frac{1}{n_s} \sum_{i=1}^{n_s} l_{ce}(g\circ h(x_i^s), y_i^s) + \\
& \frac{1}{n_s} \sum_{i=1}^{n_s} l_{ce}(g\circ h^{\prime}(z_i^{st}), y_i^s)
\end{split}
\end{equation}
\begin{equation}
\setlength{\abovedisplayskip}{2pt}
\setlength{\belowdisplayskip}{2pt}
\begin{aligned}
\min_{g,h} \max_D \mathcal{L} = \mathcal{L}_{cls}(g,h) + \mathcal{L}_{adv}(g,D^k)
\end{aligned}
\end{equation}
\end{small}

\section{Experiments}
\noindent\textbf{Datasets.} We evaluate our method based on standard BTDA tasks~\cite{chen2019blending,roy2021curriculum}: Office-31~\cite{office31}, Office-Home~\cite{officehome}, DomainNet~\cite{domainnet}, and a specialized dataset Office-Home-LMT for label shift in BTDA. Similar to Office-Home-RS-UT~\cite{jiang2020implicit}, we use \textbf{Cl}, \textbf{Pr} and \textbf{Rw} to resample two reverse long-tailed distributions and one Gaussian distributions for each of them for BTDA with label shift. For evaluation, we use one domain as the source and the rest as blended targets. The performance is evaluated as the mean accuracy of all target domains. We show a concrete example of \textbf{Ar} as the source in the label shift setting in Figure~\ref{btda_lmt}.

\begin{table*}[t]
\small
\centering
\setlength{\tabcolsep}{1.7mm}{
\begin{tabular}{@{}llcccccccccllccccccc@{}}
\toprule
\multicolumn{2}{l}{\multirow{2}{*}{Methods}} & \multicolumn{4}{c|}{\textbf{Office-31}}                                                                 & \multicolumn{5}{c|}{\textbf{Office-Home}}                                                                                & \multicolumn{2}{l}{\multirow{2}{*}{Methods}} & \multicolumn{7}{c}{\textbf{DomainNet}}                                                                                \\ \cmidrule(lr){3-11} \cmidrule(l){14-20}
\multicolumn{2}{l}{}                         & A                    & D                    & W                    & \multicolumn{1}{c|}{Avg.}          & Ar                   & Cl                   & Pr                   & Rl                   & \multicolumn{1}{c|}{Avg.}          & \multicolumn{2}{l}{}                         & Cli                  & Inf                  & Pai                  & Qui                  & Rea                  & Ske                  & Avg.                 \\ \midrule
\multicolumn{2}{l}{Source}                   & 68.6                 & 70.0                 & 66.5                 & \multicolumn{1}{c|}{68.4}          & 47.6                 & 42.6                 & 44.2                 & 51.3                 & \multicolumn{1}{c|}{46.4}          & \multicolumn{2}{l}{Source}                   & 25.6                 & 16.8                 & 25.8                 & 9.2                  & 20.6                 & 22.3                 & 20.1                 \\ \midrule
\multicolumn{2}{l}{DAN}                      & 79.5                 & 80.3                 & 81.2                 & \multicolumn{1}{c|}{80.4}          & 55.6                 & 56.6                 & 48.5                 & 56.7                 & \multicolumn{1}{c|}{54.4}          & \multicolumn{2}{l}{SE}                       & 21.3                 & 8.5                  & 14.5                 & 13.8                 & 16.0                 & 19.7                 & 15.6                 \\
\multicolumn{2}{l}{DANN }                     & 80.8                 & 82.5                 & 83.2                 & \multicolumn{1}{c|}{82.2}          & 58.4                 & 58.1                 & 52.9                 & 62.1                 & \multicolumn{1}{c|}{57.9}          & \multicolumn{2}{l}{MCD}                      & 25.1                 & 19.1                 & 27.0                 & 10.4                 & 20.2                 & 22.5                 & 20.7                 \\
\multicolumn{2}{l}{CDAN}                     & 93.6                 & 80.5                 & 81.3                 & \multicolumn{1}{c|}{85.1}          & 59.5                 & 61.0                 & 54.7                 & 62.9                 & \multicolumn{1}{c|}{59.5}          & \multicolumn{2}{l}{CDAN}                     & 31.6                 & 27.1                 & 31.8                 & 12.5                 & 33.2                 & 35.8                 & 28.7                 \\
\multicolumn{2}{l}{JAN}                       & 84.2                 & 74.4                 & 72.0                 & \multicolumn{1}{c|}{76.9}          & 58.3                 & 60.5                 & 52.2                 & 57.5                 & \multicolumn{1}{c|}{57.1}          & \multicolumn{2}{l}{DADA }                     & 26.4                 & 20.0                 & 26.5                 & 12.9                 & 20.7                 & 22.8                 & 21.5                 \\
\multicolumn{2}{l}{AMEAN}                    & 90.1                 & 77.0                 & 73.4                 & \multicolumn{1}{c|}{80.2}          & 64.3                 & 65.5                 & 59.5                 & 66.7                 & \multicolumn{1}{c|}{64.0}          & \multicolumn{2}{l}{MCC}                      & 33.6                 & 30.0                 & 32.4                 & 13.5                 & 28.0                 & 35.3                 & 28.8                 \\
\multicolumn{2}{l}{CGCT}                     & \textbf{93.9}        & 85.1                 & 85.6                 & \multicolumn{1}{c|}{88.2}          & 67.4                 & 68.1                 & 61.6                 & 68.7                 & \multicolumn{1}{c|}{66.5}          & \multicolumn{2}{l}{CGCT}                     & 36.1                 & 33.3                 & 35.0                 & 10.0                 & \textbf{39.6}        & 39.7                 & 32.3                 \\ \midrule
\multicolumn{2}{l}{Ours}                     & 92.4                 & \textbf{87.7}        & \textbf{88.8}        & \multicolumn{1}{c|}{\textbf{89.6}} & \textbf{71.7}        & \textbf{72.8}        & \textbf{68.0}        & \textbf{71.7}        & \multicolumn{1}{c|}{\textbf{71.1}} & \multicolumn{2}{l}{Ours}                     & \textbf{37.5}        & \textbf{37.3}        & \textbf{36.6}        & \textbf{17.8}        & 36.1                 & \textbf{41.4}        & \textbf{34.5}        \\ \midrule
\multicolumn{2}{l}{MTDA$^{\dag}$}     & 87.9     & 83.7     & 84.0     & \multicolumn{1}{c|}{85.2}    & 64.6      & 66.4    & 59.2    & 67.1     & \multicolumn{1}{c|}{64.3}     & \multicolumn{2}{l}{-}       & -        & -      & -      & -       & -     & -      & -       \\
\multicolumn{2}{l}{DCL$^{\dag}$}    & 92.6    & 82.5    & 84.7  & \multicolumn{1}{c|}{86.6}     & 63.0    & 63.0     & 60.0     & 67.0    & \multicolumn{1}{c|}{64.1}          & \multicolumn{2}{l}{DCL$^{\dag}$}        & 35.1        & 31.4        & 37.0        & 20.5     & 35.4     & 41.0        & 33.4          \\
\multicolumn{2}{l}{DCGCT$^{\dag}$}    & 93.4     & 86.0    & 87.1    & \multicolumn{1}{c|}{88.8}   & 70.5      & 71.6         & 66.0         & 71.2         & \multicolumn{1}{c|}{69.8}          & \multicolumn{2}{l}{DCGCT$^{\dag}$}      & 37.0                 & 32.2                 & 37.3                 & 19.3                 & 39.8                 & 40.8                 & 34.4                 \\ \midrule
                      &                      & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}               & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}               &                       &                      & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} \\
                      &                      & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}               & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}               &                       &                      & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}
\end{tabular}
}
\vspace{-0.8cm}
\caption{Accurary ($\%$) of BTDA on Office-31, Office-Home (ResNet-50), and  DomainNet (ResNet-101). Best results in Bold. Each domain represents the source and the rest domains are blended as the target. The accuracy is the mean of accuracies of all domains in the blended target. \dag indicates methods using domain labels.}
\label{btda_all}
\end{table*}

\noindent\textbf{Baselines and implementations.}
We compare our method with previous state-of-the-arts in standard BTDA and that with label shift, i.e., MTDA~\cite{nguyen2021unsupervised}, CGCT~\cite{roy2021curriculum}, MDDIA~\cite{jiang2020implicit}, CST~\cite{liu2021cycle}, and SENTRY~\cite{prabhu2021sentry}. For comparison with BTDA under label shift, we also combine the sampling strategies in MDDIA with CGCT. The detailed summary of comparison methods is in the supplementary. We followed the implementations in ~\cite{dalib}. For all datasets, we use SGD optimizer with learning rates $\eta_0 = 0.01$, $\alpha = 10$, and $\beta = 0.75$. We set the uncertainty threshold $\ \gamma = 0.05$ for all datasets. Since CST and SENTRY use AutoAugment for data augmentation, we set the number of transformations $N=1$ and the transform severity $M=2.0$ in AutoAugment~\cite{lim2019fast} for a fair comparison.

\begin{figure}[!htbp]
    \centering
    \includegraphics[width=1.6in]{btlds_homeclip2.png}
    \includegraphics[width=1.6in]{btlds_homepro_real3.png}
\caption{Label distribution shift of Office-Home-LMT.}
\label{btda_lmt}
\end{figure}

\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{3mm}{
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Methods} & Clipart & Product & Real & Avg.  \\ \midrule
Source  & 42.3    & 47.6    & 50.3 & 46.7 \\ \midrule
BSP     & 51.5    & 52.9    & 57.4 & 54.0 \\
CDAN    & 50.5    & 53.2    & 56.3 & 53.3 \\
DAN    & 51.0    & 49.2    & 56.8 & 52.3 \\
JAN    & 51.4    & 50.1    & 57.0 & 53.2 \\
DANN    & 46.6    & 50.4    & 53.3 & 50.1 \\
ADDA    & 45.0    & 49.7  & 52.8 & 49.2 \\
MCD    & 40.2    & 48.6    & 52.3 & 47.0 \\
MDD     & 43.7    & 56.0    & 57.8 & 52.5 \\
MDDIA   & 61.9    & 58.2    & 63.2 & 61.1  \\
CGCT    & 53.7    & 51.5    & 52.0 & 52.4   \\
CGCT+bal  & 57.1  & 53.0 & 56.8  & 55.7  \\
Ours    & \textbf{68.0}    & \textbf{62.3}    & \textbf{67.5} & \textbf{65.9} \\ \midrule
CST(aug)     & 58.3    & 57.4    & 63.4 & 59.7 \\
SENTRY(aug)  & 65.6    & 63.5    & 65.9 & 65.0  \\
Ours(aug)   & \textbf{69.1}  & \textbf{66.2}    & \textbf{68.9}    & \textbf{68.1}     \\ \midrule
Ours(oracle)   & 98.9  & 98.3    & 98.2    & 98.5  \\
S+T &99.7   & 99.8    & 99.8 &99.8 \\
\bottomrule
\end{tabular}
}
\caption{Accurary ($\%$) of Blended-Office-Home-LMT (ResNet-50). \textit{aug}: using 1 extra augmented data with RandAug~\cite{cubuk2020randaugment}. \textit{bal}: using balanced sampling. \textit{oracle}: discriminator trained with true source and target labels. \textit{S+T}: supervised learning on source and target.}
\label{btlds}
\end{table}

\noindent\textbf{Standard BTDA.} We summarize the standard BTDA in Table~\ref{btda_all}. Our method outperforms comparison methods with a clear margin. Concretely, our method outperforms the latest BTDA methods (e.g., AMEAN and CGCT) by 1.4\% on Office-31, 4.6\% on Office-Home, and 2.2\% on DomainNet. Moreover, even compared with methods utilizing ground truth domain labels, our method can still outperform them by 0.8\% on Office-31 and 1.3\% on Office-Home. The results validate our argument that categorical distribution alignment overwhelms in BTDA, and echos the theoretical intuition from equation~\ref{BER} that proper domain alignment is achievable even without domain labels in BTDA.

\noindent\textbf{BTDA with label shift.} We analyzed the essential label shift influence on BTDA and summarized results of the specialized Office-Home-LMT in Table~\ref{btlds}. The label distribution of each domain is different from each other. Since CST and SENTRY essentially require extra augmented data, we add 1 augmented data for each sample and evaluate CST, SENTRY, and ours in the same setting (\textit{aug}). Our method outperforms the label shift UDA method MDDIA by 4.8\% and SENTRY by 3.1\%. Compared with latest BTDA method CGCT, we get an improvement of more than 12\%. We also equip CGCT with balanced sampling strategy in MDDIA (i.e., CGCT+bal) whose result is still inferior to ours.

The result first demonstrates the label shift in BTDA seriously impedes the adaptation, especially for the marginal alignment methods. Second, it validates our proposition in theorem 1 that if categorical distribution $\Delta_{BTCE}(Y)$ can be properly minimized, the label shift only reweights the classification error $BER_{P_{\mathcal{S}}}(\hat{Y}\|Y)$, which is relatively small. Besides, our method does not require balanced sampling on target pseudo labels for every epoch, which can be trained and updated online. The extra data augmentation (e.g., RandAug) is not essentially necessary in the algorithm design.

\noindent\textbf{STDA.} We also validate the generalization ability of our method in STDA (i.e., Office-Home and DomainNet). We compare our method with SRDC~\cite{tang2020unsupervised} which considers cluster structures in STDA, and MDDIA~\cite{jiang2020implicit} which uses balanced sampling on both source and target domains. Our method achieve 72.4\% on Office-Home, 35.2\% on DomainNet which outperforms previous state-of-the-art method MDD+SCDA~\cite{li2021semantic} by 1.9\%. Please refer to supplementary for details.

\begin{table}[!htbp]
\centering
\small
\setlength{\tabcolsep}{0.5mm}{
\begin{tabular}{@{}lll|cccccccccc@{}}
\toprule
 \multicolumn{1}{l}{\multirow{2}{*}{mix}} & \multicolumn{1}{l}{\multirow{2}{*}{bal}} & \multicolumn{1}{l|}{\multirow{2}{*}{flip}} & \multicolumn{5}{c}{\textbf{Office-Home}}                                                                                &  & \multicolumn{4}{c}{\textbf{DomainNet}}                                                                    \\ \cmidrule(l){4-13}
\multicolumn{1}{c}{}                     & \multicolumn{1}{c}{}                          & \multicolumn{1}{c|}{}                      & \multicolumn{1}{c}{Art} & \multicolumn{1}{c}{Clip} & \multicolumn{1}{c}{Prod} & \multicolumn{1}{c}{Real} & \multicolumn{1}{c}{Avg.} &  & \multicolumn{1}{c}{Real} & \multicolumn{1}{c}{Info} & \multicolumn{1}{c}{Pain} & \multicolumn{1}{c}{Avg.} \\ \midrule
\checkmark &            &   &66.0  &67.3 &64.5 &70.9 &67.2  & &32.0 &31.0 &33.4 &32.1   \\
& \checkmark &  &70.6   &72.5    &67.4    &69.9    &70.1  &    &34.3     &33.5     &34.9   &34.2 \\
& & \checkmark  &65.0   &66.5   &65.6    &71.0    &67.0     &  &34.6    &30.7    &32.8   &32.7  \\
\checkmark & \checkmark & &70.2 &\textbf{73.2} &67.0 &70.6 &70.3 &  &35.7  &35.7 &36.1  &35.8    \\
\checkmark & \checkmark & \checkmark  &\textbf{71.7}   &72.8   &\textbf{68.0}   &\textbf{71.7}  &\textbf{71.1}   &   &\textbf{36.1}  &\textbf{37.3}  &\textbf{36.6}  &\textbf{36.7}  \\ \bottomrule

\end{tabular}
}
\caption{Ablations on Office-Home and the selected three domains on DomainNet. \textit{mix}: mix labeling strategy; \textit{bal}: balanced sampling strategy; \textit{flip}: low-level features.}
\label{ablation}
\end{table}

\section{Ablation and analysis}
We present ablations of MCDA in Table~\ref{ablation} on Office-Home and three domains of DomainNet. Each proposed module contributes to the improvement of the final performance.

\noindent\textbf{Effectiveness of uncertainty for guiding adversarial training.} To validate the uncertainty and mixed labels to train a categorical discriminator $D^k$ that mutually reinforces with target pseudo labels. We show the number of filtered samples below the uncertainty threshold and the corresponding pseudo label accuracy in Figure~\ref{pse_acc}. The results of DomainNet show that during the training, the uncertainty of samples gradually decreases, and more samples pass the threshold. Meanwhile, the accuracy of pseudo labels increases, which justifies our motivation.

\begin{figure}[t]
\centering
    \label{fig:subfig:sourceonly_tsne}
    \includegraphics[width=0.23\textwidth]{samples_filtered_dnet.png}
    \label{fig:subfig:mcda_tsne}
    \includegraphics[width=0.23\textwidth]{pseudo_acc_dnet.png}

\caption{Samples below uncertainty threshold and pseudo label accuracy during training process on DomainNet.}
\label{pse_acc}
\end{figure}

\begin{figure}[t]
\centering

\label{fig:subfig:onefunction}
\includegraphics[width=0.22\textwidth]{home_btlds_clip_sourceonly_dense2.png}
\label{fig:subfig:twofunction}
\includegraphics[width=0.22\textwidth]{home_btlds_clip_cadaflip_dense2.png}

\caption{t-SNE feature visualization on Office-Home-LMT with \textit{Clipart} as the source. 15 classes are sampled for conciseness. Color represents the domain and the digit represents the class. \textit{left}: sourceonly, \textit{right}: MCDA.}
\label{t-sne}
\end{figure}

\begin{figure}[t]
\centering

\includegraphics[width=0.45\textwidth]{aaai23_cam_vis.pdf}

\caption{CAM feature response maps on Office-Home-LMT. The left pervasive one is from sourceonly model while the right class-discriminative one is from MCDA.}
\label{cam}
\end{figure}

\noindent\textbf{Robustness of uncertainty threshold.} We validate the robustness of our model under various uncertainty thresholds $\lambda$ in Table~\ref{thre} for both standard BTDA and BTDA with label shift in Table~\ref{thre}. For main experiments in Table~\ref{btda_all} and~\ref{btlds}, we set $\lambda$ as 0.05. The performance is stable when $\lambda$ is selected from 0.03 to 0.07. The results of Art on Office-Home have a fluctuation range of 0.4\%, and results of Clipart on Office-Home-LMT have a fluctuation range of 1.5\%. These demonstrate the stability and generality of our model for the standard BTDA and label shift setting.

\noindent\textbf{Verification of error theorem under label shift.} We verify our error theorem in equation 1 that if conditional distributions $\Delta_{BTCE}(\hat{Y})$ is sufficiently minimized, the model is robust under label shift in BTDA since the reweighted source error $\| P_{\mathcal{S}}(Y) - P_{\mathcal{T}_j}(Y) \|_1  BER_{P_{\mathcal{S}}}(\hat{Y}\|Y)$ is relatively small. In Table~\ref{btlds}, the \textit{oracle} are results when the discriminator is trained with true labels of source and target but the classifier is trained only with source labels. In this case, the categorical discriminator is trained to minimize categorical distributions as much as possible under ground truth supervision. The \textit{S+T} are results where the classifier is trained with both source and target labels. These two results approximate to each other which verifies the theorem.

\begin{table}[!htbp]
\centering
\setlength{\tabcolsep}{1.5mm}{
\scalebox{1.0}{
\begin{tabular}{@{}lcccccc@{}}
\toprule
\textbf{Thresholds} & 0.01 & 0.03 & 0.05 & 0.07 & 0.09 \\ \midrule
Art      & 71.9    & 71.9    & 71.7    & 72.3  & 71.8   \\
Clipart  & 66.5    & 67.4    & 68.0    & 66.9  & 66.4 \\
\bottomrule
\end{tabular}
}
}
\caption{Accuracy ($\%$) of different uncertainty thresholds for \textit{Art} in Office-Home and \textit{Clipart} in Office-Home-LMT for BTDA (ResNet-50).}

\label{thre}
\end{table}

\noindent\textbf{Essential of domain labels.} Our theoretical formulation in equation 1 does not require domain labels to minimize the target error rate in BTDA. The bound is mainly related to the categorical distribution constraint.
In comparison with methods (i.e., CGCT~\cite{roy2021curriculum} and DCL~\cite{nguyen2021unsupervised}) using domain labels $\dag$ in Table~\ref{btda_all}, our method outperforms previous state-of-the-arts by 0.8\% on Office-31, 1.3\% on Office-Home, and 0.1\% on DomainNet even without domain labels.
This validates our proposition that adaptation can be done without domain labels in BTDA if categorical distributions is sufficiently aligned.

\noindent\textbf{Feature visualization.} To show our method learns a regular and meaningful categorical feature space, we visualize features of last convolution layer with t-SNE in Figure~\ref{t-sne} and CAM~\cite{selvaraju2017grad} in Figure~\ref{cam}. The t-SNE visualization further shows that the sourceonly model generates a hybrid feature space while MCDA produces a more class-discriminative feature space, which corroborates our observation on the cluster assumption and categorical alignment on BTDA. The CAM results shows feature response maps of the sourceonly model is pervasive while those of MCDA are more category-discriminative. This validates that MCDA make the classifier learns more task-relevant features and achieve better categorical alignment. More visualization results are discussed in the supplementary.

\section{Conclusion}
In this paper, we demonstrate that adaptation can be well achieved without domain labels for BTDA only if the categorical distributions are sufficiently aligned, even facing the cross-domain label shift. Following this, we present the mutual conditional domain adaptation framework. We explore the uncertainty-guided mechanism and source-only balanced sampling strategy to train a categorical domain discriminator for efficiently modeling categorical distributions in BTDA. And we explore low-level features to correct the biased classifier. Extensive experimental results demonstrate the state-of-the-art performance of the framework in single target DA and BTDA tasks under label shift.

\section{Acknowledgement}
We appreciate constructive feedback from anonymous reviewers and meta-reviewers. This work is supported by Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program.


\section{Introduction}
\label{sec:intro}

The COVID-19 pandemic has and continues to affect education at all levels for both instructors and students. Courses were quickly moved to an emergency, fully remote, online formats as the pandemic took hold in the early months of 2020. With little notice, no preparation, and limited resources, educators were required to convert entire courses online and shift to remote instructional approaches. Students were forced to enroll in online courses, scramble to obtain the necessary technology, and make decisions about housing, family responsibilities, and more. As a result, the development of emergency online courses by instructors and the learning environment experienced by students differed significantly from traditional online courses.

In this paper, we present some of the challenges and successes of emergency online teaching specifically as they pertain to statistics and data science education. The three areas considered are 1) Technology, 2) Classroom Community and Feedback, and 3) Student-Content Engagement. We begin by summarizing some of the literature in these areas, followed by separate sections for each area of focus. Within each section, we discuss the most successful teaching strategies we found to mitigate these challenges. Finally, we consider ways to integrate these methods into the classroom as courses are moved back in person or offered in hybrid formats. Throughout each section, we consider the impacts on both students and instructors. Although many of the strategies found in this paper overlap with current findings in online education research, increased instructor flexibility remains the heart of the work done to create positive learning environments during the pandemic.

\subsection{Technology}

The use of technology was essential to deliver material to students during the pandemic. With the unplanned migration to remote learning, educators faced several challenges in delivering mathematical and statistical concepts to students. \cite{patricia_aguilera-hermida_college_2020} and \cite{bower_technologymediated_2019} notes how students' lack of confidence in learning technology reduces their engagement in the material and lower students’ learning outcomes. Additionally, educators introducing new technology may have students focus on learning the technology instead of learning the course material \citep{lopez_pandemic_2021}. Lastly, educators noted that reviewing students' work on mathematical derivations posed a challenge virtually, especially if students did not have access to pens and tablets \citep{gilbert_everyone_2021}.

Contrary to the challenges, the use of technology brought new benefits to delivering course material \citep{robinson}. Several researchers noted the benefits of providing live-recorded lectures for their mathematics, statistics, and programming courses \citep{abrahamsson_comparison_2021, delgado_teaching_2021}; moreover, \cite{jones_engaging_2022} found the use of virtual whiteboards to be beneficial in delivering content to students. Educators found innovative ways to deliver their material to students; however, several educators agree that emergency remote learning takes a considerable amount to prepare material and does not have the same effectiveness as in-person instruction \citep{lopez_pandemic_2021}.

\subsection{Classroom Community and Feedback}

A sense of classroom community plays an important role in the student experience. Both sense of community and perceived student learning are strongly related to course satisfaction \citep{baturay}. However, students in science courses tend to feel a lower sense of community with both instructors and classmates as compared to students in non-science courses \citep{young}. A variety of strategies have been suggested to improve student sense of classroom community, including problem-based projects \citep{baturay2} and formative assessment \citep{gikandi}. Discussion boards may also increase a sense of classroom community, especially when posting is required and graded \citep{rovai}. Rovai also finds that stronger feelings of classroom connectedness relate to stronger feelings of satisfaction about educational goals (\citeyear{rovai}).

Little has been written on providing effective feedback in the online classroom, especially with respect to handwritten mathematics and statistics assignments. However, feedback has been shown to promote the regulation of learning and is associated with both improved performance and higher levels of student satisfaction in the online learning environment \citep{espasa}.

\subsection{Student-Content Engagement}

Learner-content interaction is the way in which a student engages with course content to change their understanding or perspective \citep{moore}. Moore describes it as a defining characteristic of education. In a traditional face-to-face course, student-content engagement can be achieved by attending lectures, working on assignments, completing assessments, and more. In an online environment, student-content engagement can also include watching instructional videos and searching for information online \citep{abrami}. Findings show that the instructor's teaching style and visual presence are instrumental in engaging students with the content \citep{everett} and that both synchronous and asynchronous content delivery are effective methods for helping online students access content for critical student-content interaction \citep{banna}. Asynchronous formats specifically, provide students with more time to think critically and reflect on the material \citep{robinson}.

A substantial amount of research exists that compares face-to-face and online strategies for engaging students with content. However, limited information is available on the effectiveness of strategies during emergency online learning; though it is emerging as a result of the COVID-19 pandemic. One such paper provides a discussion on effective engagement strategies, finding that students perceived student–content engagement strategies to be more effective than student–instructor and student–student strategies during emergency online learning \citep{abou}. Discipline specific strategies for online courses are sparse and nearly nonexistent for emergency online learning.

\subsection{Emergency Online Learning}

Despite significant literature on the online classroom, it is not immediately clear how emergency online learning impacts these findings. For example, much of the literature rests on the implicit assumption that students choose the online classroom. Are these ``best practices” the same for emergency remote classrooms in the midst of a global pandemic? How do we adapt pedagogy when students are forced into online learning or experiencing ongoing traumatic events? This paper addresses some of the ways in which we modified our pedagogy to support our students during the COVID-19 pandemic. We utilized existing approaches to online education where possible and made adjustments as necessary based on time constraints and student needs. Given the evolving situation with COVID-19 and many university administrations’ desire to see more online offerings, this is and will continue to be an important question in education research.

\section{Technology}

As educators rushed to teach their courses online, many new challenges arose with delivering material virtually. Instructors and universities worked tirelessly to ensure students could continue their education remotely. This meant addressing challenges associated with the basic necessities of remote learning - ensuring access to things like computers, high-speed internet, video cameras, speakers, microphones, textbooks, and other required course materials. In addition, providing access to non-academic related services was of great concern (e.g., counseling and psychological services, health services, etc.). Furthermore, statistics educators experienced unique challenges in delivering courses due to the need for teaching statistical concepts, teaching programming methods, creating accessible material, and troubleshoot problems for students.

\textbf{Use tablets and various note-taking software to present formulas, demonstrate problem solutions, and the ability to sketch visual aids.} Statistics programs have courses that are both applied and theoretical. Each case requires careful thought around delivering material. Elementary statistics courses introduce students to formulas for computing different probabilities and statistics. While instructors can provide formulas on slides, there is no replacement for hand-writing a solution to a problem and highlighting important concepts. Being able to manipulate a formula or draw a figure is necessary to explain ideas. To accommodate this need, we utilized tablets and various note-taking software. This allowed for the flexibility of easily manipulating a problem and providing visual aids to student questions. Afterwards, notes can be posted online for students to access and review as needed.

\textbf{Provide troubleshooting documentation and utilize accessible online platforms for statistical software.} Elementary statistics courses analyze data using statistical software. Regardless of the software, students need to gain access to or install the software on their computers. Detailed instructions were provided to install software and troubleshoot problems for students. To minimize the burden on students when installing software, a couple of approaches were deployed. We used online platforms such as RStudio Cloud (now called Posit Cloud\nocite{posit}) to teach R\nocite{rstats} without the concern of software or computing issues. Students on some campuses have access to virtual labs, which provide students with a virtual desktop filled with the applications they need to conduct analysis. Lastly, R packages were used to deliver material that students use to learn how to program in R. These packages can be updated to install other packages or data. For elementary statistics courses without a programming component, we took advantage of online resources such as the Rossman and Chance applet collection (\citeyear{rossmanchance}). The use of cloud platforms ensure that students have access to the computing resources needed for completing assignments.

In statistical computing courses, educators needed to find ways to deliver computational and programming concepts to students. Computing courses are easier to deliver online but developing material and having students work on both individual and group assignments were challenging. For many students, a statistical computing course requires a deep understanding of programming with statistical packages. Delivering programming material with current Learning Management Systems (LMS) poses new challenges. Additionally, debugging code via video conferencing or email is difficult since interacting with the computer to identify the error is limited. To address these issues, we utilized several tools designed by Posit for delivering content and troubleshooting code. We used RMarkdown\nocite{rmarkdown} documents to explain statistical computing concepts and provide  the corresponding R code. These documents can be converted to web pages that can be hosted on GitHub\nocite{github}. We also had students work with RMarkdown and send source files when debugging was necessary. This allowed us to collect the corresponding HTML file for assignments. For non-programming courses, Overleaf\nocite{overleaf} provided a nice alternative for Latex documents that allow students to write reports without worrying about the Tex installation process.

\textbf{Provide ‘practice’ assignments so students have the opportunity to get comfortable with new or unfamiliar technology.} In all types of statistics classes, homework and assessments were a significant challenge since we wanted to deliver accessible assignments virtually while maintaining academic integrity. For elementary statistics courses, instructors were able to provide some assignments and exams using online software or assessment tools from within their LMS. More advanced statistics courses that require students to show mathematical derivations of concepts called for alternative approaches such as having students submit scanned documents for homework and exams. This led to challenges with file formats and legibility. To ensure students learned how to scan documents with their smartphones, we had assignments at the beginning of the course with the sole purpose of letting students practice scanning their documents. We then provided feedback to notify students whose scanned documents were in an incorrect file format or were illegible. Accounting for students who have never used smartphone applications to scan documents or submit a file as a portable document file (PDF) is an important consideration. Allowing for practice and multiple attempts for submitting files saved time and frustration for both sides.

\section{Classroom Community and Feedback}

\textbf{Utilize discussion boards to build a sense of classroom community.} A strong sense of classroom community can be difficult to achieve in the online classroom. We were surprised, at first, to find that many students were not interested in speaking to one another in the Zoom classroom, even in small group breakout sessions\nocite{zoom}. Although students may have the opportunity to work together in a face-to-face statistics classroom, there are often few opportunities for deeper classroom discussion, especially in large lecture classes. The collaborative work accomplished in the classroom, such as think-pair-share activities or group work, is difficult to accomplish over Zoom and/or the LMS. Further, math anxiety may play a role in students’ hesitation to work together, even in face-to-face classrooms. As a result, the online statistics classroom may be more isolating than other disciplines.

For the kind of student-student interaction needed for a strong community, we utilized discussion boards. It may be useful to create a student code of conduct for discussion boards and some of our colleagues used this as an opportunity to get students involved in creating their own community expectations. We found that including a student conduct statement in the course syllabus was a good way to set expectations for both discussion boards and the rest of the course.

Discussion boards were graded based primarily on participation to encourage students to get involved in the classroom community in a low-pressure setting. Discussion board topics ranged from informal ``get to know you” type posts to posts designed to get students working collaboratively. Prompts might include asking students to share information such as their major and why they selected it, likes/dislikes of online learning, or extracurricular interests; academic strategies regarding time management or studying techniques; reflections on personal achievements or strengths/weaknesses and how to leverage them; or ‘just-for-fun’ prompts. For example, two recent just-for-fun prompts were ``If you could have any magical powers, what would you choose and how would you use them?” and ``Please share a fun fact (it doesn't have to be about you!)”.

Students are also encouraged to respond to their peers and connect over shared majors, career goals, or hobbies. Students seemed to enjoy these discussion boards and prompts, expressing gratitude for attempting to engage with everyone in the class and fostering a sense of community instead of the ‘normal’ disconnected learning experience found in asynchronous courses. They appreciated seeing the instructor and teaching assistant posting answers to prompts, stating that it strengthens student-instructor connections. Instructors were also able to better understand their students’ perspectives, situations, and backgrounds, particularly in large courses.

Collaborative discussions related to course content might involve asking and answering questions (e.g., through an open Q\&A discussion board) or working on a challenging problem. One strategy is to assign a problem that most students will not be able to solve on their own and have them post an initial approach or solution. For example, the Monty Hall and Birthday problems are both solvable using concepts introduced in our introductory statistics courses, but tend to be challenging for students. As a class or in groups, they go through each other’s posts and are able to reach a solution collaboratively. This activity was initially suggested by a student, and several students mentioned they found it helpful to be able to see other students' approaches to problem-solving. However, it is important to note that in this setting, students often need reassurance upfront that they are not expected to get an answer right away and that you are more interested in the learning process than in ``correctness”.

\textbf{Conduct student check-ins, including asking for feedback regarding the course structure and what is and is not working well.} An additional consequence of remote learning was lack of interaction and nonverbal feedback between students and instructors, where instructors includes both lecturers and teaching assistants. Students often attended virtual class meetings with their cameras off, leaving the instructor to speak into the void. In person, presenters are in constant ``conversation” with the audience, monitoring verbal and nonverbal feedback. In a typical face-to-face conversation, people gain information on agreement and understanding from head and eye movements \citep{kleinke}. Even in a lecture-heavy classroom, these kinds of nonverbal cues provide conversational input which allow the instructor to adjust on the fly. In a virtual classroom where most students have their cameras off, this information is minimized or nonexistent. Further, when there are fewer communication cues available - as is the case in a virtual classroom with minimized point of view - the few available cues have a larger impact than they would otherwise \citep{walther1996}.

The lack of feedback often resulted in minimal direct student-teacher interaction. Less face-to-face time may make it more difficult for students to feel comfortable reaching out to faculty. As a result, we found that fewer students attended office hours. Further, students were unable to ask questions while interacting with asynchronous content. Compounding this problem was the difficulty in providing feedback while grading online. We did see an encouraging increase in student questions during live Zoom class sessions, possibly due to the ability to ask questions anonymously. However, these were difficult to monitor and students often struggled to ask mathematics questions over chat because of limitations related to mathematical notation.

Check-ins can be conducted both privately (student-to-instructor) or on a course discussion board. Questions we found useful are provided here: (1) `What do you like about this class? What are some things that are working for you?' (2) `What is something that you enjoy from one of your other courses that we are not doing in this class?' (3) `What would your ideal course look like right now (given that we are online and at home)? Assume the instructor of your ideal course has unlimited time and resources.' (4) `How are you doing (both in this class and in general)? Are you holding up okay? Is there anything else you'd like to share with me? If you choose to write something, please let me know if it's okay for me to follow up with you about it.' Much of the student feedback referenced throughout this paper came from these check-ins.

Students seemed to really appreciate the discussion board check-ins, commenting that it was nice to see how their classmates were feeling. In private responses to (4), multiple students noted that it meant a lot that someone had taken the time to ask how they were doing. On the other hand, many students chose to share how traumatic the pandemic had been for them, describing feelings of isolation, depression, and hopelessness. Checking in with students who are struggling can be a deeply painful experience and these responses were difficult to read and process. Instead, instructors may find it more manageable to recommend any student counseling services up front, making sure to mention that these services are free or low-cost for students. Students also seem receptive to hearing about their instructor’s use of these services when they were in school, or any other instructor experience that can help to demystify or destigmatize the process.

Another approach is to include a Student Support section in the course syllabus with information and links to campus support services such as Counseling and Psychological Services, Student Health Services, Food Pantry Services, Case Management Services, Title IX Office, and more. This might include brief descriptions of why and how a student would access these services. Having these resources easily accessible within the syllabus communicates to students that not only are we aware of the possibility that they may need to utilize them, but we also care about their well-being. Students may also appreciate your linking to these items on a separate LMS page, especially if your syllabus is fairly long.

In the virtual classroom, it may be more difficult for students to feel comfortable connecting with instructors to ask for assistance. For further student-instructor interaction, we conducted check-ins on how students were doing academically. Each week, students answered a series of prompts including (1) `Write down one thing you learned that you have not thought about before', (2) `What questions or confusions do you have about this week’s content?' and (3) `Additional thoughts or comments about the material or the course'. Making a point to address questions - anonymously - in written materials or videos posted on the LMS helped to encourage students to reach out each week with their concerns. These types of weekly check-ins can also be done informally based on interactions with students via email or during office hours. Requesting questions outside of the classroom gives students an opportunity to think about or work with the material before asking for clarification. The questions then provide a scaffold for what topics may be reviewed or clarified each week. They also give the instructor insight into how to approach these topics in the future.

\section{Student-Content Engagement}

Finding ways to engage students with course content can be a challenging endeavor regardless of the discipline and course format (face-to-face, hybrid, or remote).  This challenge was exacerbated in our online introductory level statistics courses where students were being exposed to unfamiliar terminology, concepts, and applications while being taught from a distance. Long Zoom\nocite{zoom} sessions alone are difficult to pay attention to; add in long lectures riddled with new terms, formulas, and ideas, and students quickly become overwhelmed. To combat these challenges and garner student engagement, we attempted various course structures in both synchronous and asynchronous formats with mixed success.

\textbf{Provide student support and encourage engagement with course material through asynchronous content.} One of the simplest strategies is to record synchronous lectures and provide them to our students for review. This allowed students the flexibility to engage with course meetings outside of regular course time, as well as the ability to rewatch all or part of a discussion. We did not record office hours in order to give students time to freely ask questions regarding course material, as well as to ensure privacy for students needing to discuss other matters.

To further promote flexibility and engagement, we shared asynchronous content such as short video ``micro-lessons”, informational pages, and external website links through the LMS. The primary component of the asynchronous material were micro-lessons. These short topic videos designed for asynchronous viewing were used to teach, review, or demonstrate concepts in a way that was both easy to find (without having to skip through an hour-long recording) and easy to pay attention to. The length of the recordings were 3-20 minutes, with a majority being less than 8 minutes. Keywords were added to video descriptions to make finding topics efficient and straightforward. The keywords intentionally consisted of foundational statistical terms (e.g., variability, mean, distribution, etc.) to help students connect ideas and review specific topics as needed. The recordings also provided an organized system for students to review material in preparation for quizzes and exams, or while working through homework assignments. These same micro-lessons have since been used during “the return to campus” both as complimentary material to face-to-face lectures and/or when instructors are absent due to conferences or illness.

For instructors, creating the micro-lectures was extremely time consuming. In total, the authors of this article created over 400 short recordings.
However, the resulting benefit of the extensive reusability of the recordings, which can be incorporated into both future online and face-to-face learning environments makes the initial time commitment worthwhile. Another added benefit of the micro-lectures is the ability to create a semi-flipped classroom environment where students could watch the recordings ahead of time and the synchronous course meetings (whether online or face-to-face) could then be dedicated in part to running practice problems and answering student questions. This format introduces some new challenges associated with attendance issues and students’ dislike of utilizing breakout rooms for group work. We found that low attendance makes it difficult to utilize breakout rooms and when used, students either left the sessions (by signing out of Zoom) or were hesitant to interact with their classmates in these small group settings.

Students appreciated the asynchronous content with many choosing to highlight their experience with the micro-lectures. In a Statistical Computing course, on average, 71\% of students watched the entire length of the video. For videos containing supplementary material, just 21\% of students watched the full length of the video - indicating that most students will only watch the required videos. Common themes from student feedback were that the short videos were easy to focus on; there was a clear delineation between topics; and videos could easily be paused and rewatched, watched with captions, or even watched at faster/slower speeds (particularly when studying for exams). Students also commented that it was helpful for them to watch the short content videos outside of class time and use the live time to ask questions and review the content through examples. Other supplementary materials provided to students were the use of websites and R Packages that contained code and explanations of statistical concepts. Written summaries and step-by-step examples were also provided. Students commented positively regarding websites that were accessible after the course for use in other classes.

\section{Future Considerations}

\subsection{Technology}

With emergency online learning, instructors were more willing to adopt new technology and deploy new strategies. As we have transitioned back to face-to-face classes, there are several technologies that we may continue to use in the classroom. The creation of videos and the use of online resources provides a repository of teaching materials that can be utilized in subsequent terms. Videos can be employed to create flipped classrooms, allowing class time to be used for discussions on statistical and programming concepts. Additionally, educators are incorporating new technology to manage their classrooms and deliver the material. Several instructors may use GitHub to collect assignments, host course material, and R packages. Instructors may continue using RMarkdown/Quarto\nocite{quarto} to create documents and presentations for their students.

As we move forward, there are several new technologies powered by artificial intelligence that will have an impact on how statistics courses will be taught. With the rise of ChatGPT \citep{openai}, students are able to ask questions and receive responses to both conceptual and programming questions. This presents new opportunities and challenges when teaching statistics and programming to students. While it can be a great tool to help students understand concepts or discover new programming techniques, it may also misguide students when they are not able to identify errors in the responses. We should caution students about using new technology such as ChatGPT as it may provide erroneous results. This technology will continue to improve over time and the Statistics community needs to begin discussing how it will shape the ways in which we educate our students.

\subsection{Classroom Community and Feedback}

The modern student is used to socializing and finding information online. Emergency online teaching gave educators an opportunity to use this as a strength. To foster a sense of classroom community, it can be helpful to encourage community outside of the classroom itself. While in-person study groups can be useful for many students, virtual community spaces allow students to engage in more flexible ways. Instructors may wish to create discussion boards through the LMS or to allow students the freedom to create their own virtual spaces. Instructor-created discussion spaces can include opportunities for students to work together on the material or create connections. These may be especially helpful in large lectures, where students are often unable to interact with many of their peers. Additionally, students may create virtual spaces on other platforms such as Discord servers. Instructors may choose to interact in these spaces to answer student questions, or they may prefer to leave those spaces for students only. Either way, students should be expected to follow the institution’s code of conduct. A classroom code of conduct is also recommended and is an opportunity to get students involved in creating positive spaces.

To give feedback to students, the LMS allows the instructor to grade and provide feedback locally. However, the LMS is often poorly set up for grading mathematics and statistics work. Services like GradeScope\nocite{gradescope} are a more efficient way to provide feedback when grading online, especially for mathematics work. We recommend creating a practice assignment to get students used to submitting work on GradeScope/LMS which can help further streamline the process.

To facilitate feedback from students, regular check-ins may be conducted through the LMS. These can be in the form of short weekly check-ins about the course material, or through mid-term surveys requesting feedback on the course. Weekly surveys can be used to keep track of student learning and areas where students are falling behind; mid-term surveys can be used to check in more broadly with how students feel about the class overall. These give the instructor an opportunity to review areas where students are falling behind and to address any student concerns.

Finally, the classroom community may benefit from the incorporation of health and wellbeing. If time permits, instructors can request relevant offices to deliver presentations in their classrooms, or even take their students on field trips to offices and tutoring centers. Otherwise, instructors may find it helpful to reference these resources in several different places and throughout the term. For example, on the syllabus, through direct links in the LMS, and verbally before and after each important assessment.

\subsection{Student-Content Engagement}

By far, the biggest success for student engagement with course content is providing asynchronous material that can be accessed by students anytime, anywhere. Whether a course is taught fully online, hybrid, or face-to-face, the availability of course materials outside of the classroom has become a critical component of our course designs. For best use, the asynchronous material should be free and easily accessible. Some instructors may prefer creating their own material to ensure perfect alignment with their course organization and/or to have complete control over the content but should be forewarned that it is extremely time consuming and arduous when starting from scratch. Building a repository through each iteration of teaching a course is the best route so as not to become overwhelmed and to ensure high-quality material. Student creation of course content is also encouraged (such as creating a short topic video as a group term-project) but requires careful review for accuracy and permission should be granted from the students to use their videos as part of your course materials.

An alternative to self-creating asynchronous course content, particularly for introductory level courses, is to access the ever-growing availability of online resources and open source textbooks that provide reading, problems, data sets, videos, and more. OpenIntro\nocite{openintro}, MyOpenMath\nocite{openmath}, and CourseKata\nocite{coursekata} are just a few examples of readily available and easily accessible resources for serving as primary or supplementary course material. For courses with a heavier focus on computation using R, a variety of free online textbooks can be found on the Bookdown\nocite{bookdown2} website with books ranging from R Programming to Data Visualization to Machine Learning and Regression.

Finally, though students have the desire to have asynchronous material available, they may not  consistently utilize it. Supplementary or optional asynchronous material allows students to access additional resources as needed. However, if a course is designed so that the asynchronous material is mandatory, instructors should strongly consider assigning at least some portion of the course grade to it. Options that work well for grading access to asynchronous material include participation points, discussion posts, or quizzes (either embedded into the material or separately) which can be easily integrated into the LMS.

\section{Conclusion}

Through several terms and various course formats of emergency online teaching for a range of statistics courses, we consistently found the following strategies created an effective online learning environment for our students.

\begin{enumerate}
    \item Use tablets and various note-taking software for presenting formulas, demonstrating problem solutions, and the ability to sketch visual aids.
    \item Provide troubleshooting documentation and utilize accessible online platforms for statistical software.
    \item Provide ‘practice’ assignments so students have the opportunity to get comfortable with new or unfamiliar technology.
    \item Utilize discussion boards to build a sense of classroom community.
    \item Conduct student check-ins, including asking for feedback regarding the course structure.
    \item Provide student support and encourage engagement with course material through asynchronous content.
\end{enumerate}

\noindent As we have moved back to face-to-face, traditional online, and hybrid courses, these strategies have continued to be implemented with success.

For future research, we are interested in understanding student and faculty perspectives with emergency online statistics courses at the beginning of the pandemic versus online courses as we are moving beyond the pandemic.

\section*{Acknowledgements}

We would like to thank the attendees of our Joint Statistical Meeting 2021 Birds of a Feather Discussion and our colleagues for sharing their own challenges and successes with us. We also appreciate the comments from the reviewers and editors at JSDSE.

\bibliographystyle{apalike}