text:
\section{Introduction}
Real-world classification datasets often suffer from two issues, i.e., label noise~\cite{songLearningNoisyLabels2021} and class imbalance~\cite{heLearningImbalancedData2009}.
On the one hand, label noise often results from the limitation of data generation, e.g., sensor errors~\cite{elhadySystematicSurveySensor2018a} and mislabeling from crowdsourcing workers~\cite{tongxiaoLearningMassiveNoisy2015}.
Label noise misleads the training process of DNNs and degrades the model performance in various aspects~\cite{alganLabelNoiseTypes2020b,zhuClassNoiseVs2004a,frenayClassificationPresenceLabel2014a}.
On the other hand, imbalanced datasets are either naturally long-tailed~\cite{zhaoLongTailDistributionsUnsupervised2012a,vanhornDevilTailsFinegrained2017a} or biased from the real-world distribution due to imperfect data collection~\cite{pavonAssessingImpactClassImbalanced2011a,patelReviewClassificationImbalanced2020a}.
Training with imbalanced datasets usually results in poor classification performance on weakly represented classes~\cite{dongClassRectificationHard2017a,cuiClassBalancedLossBased2019,sinhaClassWiseDifficultyBalancedLoss2021a}.
Even worse, these two issues often coexist in real-world datasets~\cite{johnsonSurveyDeepLearning2019a}.

To prevent the model from memorizing noisy information, many important works have been proposed, including label smoothing~\cite{szegedyRethinkingInceptionArchitecture2016a}, noise adaptation~\cite{goldbergerTrainingDeepNeuralnetworks2017}, importance weighting~\cite{liuClassificationNoisyLabels2014}, GLC~\cite{hendrycksUsingTrustedData2018}, and Co-teach~\cite{hanCoteachingRobustTraining2018a}.
Meanwhile, \cite{dongClassRectificationHard2017a,cuiClassBalancedLossBased2019,sinhaClassWiseDifficultyBalancedLoss2021a,linFocalLossDense2020} propose effective methods to tackle class imbalance.
However, these methods inevitably introduce hyper-parameters (e.g., the weighting factor in~\cite{cuiClassBalancedLossBased2019} and the focusing parameter in~\cite{linFocalLossDense2020}), compounding real-world deployment.

Inspired by recent advances in meta-learning,
some works~\cite{renLearningReweightExamples2018,shuMetaWeightNetLearningExplicit2019a,huLearningDataManipulation2019a,wangOptimizingDataUsage2020b} propose to solve both issues by leveraging a clean and unbiased meta set.
These methods treat instance weights as hyper-parameters and
dynamically update these weights to circumvent hyper-parameter tuning.
Specifically, MWNet~\cite{shuMetaWeightNetLearningExplicit2019a} adopts an MLP with the instance loss as input and the instance weight as output.
Due to the MLP, MWNet has better scalability on large datasets compared with INSW~\cite{huLearningDataManipulation2019a} which assigns each instance with a learnable weight.
Although these methods can handle label noise and class imbalance to some extent, they cannot fully utilize class-level information within each instance, resulting in the potential loss of useful information.
For example, in a three-class classification task, every instance has three logits.
As shown in Figure~\ref{fig:motiv}, every logit corresponds to a class-level gradient flow which stems from the loss function and back-propagates.
These gradient flows represent three kinds of information: "not cat", "dog", and "not bird".
Instance weighting methods~\cite{shuMetaWeightNetLearningExplicit2019a,renLearningReweightExamples2018} alleviate label noise by downweighting all the gradient flows of the instance, which discards three kinds of information simultaneously.
Yet, downweighting the "not bird" gradient flow is a waste of information.
Similarly, in class imbalance scenarios, different gradient flows represent different class-level information.
\begin{wrapfigure}[19]{l}{0.5\textwidth}
    \centering
    \includegraphics[scale=0.19]{figures/intro_motivation.png}
    \caption{Motivation for class-level weighting. For a noisy instance (e.g. cat mislabeled as "dog"), all gradient flows are downweighted by instance weighting. Although the gradient flows for "dog" and "not cat" contain harmful information, the gradient flow for "not bird" is still valuable for training, which should not be downweighted.}
    \label{fig:motiv}
\end{wrapfigure}
Therefore, it is necessary to reweight instances at the class level for better information usage.

To this end, we propose Generalized Data Weighting~(\textbf{GDW}) to tackle label noise and class imbalance by class-level gradient manipulation.
Firstly, we introduce class-level weights to represent the importance of different gradient flows and manipulate the gradient flows with these class-level weights.
Secondly, we impose a zero-mean constraint on class-level weights for stable training.
Thirdly, to efficiently obtain class-level weights, we develop a two-stage weight generation scheme embedded in  bi-level optimization.
As a side note, the instance weighting methods~\cite{renLearningReweightExamples2018,shuMetaWeightNetLearningExplicit2019a,huLearningDataManipulation2019a,wangOptimizingDataUsage2020b} can be considered special cases of GDW when class-level weights within any instance are the same.
In this way, GDW achieves impressive performance improvement in various settings.

To sum up, our contribution is two-fold:
\begin{enumerate}
    %
    \item For better information utilization, we propose GDW, a generalized data weighting method, which better handles label noise and class imbalance. To the best of our knowledge, we are the first to propose single-label class-level weighting on gradient flows.
    %
    \item To obtain class-level weights efficiently, we design a two-stage scheme embedded in a bi-level optimization framework, which does not introduce any extra computational cost.
    %
    To be specific, during the back-propagation we store intermediate gradients, with which we update class-level weights via a gradient descent step.
\end{enumerate}
\section{Related Works}
\subsection{Non-Meta-Learning Methods for Label Noise}
Label noise is a common problem in classification tasks~\cite{zhuClassNoiseVs2004a,frenayClassificationPresenceLabel2014a,alganLabelNoiseTypes2020b}.
To avoid overfitting to label noise, \cite{szegedyRethinkingInceptionArchitecture2016a} propose label smoothing to regularize the model.
\cite{goldbergerTrainingDeepNeuralnetworks2017,vahdatRobustnessLabelNoise2017a} form different models to indicate the relation between noisy instances and clean instances.
\cite{liuClassificationNoisyLabels2014} estimate an importance weight for each instance to represent its value to the model.
\cite{hanCoteachingRobustTraining2018a} train two models simultaneously and let them teach each other in every mini-batch.
However, without a clean dataset, these methods cannot handle severe noise~\cite{renLearningReweightExamples2018}.
\cite{hendrycksUsingTrustedData2018} correct the prediction of the model by estimating the label corruption matrix via a clean validation set, but this matrix is the same across all instances.
Instead, our method generates dynamic class-level weights for every instance to improve training.

\subsection{Non-Meta-Learning Methods for Class Imbalance}
Many important works have been proposed to handle class imbalance~\cite{elkanFoundationsCostsensitiveLearning2001,chawlaSMOTESyntheticMinority2002a,anandApproachClassificationHighly2010a,dongClassRectificationHard2017a,khanCostSensitiveLearningDeep2018,cuiClassBalancedLossBased2019,kangDecouplingRepresentationClassifier2019,linFocalLossDense2020,sinhaClassWiseDifficultyBalancedLoss2021a}.
\cite{chawlaSMOTESyntheticMinority2002a,anandApproachClassificationHighly2010a} propose to over-sample
the minority class and under-sample the majority class.
\cite{elkanFoundationsCostsensitiveLearning2001,khanCostSensitiveLearningDeep2018} learn a class-dependent cost matrix to obtain robust representations for both majority and minority classes.
\cite{dongClassRectificationHard2017a,cuiClassBalancedLossBased2019,linFocalLossDense2020,sinhaClassWiseDifficultyBalancedLoss2021a} design a
reweighting scheme to rebalance the loss for each class.
These methods are quite effective, whereas they need to manually choose loss functions or hyper-parameters.
\cite{liuLargeScaleLongTailedRecognition2019,wangLongtailedRecognitionRouting2020} manipulate the feature space to handle class imbalance while introducing extra model parameters.
\cite{kangDecouplingRepresentationClassifier2019} decouple representation learning and classifier learning on long-tailed datasets, but with extra hyper-parameter tuning.
In contrast, meta-learning methods view instance weights as hyper-parameters and dynamically update them via a meta set to avoid hyper-parameter tuning.

\subsection{Meta-Learning Methods}
With recent development in meta-learning~\cite{lakeHumanlevelConceptLearning2015a,franceschiBilevelProgrammingHyperparameter2018a,liuDARTSDifferentiableArchitecture2018}, many important methods have been proposed to handle label noise and class imbalance via a meta set~\cite{wangLearningModelTail2017,jiangMentorNetLearningDataDriven2018a,renLearningReweightExamples2018,liLearningLearnNoisy2019a,shuMetaWeightNetLearningExplicit2019a,huLearningDataManipulation2019a,wangOptimizingDataUsage2020b,alganMetaSoftLabel2021a}.
\cite{jiangMentorNetLearningDataDriven2018a} propose MentorNet to provide a data-driven curriculum for the base network to focus on correct instances.
To distill effective supervision, \cite{zhangDistillingEffectiveSupervision2020a} estimate pseudo labels for noisy instances with a meta set.
To provide dynamic regularization, \cite{vyasLearningSoftLabels2020b,alganMetaSoftLabel2021a} treat labels as learnable parameters and adapt them to the model's state.
Although these methods can tackle label noise, they introduce huge amounts of learnable parameters and thus cannot scale to large datasets.
To alleviate class imbalance, \cite{wangLearningModelTail2017} describe a method to learn from long-tailed datasets.
Specifically, \cite{wangLearningModelTail2017} propose to encode meta-knowledge into a meta-network and model the tail classes by transfer learning.

\begin{table}
  \caption{Related works comparison.
  %
"Noise" and "Imbalance" denote whether the method can solve label noise and class imbalance.
"Class-level" denotes whether the method utilizes class-level information in each instance, and "Scalability" denotes whether the method can scale to large datasets.}
  \label{tab: comparison_methods}
  \scalebox{0.6}{
  \centering
  \begin{tabular}{@{}c c c c c c c c c c c@{}}
    \toprule
     & Focal \cite{linFocalLossDense2020} & Balanced \cite{cuiClassBalancedLossBased2019} & Co-teaching \cite{hanCoteachingRobustTraining2018a} & GLC \cite{hendrycksUsingTrustedData2018} & L2RW \cite{renLearningReweightExamples2018} & INSW \cite{huLearningDataManipulation2019a} & MWNet \cite{shuMetaWeightNetLearningExplicit2019a} & Soft-label \cite{vyasLearningSoftLabels2020b} & Gen-label \cite{alganMetaSoftLabel2021a}& \textbf{GDW} \\
    \midrule
    Noise & \XSolid & \XSolid & \Checkmark & \Checkmark & \Checkmark & \Checkmark &\Checkmark &\Checkmark &\Checkmark &\Checkmark     \\
    Imbalance & \Checkmark& \Checkmark& \XSolid & \XSolid & \Checkmark &  \Checkmark & \Checkmark & \XSolid & \XSolid & \Checkmark     \\
    Class-level &  \XSolid & \XSolid &  \XSolid & \XSolid &  \XSolid & \XSolid &  \XSolid & \Checkmark &  \Checkmark & \Checkmark  \\
    Scalability & \Checkmark& \Checkmark& \Checkmark & \Checkmark & \Checkmark &  \XSolid & \Checkmark & \XSolid & \XSolid & \Checkmark     \\
    \bottomrule
  \end{tabular}
  }
\end{table}

Furthermore, many meta-learning methods propose to mitigate the two issues by reweighting every instance~\cite{renLearningReweightExamples2018,saxenaDataParametersNew2019a,shuMetaWeightNetLearningExplicit2019a,huLearningDataManipulation2019a,wangOptimizingDataUsage2020b}.
\cite{saxenaDataParametersNew2019a} equip each instance and each class with a learnable parameter to govern their importance.
By leveraging a meta set, \cite{renLearningReweightExamples2018,shuMetaWeightNetLearningExplicit2019a,huLearningDataManipulation2019a,wangOptimizingDataUsage2020b} learn instance weights and model parameters via bi-level optimization to tackle label noise and class imbalance.
\cite{renLearningReweightExamples2018} assign weights to training instances only based on their gradient directions.
Furthermore, \cite{huLearningDataManipulation2019a} combine reinforce learning and meta-learning, and treats instance weights as rewards for optimization.
However, since each instance is directly assigned with a learnable weight, INSW can not scale to large datasets.
Meanwhile, \cite{shuMetaWeightNetLearningExplicit2019a,wangOptimizingDataUsage2020b} adopt a weighting network to output weights for instances and use bi-level optimization to jointly update the weighting network parameters and model parameters.
Although these methods handle label noise and class imbalance by reweighting instances, a scalar weight for every instance cannot capture class-level information, as shown in Figure~\ref{fig:motiv}.
Therefore, we introduce class-level weights for different gradient flows and adjust them to better utilize class-level information.

We show the differences between GDW and other related methods in Table \ref{tab: comparison_methods}.

\section{Method}
\label{method}
\subsection{Notations}
In most classification tasks, there is a training set $D_{train}=\{(x_i, y_i)\}_{i=1}^N$ and we assume there is also a clean unbiased meta set $D_{meta}=\{(x_i^v, y_i^v)\}_{i=1}^M$.
We aim to alleviate label noise and class imbalance in  $D_{train}$ with $D_{meta}$.
The model parameters are denoted as $\boldsymbol \theta$, and the number of classes is denoted as $C$.

\subsection{Class-level Weighting by Gradient Manipulation}
\label{subsec:gradManip}
To utilize class-level information, we learn a class-level weight for every gradient flow instead of a scalar weight for all $C$ gradient flows in~\cite{shuMetaWeightNetLearningExplicit2019a}.
Denote $\mathcal{L}$ as the loss of any instance.
Applying the chain rule, we unroll the gradient of $\mathcal{L}$ w.r.t. $\boldsymbol \theta$ as
\begin{equation}
    \label{eq:chainRule}
    \nabla_{\boldsymbol \theta} \mathcal{L}= \frac{\partial \mathcal{L}}{\partial \mathbf{\boldsymbol \theta}} = \frac{\partial \mathcal{L}}{\partial \mathbf{l}} \frac{\partial \mathbf{l}}{\partial \mathbf{\boldsymbol \theta}} \doteq \mathbf{D}_{1}\mathbf{D}_{2},
\end{equation}
where $\mathbf{l}\in\mathbb{R}^C$ represents the predicted logit vector of the instance.
We introduce class-level weights $\boldsymbol \omega \in\mathbb{R}^C$ and denote the $j^{th}$ component of $\boldsymbol \omega$ as $\boldsymbol \omega_j$.
To indicate the importance of every gradient flow, we perform an element-wise product $f_{\boldsymbol \omega}(\cdot)$ on $\mathbf{D}_1$ with $\boldsymbol \omega$.
After this manipulation, the gradient becomes
\begin{equation}
    \label{eq:newGrad}
    f_{\boldsymbol \omega}\left(\nabla_{\boldsymbol \theta} \mathcal{L}\right) \doteq \left(\boldsymbol \omega \otimes \frac{\partial \mathcal L}{\partial \mathbf{l}}\right) \frac{\partial \mathbf{l}}{\partial \boldsymbol \theta} = \left(\boldsymbol \omega \otimes \mathbf{D}_1\right) \mathbf{D}_2 \doteq \mathbf{D}'_1\mathbf{D}_2,
\end{equation}
where $\otimes$ denotes the element-wise product of two vectors.
Note that $\boldsymbol \omega_j$ represents the importance of the $j^{th}$ gradient flow.
Obviously, instance weighting is a special case of GDW when elements of $\boldsymbol \omega$ are the same.
Most classification tasks~\cite{howardMobileNetsEfficientConvolutional2017a,qinRethinkingSoftmaxCrossEntropy2020a,zhaoBetterAccuracyefficiencyTradeoffs2021a} adopt the \textit{Softmax-CrossEntropy} loss.
In this case, we have $\mathbf{D}_1 = \mathbf p - \mathbf y$, where $\mathbf p \in \mathbb{R}^C$ denotes the probability vector output by \textit{softmax} and $\mathbf y\in\mathbb{R}^C$ denotes the one-hot label of the instance~(see Appendix~\ref{section: Appendx_A} for details).

As shown in Figure~\ref{fig:motiv}, for a noisy instance (e.g., cat mislabeled as "dog"), instance weighting
methods assign a low scalar weight to all gradient flows of the instance.
Instead, GDW assigns class-level weights to different gradient flows by leveraging the meta set.
In other words, GDW tries to downweight the gradient flows for "dog" and "not cat", and upweight the gradient flow for "not bird".
Similarly, in imbalance settings, different gradient flows have different class-level information.
Thus GDW can also better handle class imbalance by adjusting the importance of different gradient flows.

\subsection{Zero-mean Constraint on Class-level Weights}
\label{subsec:restOn}
To retain the \textit{Softmax-CrossEntropy} loss structure, i.e. the $\boldsymbol p -\boldsymbol y$ form, after the manipulation, we impose a zero-mean constraint on $\mathbf{D}'_1$.
That is, we analyze the $j^{th}$ element of $\mathbf{D}'_1$~(see Appendix \ref{subsection: Appendix_B.1} for details):
\begin{align}
    \boldsymbol{\omega}_j(\boldsymbol{p}_j-\mathbf{y}_j) =& \boldsymbol{\omega}_t\left({\boldsymbol{p}'_j} - {\mathbf{y}_j}\right) + \left(\sum_k{{\boldsymbol{\omega}_k} {\boldsymbol{p}_k}}-\boldsymbol{\omega}_t\right){\boldsymbol{p}'_j}\label{eq:grad1'},
\end{align}
where $\boldsymbol{p}'_j \doteq \frac{\boldsymbol{\omega}_j\boldsymbol{p}_j}{\sum_k \boldsymbol{\omega}_k\boldsymbol{p}_k}$ is the weighted probability, and $\boldsymbol{\omega}_t$ denotes the class-level weight at the target (label) position.
We observe that the first term in Eq.~(\ref{eq:grad1'}) satisfies the structure of the gradient of the \textit{Softmax-CrossEntropy} loss, and thus propose to eliminate the second term which messes the structure.
Specifically, we let
\begin{equation}
    \label{eq:constraint}
     \sum_k{{\boldsymbol{\omega}_k} {\boldsymbol{p}_k}} - \boldsymbol{\omega}_t = 0\Rightarrow \boldsymbol{\omega}_t = \frac{\sum_{j\neq t}\boldsymbol{\omega}_j \boldsymbol{p}_j}{1 - \boldsymbol{p}_t},
\end{equation}
where $\boldsymbol{p}_t$ is the probability of the target class.
Note that $\sum_j \boldsymbol{\omega}_j\mathbf{y}_j = \boldsymbol{\omega}_t$, and thus we have
\begin{equation}
    \label{eq:constraint2}
    \sum_j\boldsymbol{\omega}_j(\boldsymbol{p}_j-\mathbf{y}_j) = 0.
\end{equation}
This restricts the mean of $\mathbf{D}'_1$ to be zero.
Therefore,  we name this constraint as the \textbf{zero-mean constraint}.
With this, we have
\begin{equation}
    \label{eq:d1prime}
\mathbf D_1'= \boldsymbol{\omega}_t \left( \boldsymbol{p}' - \mathbf y\right).
\end{equation}
Eq. (\ref{eq:d1prime}) indicates that $\boldsymbol \omega$ adjust the gradients in two levels, i.e., instance level and class level.
Namely, the scalar $\boldsymbol{\omega}_t$ acts as the instance-level weight in previous instance weighting methods~\cite{renLearningReweightExamples2018,shuMetaWeightNetLearningExplicit2019a,huLearningDataManipulation2019a,wangOptimizingDataUsage2020b}, and the $\boldsymbol \omega_j$'s are the class-level weights manipulating gradient flows by adjusting the probability from $\mathbf p$ to $\mathbf p'$.

\subsection{Efficient Two-stage Weight Generation Embedded in Bi-level Optimization}
\label{subsec:effiBilevel}

In this subsection, we first illustrate the three-step bi-level optimization framework in \cite{shuMetaWeightNetLearningExplicit2019a}.
Furthermore, we embed a two-stage scheme in the bi-level optimization framework to efficiently obtain class-level weights, with which we manipulate gradient flows and optimize model parameters.

\textbf{Three-step Bi-level Optimization.}
Generally, the goal of classification tasks is to obtain the optimal model parameters $\boldsymbol \theta^*$ by minimizing the average loss on $D_{train}$, denoted as $\frac{1}{N}\sum_{i=1}^N l_{train}(x_i, y_i;\boldsymbol \theta)$.
As an instance weighting method, \cite{shuMetaWeightNetLearningExplicit2019a} adopt a three-layer MLP parameterized by $\boldsymbol \phi$ as the weighting network and take the loss of the $i^{th}$ instance as input and output a scalar weight $\omega_i$.
Then $\boldsymbol \theta^*$ is optimized by minimizing the instance-level weighted training loss:
\begin{equation}
    \label{eq:metaTheta}
    \boldsymbol \theta^*(\boldsymbol \phi) = \mathop{\arg\min}_{\boldsymbol \theta} \frac{1}{N}\sum_{i=1}^N\omega_i(\boldsymbol \phi)l_{train}(x_i, y_i;\boldsymbol \theta).
\end{equation}
To obtain the optimal $\omega_i$, they propose to use a meta set as meta-knowledge and minimize the meta-loss to obtain $\boldsymbol \phi^*$:
\begin{equation}
    \label{eq:metaPhi}
    \boldsymbol \phi^* = \mathop{\arg\min}_{\boldsymbol \phi}\frac{1}{M}\sum_{i=1}^M l_{val}(x_i^v, y_i^v;\boldsymbol \theta^*(\boldsymbol \phi)).
\end{equation}
Since the optimization for $\boldsymbol \theta^*(\boldsymbol \phi)$ and $\boldsymbol \phi^*$ is nested, they adopt an online strategy to update $\boldsymbol \theta$ and $\boldsymbol \phi$ with a three-step optimization loop for efficiency.
Denote the two sets of parameters at the $\tau^{th}$ loop as $\boldsymbol \theta _\tau$ and $\boldsymbol \phi _\tau$ respectively, and then the three-step loop is formulated as:
\begin{enumerate}[label=\textbf{Step \arabic*}]
    \item Update $\boldsymbol \theta_{\tau-1}$ to $\hat {\boldsymbol \theta}_\tau(\boldsymbol \phi)$ via an SGD step on a mini-batch training set by Eq. (\ref{eq:metaTheta}).
    \item With $\hat{\boldsymbol \theta}_\tau (\boldsymbol \phi)$, update $\boldsymbol \phi _{\tau-1}$ to $\boldsymbol \phi _\tau$ via an SGD step on a mini-batch meta set by Eq. (\ref{eq:metaPhi}).
    \item With $\boldsymbol \phi _\tau$, update $\boldsymbol \theta_{\tau-1}$ to $\boldsymbol \theta_\tau$ via an SGD step on the same mini-batch training set by Eq. (\ref{eq:metaTheta}).
\end{enumerate}
Instance weights in \textbf{Step 3} are better than those in \textbf{Step 1}, and thus are used to update $\boldsymbol \theta_{\tau-1}$.
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/flow_chart.png}
    \caption{Two-stage Weight Generation. "BP" denotes the back-propagation in \textbf{Step 2} of the bi-level optimization framework. $\mathbf g$ denotes the intermediate gradients w.r.t. $\boldsymbol \omega$. $\ominus$ denotes the minus operator. Note that $\boldsymbol \omega$ is the first-stage (instance-level) weight and $\boldsymbol \omega'$ is the second-stage (class-level) weight.}
    \label{fig:flowChart}
\end{figure}

\textbf{Two-stage Weight Generation.}
To guarantee scalability, we apply the same weighting network in \cite{shuMetaWeightNetLearningExplicit2019a} to obtain weights.
To efficiently train $\boldsymbol \phi$ and $\boldsymbol \theta$, we also adopt the three-step bi-level optimization framework.
Moreover, we propose an efficient two-stage scheme embedded in \textbf{Step 1-3} to generate class-level weights.
This process does not introduce any extra computational cost compared to \cite{shuMetaWeightNetLearningExplicit2019a}.
We keep the notations of $\boldsymbol \theta_\tau$ and $\boldsymbol \phi_\tau$ unchanged.

The first stage is embedded in \textbf{Step 1}.
Explicitly, we obtain the first-stage class-level weights $\boldsymbol \omega_i = \omega_i \boldsymbol 1$, by cloning the output of the weighting network for $C$ times.
Then we leverage the cloned weights $\boldsymbol \omega_i$ to manipulate gradients and update $\boldsymbol \theta$ with a mini-batch of training instances:
\begin{equation}
    \label{eq:thetaHat}
    \hat {\boldsymbol\theta}_\tau \left(\boldsymbol \phi _{\tau-1}\right) \leftarrow \boldsymbol \theta_{\tau-1} - \eta_{\boldsymbol \theta} \frac{1}{n}\sum_{i=1}^{n}f_{\boldsymbol \omega _i\left(\boldsymbol \phi _{\tau-1}\right)} \left(\nabla_{\boldsymbol \theta} l_{train}(x_i, y_i;\boldsymbol \theta_{\tau-1})\right),
\end{equation}
where $n$ is the mini-batch size, $\eta_{\boldsymbol \theta}$ is the learning rate of $\boldsymbol \theta$, and $f_{\boldsymbol \omega _i\left(\boldsymbol \phi _{\tau - 1} \right)}(\cdot)$ is the gradient manipulation operation defined in Eq. (\ref{eq:newGrad}).

The second stage is embedded in \textbf{Step 2} and \textbf{Step 3}.
Specifically in \textbf{Step 2}, GDW optimizes $\boldsymbol \phi$ with a mini-batch meta set:
\begin{equation}
    \label{eq:phi}
    \boldsymbol \phi _{\tau} \leftarrow \boldsymbol \phi _{\tau-1} - \eta_{\boldsymbol \phi} \frac{1}{m} \sum_{i=1}^{m} \nabla_{\boldsymbol \phi _{\tau-1}} l_{meta}(x_i^v, y_i^v;\hat{\boldsymbol \theta} _\tau(\boldsymbol \phi_{\tau-1})),
\end{equation}
where $m$ is the mini-batch size and $\eta_{\boldsymbol \phi}$ is the learning rate of $\boldsymbol \phi$.
During the back-propagation in updating $\boldsymbol \phi_\tau$, GDW generates the second-stage weights using the intermediate gradients $\mathbf g_i$ on $\boldsymbol \omega_i$.
Precisely,
\begin{equation}
    \label{eq:newOmega}
    \boldsymbol \omega'_i = \boldsymbol \omega_i - \eta_{\boldsymbol\omega}  \rm{clip}(\frac{\mathbf{g_i}}{\|\mathbf{g_i}\|_1}, -c, c),
\end{equation}
where $\mathbf{g_i}$ represents $\frac{1}{m}\sum_{i=1}^m\nabla_{\boldsymbol\omega_i} l_{meta}(x_i^v, y_i^v;\hat{\boldsymbol \theta}_\tau(\boldsymbol \phi_{\tau-1}))$ and $c=0.2$ denotes the clip parameter.
Then we impose the zero-mean constraint proposed in Eq. (\ref{eq:constraint}) on $\boldsymbol \omega'_i$, which is later used in \textbf{Step 3} to update $\boldsymbol \theta_{\tau-1}$.
Note that the two-stage weight generation scheme does not introduce any extra computational cost compared to MWNet because this generation process only utilizes the intermediate gradients during the back-propagation.
In \textbf{Step 3}, we use $\boldsymbol \omega'_i$ to manipulate gradients and update the model parameters $\boldsymbol \theta_{\tau-1}$:
\begin{equation}
    \label{eq:newTheta}
    \boldsymbol \theta_\tau \leftarrow\boldsymbol \theta_{\tau-1} - \eta_{\boldsymbol \theta} \frac{1}{n} \sum_{i=1}^{n} f_{\boldsymbol \omega'_{i}}\left(\nabla_{\boldsymbol \theta} l_{train}(x_i, y_i;\boldsymbol \theta_{\tau-1})\right).
\end{equation}
The only difference between \textbf{Step 1} and \textbf{Step 3} is that we use $\boldsymbol \omega'_i$ instead of the cloned output of the weighting network $\boldsymbol \omega_i$ to optimize $\boldsymbol \theta$.
Since we only introduce $\boldsymbol \phi$ as extra learnable parameters, GDW can scale to large datasets.
We summarize GDW in Algorithm \ref{alg:GDW}.
Moreover, we visualize the two-stage weight generation process in Figure~\ref{fig:flowChart} for better demonstration.

\begin{algorithm}
\caption{Generalized Data Weighting via Class-Level Gradients Manipulation}\label{alg:GDW}
\hspace*{\algorithmicindent} \textbf{Input:} Training set: $D_{train}$, Meta set: $D_{meta}$, batch size $n,m$, \# of iterations $T$ \\
\hspace*{\algorithmicindent} Initial model parameters: $\boldsymbol \theta_0$, initial weighting network parameters: $\boldsymbol \phi_0$\\
\hspace*{\algorithmicindent} \textbf{Output:} Trained model: $\boldsymbol \theta_T$
\begin{algorithmic}[1]
\For{$\tau \leftarrow1$ \textbf{to} $T$}

\State $\{x_i, y_i\}_{i=1}^n\leftarrow$ SampleFrom$(D_{train})$
\State $\{x_i^v, y_i^v\}_{i=1}^m\leftarrow$ SampleFrom$(D_{meta})$
\State Generate $\boldsymbol \omega_i$ from $\mathcal{L}_i$ via the weighting network parameterized by $\boldsymbol \phi _{\tau-1}$
\State Manipulate gradients by Eq. (\ref{eq:newGrad}) and update $\hat{\boldsymbol \theta}_\tau$ by Eq. (\ref{eq:thetaHat})
\State Update $\boldsymbol \phi_\tau$ by Eq. (\ref{eq:phi});
\State Update $\boldsymbol \omega_i$ to $\boldsymbol \omega'_i$ by Eq. (\ref{eq:newOmega}) and constrain $\boldsymbol \omega'_i$ by Eq. (\ref{eq:constraint})
\State Manipulate gradients with $\boldsymbol \omega'_i$ by Eq. (\ref{eq:newGrad}) and update $\boldsymbol \theta_\tau$ by Eq. (\ref{eq:newTheta})
\EndFor
\end{algorithmic}
\end{algorithm}

\section{Experiments}
We conduct extensive experiments on classification tasks to examine the performance of GDW.
We compare GDW with other methods in the label noise setting and class imbalance setting in Section~\ref{sec:noise} and Section~\ref{sec:imbalance}, respectively.
Next, we perform experiments on the real-world dataset Clothing1M~\cite{tongxiaoLearningMassiveNoisy2015} in Section~\ref{sec:real-world}.
We conduct further experiments to verify the performance of GDW in the mixed setting, i.e. the coexistence of label noise and class imbalance~(see Appendix~\ref{section: Appendix_F} for details).

\subsection{Label Noise Setting}
\label{sec:noise}
\noindent \textbf{Setup.} Following \cite{shuMetaWeightNetLearningExplicit2019a}, we study two settings of label noise: a) Uniform noise: every instance's label uniformly flips to other class labels with probability $p$; b) Flip noise: each class randomly flips to another class with probability $p$.
Note that the probability $p$ represents the noise ratio.
We randomly select $100$ clean images per class from CIFAR10~\cite{krizhevskyLearningMultipleLayers2009} as the meta set ($1000$ images in total).
Similarly, we select a total of $1000$ images from CIFAR100 as its meta set.
We use ResNet-32~\cite{heDeepResidualLearning2016} as the classifier model.

\noindent \textbf{Comparison methods.} We mainly compare GDW with meta-learning methods: 1) L2RW~\cite{renLearningReweightExamples2018}, which assigns weights to instances based on gradient directions;
2) INSW~\cite{huLearningDataManipulation2019a}, which derives instance weights adaptively from the meta set;
3) MWNet~\cite{shuMetaWeightNetLearningExplicit2019a}; 4) Soft-label~\cite{vyasLearningSoftLabels2020b}, which learns a label smoothing parameter for every instance; 5) Gen-label~\cite{alganMetaSoftLabel2021a}, which generates a meta-soft-label for every instance.
We also compare GDW with some traditional methods: 6) BaseModel, which trains ResNet-32 on the noisy training set; 7) Fine-tuning, which uses the meta set to fine-tune the trained model in BaseModel; 8) Co-teaching~\cite{hanCoteachingRobustTraining2018a}; 9) GLC~\cite{hendrycksUsingTrustedData2018}.

\noindent \textbf{Training.} Most of our training settings follow \cite{shuMetaWeightNetLearningExplicit2019a} and we use the cosine learning rate decay schedule~\cite{loshchilovSGDRStochasticGradient2016} for a total of $80$ epochs for all methods.
See Appendix \ref{section: Appendix_C} for details.

\begin{table}
	\centering
	\caption{Test accuracy on CIFAR10 and CIFAR100 with different uniform noise ratios.}
	\label{tab:unif_noise_table}
	\scalebox{0.86}{
	\begin{tabular}{ccccccc}
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		\multirow{2}*{Dataset} & \multicolumn{3}{c}{CIFAR10} &  \multicolumn{3}{c}{CIFAR100} \\
		\cmidrule(lr){2-4}
        \cmidrule(lr){5-7}

		 & \multicolumn{1}{c}{$0\%$} & \multicolumn{1}{c}{$40\%$}  & \multicolumn{1}{c}{$60\%$}  & \multicolumn{1}{c}{$0\%$}  & \multicolumn{1}{c}{$40\%$}  & \multicolumn{1}{c}{$60\%$}  \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		BaseModel & $92.73 \pm 0.37$ & $84.38 \pm 0.32$ & $77.92 \pm 0.29$ & $70.42 \pm 0.54$ & $57.28 \pm 0.80$ & $46.86 \pm 1.54$ \\
		Fine-tuning  & $92.77 \pm 0.37$ & $84.73 \pm 0.47$ & $78.41 \pm 0.31$ & $70.52 \pm 0.57$ & $57.38 \pm 0.87$ & $47.06 \pm 1.47$ \\
		Co-teaching  & $91.54 \pm 0.39$ & $85.26 \pm 0.56$ & $78.90 \pm 6.64$ & $68.33 \pm 0.13$ & \uline{$59.58 \pm 0.83$} & $37.74 \pm 2.60$ \\
		GLC  & $90.85 \pm 0.22$ & $86.12 \pm 0.54$ & \uline{$81.55 \pm 0.60$} & $65.05 \pm 0.59$ & $56.99 \pm 0.82$ & $41.74 \pm 1.98$ \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		L2RW  & $89.70 \pm 0.50$ & $84.66 \pm 1.21$ & $79.98 \pm 1.18$ & $63.40 \pm 1.31$ & $47.06 \pm 4.84$ & $36.02 \pm 2.17$ \\
		INSW  & $92.70 \pm 0.57$ & $84.88 \pm 0.64$ & $78.77 \pm 0.82$ & $70.52 \pm 0.39$ & $57.11 \pm 0.66$ & $48.00 \pm 1.16$ \\
		MWNet  & \textbf{92.95} $\pm$ \textbf{0.33} & $86.46 \pm 0.31$ & $81.14 \pm 0.94$ & \uline{$\textbf{70.64} \pm \textbf{0.31}$} & {$58.37 \pm 0.33$} & \uline{$50.21 \pm 2.98$} \\
		Soft-label  & $92.63 \pm 0.27$ & \uline{$86.52 \pm 0.10$} & $80.94 \pm 0.25$ & $70.50 \pm 0.44$ & $57.48 \pm 0.43$ & $48.18 \pm 0.89$ \\
		Gen-label  & $92.56 \pm 0.56$ & $84.68 \pm 0.57$ & $78.32 \pm 0.94$ & $70.46 \pm 0.37$ & $57.86 \pm 0.50$ & $48.08 \pm 0.98$ \\
		\textbf{GDW}  & \uline{\textbf{92.94} $\pm$ \textbf{0.15}} & \textbf{88.14} $\pm$ \textbf{0.35} & \textbf{84.11} $\pm$ \textbf{0.21} & \textbf{70.65} $\pm$ \textbf{0.52} & \textbf{59.82} $\pm$ \textbf{1.62} & \textbf{53.33} $\pm$ \textbf{3.70} \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
	\end{tabular}
	}
\end{table}

\begin{table}
	\centering
	\caption{Test accuracy on CIFAR10 and CIFAR100 with different flip noise ratios.}
	\label{tab:flip_noise_table}
	\scalebox{0.86}{
	\begin{tabular}{ccccccc}
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		\multirow{2}*{Dataset} & \multicolumn{3}{c}{CIFAR10} &  \multicolumn{3}{c}{CIFAR100} \\
		\cmidrule(lr){2-4}
        \cmidrule(lr){5-7}

		 & \multicolumn{1}{c}{$0\%$} & \multicolumn{1}{c}{$20\%$}  & \multicolumn{1}{c}{$40\%$}  & \multicolumn{1}{c}{$0\%$}  & \multicolumn{1}{c}{$20\%$}  & \multicolumn{1}{c}{$40\%$}  \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		BaseModel & $92.73 \pm 0.37$ &$ 90.14 \pm 0.35$  &  $81.20 \pm 0.93$ &  $70.42 \pm 0.54$ & $64.96 \pm 0.16$ & $49.83 \pm 0.82$ \\
		Fine-tuning  & $92.77 \pm 0.37$  & $90.15\pm 0.36$ & $81.53 \pm 0.96$ & $70.52 \pm 0.57$ & $65.02 \pm 0.22$ & $50.23 \pm 0.71$ \\
		Co-teaching  & $91.54 \pm 0.39$ & $89.27 \pm 0.24$ & $69.77 \pm 3.97$ & $68.33 \pm 0.13$ & $62.96 \pm 0.73$ & $42.54 \pm 1.68$ \\
		GLC  & $90.85 \pm 0.22$ & \uline{$90.22 \pm 0.13$} & \textbf{89.74} $\pm$ \textbf{0.19} & $65.05 \pm 0.59$ & $64.11 \pm 0.40$ & \textbf{63.11} $\pm$ \textbf{0.93} \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		L2RW  & $89.70 \pm 0.50$ & $88.21 \pm 0.49$ & $82.90 \pm 1.27$ &  $63.40 \pm 1.31$ & $55.27 \pm 2.27$ & $45.41 \pm 2.53$ \\
		INSW  & $92.70 \pm 0.57$ &  $89.90 \pm 0.45$ & $80.09 \pm 2.00$ &$70.52 \pm 0.39$ & \uline{$65.32 \pm 0.27$} & $50.13 \pm 0.39$ \\
		MWNet  & \textbf{92.95} $\pm$ \textbf{0.33} &  $89.93 \pm 0.17$ & $85.55 \pm 0.82$ &\uline{$\textbf{70.64} \pm \textbf{0.31}$} & $64.72 \pm 0.68$ & $50.62 \pm 0.46$ \\
		Soft-label  & $92.63 \pm 0.27$ & $90.17 \pm 0.47$ & $85.52 \pm 0.78$ & $70.50 \pm 0.44$ & $65.20 \pm 0.45$ & $50.97 \pm 0.41$ \\
		Gen-label      & $92.56 \pm 0.56$ & $90.18 \pm 0.13$ & $80.93 \pm 1.29$ & $70.46 \pm 0.37$ & $64.94 \pm 0.53$ & $49.93 \pm 0.55$ \\
		\textbf{GDW}  & \uline{\textbf{92.94} $\pm$ \textbf{0.15}} & \textbf{91.05} $\pm$ \textbf{0.26} & \uline{$87.70 \pm 0.37$} & \textbf{70.65} $\pm$ \textbf{0.52} & \textbf{65.41} $\pm$ \textbf{0.75} & \uline{$52.44 \pm 0.79$} \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
	\end{tabular}
	}
\end{table}

\noindent \textbf{Analysis.} For all experiments, we report the mean and standard deviation over $5$ runs in Table \ref{tab:unif_noise_table} and Table \ref{tab:flip_noise_table}, where
the best results are in \textbf{bold} and the second-best results are marked by underlines.
First, we can observe that GDW outperforms nearly all the competing methods in all noise settings except for the $40\%$ flip noise setting.
Under this setting, GLC estimates the label corruption matrix well and thus performs the best, whereas the flip noise assumption scarcely holds in real-world scenarios.
Note that GLC also performs much better than MWNet under the $40\%$ flip noise setting as reported in \cite{shuMetaWeightNetLearningExplicit2019a}.
Besides, under all noise settings, GDW has a consistent performance gain compared with MWNet, which aligns with our motivation in Figure \ref{fig:motiv}.
Furthermore, as the ratio increases from $40\%$ to $60\%$ in the uniform noise setting, the gap between GDW and MWNet increases from $1.68\%$ to $2.97\%$ in CIFAR10 and $1.45\%$ to $3.12\%$ in CIFAR100.
Even under $60\%$ uniform noise, GDW still has low test errors in both datasets and achieves more than $3\%$ gain in CIFAR10 and $6\%$ gain in CIFAR100 compared with the second-best method.
Last but not least, GDW outperforms Soft-label and Gen-label in all settings.
One possible reason is that manipulating gradient flows is a more direct way to capture class-level information than learning labels.

\begin{figure}
\centering
\begin{minipage}[t]{.5\textwidth}
  \centering
    \captionsetup{width=.95\linewidth}
    \includegraphics[width=0.8\columnwidth]{figures/clean_and_noise_w_cifar10.png}
    \caption{Class-level target weight~($\boldsymbol{\omega}_t$) distribution on CIFAR10 under $40\%$ uniform noise.
    $\boldsymbol{\omega}_t$ of most clean instances are larger than that of most noisy instances, which means $\boldsymbol{\omega}_t$
    can differentiate between clean and noisy instances.
    }
    \label{fig:clean_and_noise_w_cifar10}
\end{minipage}%
\begin{minipage}[t]{.5\textwidth}
  \centering
    \captionsetup{width=.95\linewidth}
    \includegraphics[width=0.8\columnwidth]{figures/exp_verification.png}
    \caption{The change of class-level weights in an iteration for a noisy instance (cat mislabeled as "dog").  MWNet downweights all gradient flows. In contrast, GDW upweights the "not bird" gradient flow for better information use.}
    \label{fig:exp_verification}
\end{minipage}
\end{figure}

In Figure \ref{fig:clean_and_noise_w_cifar10}, we show the distribution of class-level target weight~($\boldsymbol\omega_t$) on clean and noisy instances in one epoch under the CIFAR10 40\% uniform noise setting.
We observe that $\boldsymbol\omega_t$ of most clean instances are larger than that of most noisy instances, which indicates that $\boldsymbol\omega_t$ can distinguish between clean instances and noisy instances.
This is consistent with Eq. (\ref{eq:grad1'})  that $\boldsymbol\omega_t$ serves as the instance weight.

To better understand the changing trend of non-target class-level weights, we visualize the ratio of increased weights in one epoch in Figure \ref{fig:cifar10_unif_ratio_trend} under the CIFAR10 40\% uniform noise setting.
Specifically, there are three categories: \textbf{n}on-\textbf{t}arget \textbf{w}eights on \textbf{c}lean instances (\textbf{$\boldsymbol\omega^c_{nt}$}), \textbf{t}rue \textbf{t}arget \textbf{w}eights on \textbf{n}oisy instances (\textbf{$\boldsymbol\omega^{n}_{tt}$}) and \textbf{n}on-\textbf{t}arget (excluding true targets) \textbf{w}eights on \textbf{n}oisy instances ($\boldsymbol\omega^n_{nt}$).
Formally, "target weight" means the class-level weight on the label position.
"true-target weight" means the class-level weight on the true label position, which are only applicable for noisy instances. "non-target weight" means the class-level weight except the label position and the true label position.
For example, as shown in Figure \ref{fig:motiv} where a cat is mislabeled as "dog", the corresponding meanings of the notations are as follows: 1)~$\boldsymbol\omega^n_{t}$ means $\boldsymbol\omega_{dog}$ ("dog" is the target); 2)~$\boldsymbol\omega^n_{tt}$ means $\boldsymbol\omega_{cat}$ ("cat" is the ture target); 3)~$\boldsymbol\omega^n_{nt}$ means $\boldsymbol\omega_{bird}$ ("bird" is one of the non-targets).
For a correctly labeled cat, the corresponding meanings are: 1)~$\boldsymbol\omega^c_{t}=\boldsymbol\omega^c_{tt}$ means $\boldsymbol\omega_{cat}$~("cat" is both the target and the ture target); 2)~$\boldsymbol\omega^c_{nt}$ means $\boldsymbol\omega_{dog}$ and $\boldsymbol\omega_{bird}$ ("dog" and "bird" are both non-targets).

Note that in Figure \ref{fig:motiv}, $\boldsymbol\omega^n_{tt}$ represents the importance of the "not cat" gradient flow and $\boldsymbol\omega^n_{nt}$ represents the importance of the "not bird" gradient flow.
If the cat image in Figure \ref{fig:motiv} is correctly labeled as "cat", then
the two non-target weights $\boldsymbol\omega^c_{nt}$ are used to represent the importance of the "not dog" and the "not bird" gradient flows, respectively.
In one epoch, we calculate \textbf{the ratios of} the number of increased $\boldsymbol\omega^c_{nt}$, $\boldsymbol\omega^{n}_{tt}$ and $\boldsymbol\omega^n_{nt}$ \textbf{to} the number of all corresponding weights.
$\boldsymbol\omega^c_{nt}$ and $\boldsymbol\omega^n_{nt}$ are expected to increase since their gradient flows contain valuable information,
whereas $\boldsymbol\omega^{n}_{tt}$ is expected to decrease because the "not cat" gradient flow contains harmful information.
Figure \ref{fig:cifar10_unif_ratio_trend} aligns perfectly with our expectation.
Note that the lines of $\boldsymbol\omega^c_{nt}$ and $\boldsymbol\omega^n_{nt}$ nearly coincide with each other and fluctuate around $65\%$.
This means non-target weights on clean instances and noisy instances share the same changing pattern, i.e., around $65\%$ of $\boldsymbol\omega^c_{nt}$ and $\boldsymbol\omega^n_{nt}$ increase.
Besides, less than $20\%$ of $\boldsymbol\omega^{n}_{tt}$ increase and thus more than $80\%$ decrease, which means the gradient flows of $\boldsymbol\omega^{n}_{tt}$ contain much harmful information.

In Figure \ref{fig:exp_verification}, we show the change of class-level weights in an iteration for a noisy instance, i.e., a cat image mislabeled as "dog".
The gradient flows of "not cat" and "dog" contain harmful information and thus are downweighted by GDW.
In addition, GDW upweights the valuable "not bird" gradient flow from $0.45$ to $0.63$.
By contrast, unable to capture class-level information, MWNet downweights all gradient flows from $0.45$ to $0.43$, which leads to information loss on the "not bird" gradient flow.

\noindent \textbf{Training without the zero-mean constraint.} We have also tried training without the zero-mean constraint in Section \ref{subsec:restOn} and got poor results~(see Appendix \ref{subsection: Appendix_B.2} for details).
Denote the \textbf{t}rue \textbf{t}arget as $tt$ and one of the \textbf{n}on-\textbf{t}arget labels as $nt$ ($nt\neq tt$).
Note that the gradient can be unrolled as (see Appendix \ref{subsection: Appendix_B.2} for details):
\begin{equation} \label{eq2}
f_{\boldsymbol \omega}\left(\nabla_{\boldsymbol \theta} \mathcal{L}\right) = \boldsymbol{\omega}_t \sum_j\left({\boldsymbol{p}'_j}-\mathbf{y}_j\right)\frac{\partial \boldsymbol{l}_j}{\partial \boldsymbol \theta} + \left(\sum_k{{\boldsymbol {\omega}_k} {\boldsymbol{p}_k}}-\boldsymbol{\omega}_t\right) \sum_j {\boldsymbol{p}'_j}\frac{\partial \boldsymbol{l}_j}{\partial \boldsymbol \theta}.
\end{equation}

If $\sum_k{{\boldsymbol \omega_k} {\boldsymbol p_k}}-\boldsymbol \omega_t$ is positive and the learning rate is small enough, $\left(\sum_k{{\boldsymbol \omega_k} {\boldsymbol p_k}}-\boldsymbol\omega_t\right) \boldsymbol p'_{tt} \frac{\partial \boldsymbol l_{tt}}{\partial \boldsymbol \theta}$ contributes to the decrease of the true target logit $\boldsymbol l_{tt}$  after a gradient descent step.
If negative,  $\left(\sum_k{{\boldsymbol \omega_k} { \boldsymbol p_k}}-\boldsymbol \omega_t\right) \boldsymbol p'_{nt} \frac{\partial \boldsymbol l_{nt}}{\partial \boldsymbol \theta}$ contributes to the increase of the non-target logit $\boldsymbol l_{nt}$.
Therefore, without the zero-mean constraint, the second term in Eq. (\ref{eq2}) may hurt the performance of the model regardless of the sign of $\sum_k{{\boldsymbol \omega_k} {\boldsymbol p_k}}-\boldsymbol \omega_t$.
Similarly, training without the constraint results in poor performance in other settings.
Hence we omit those results in the following subsections.

\begin{figure}
\centering
\begin{minipage}[t]{.5\textwidth}
  \centering
    \includegraphics[width=0.8\columnwidth]{figures/cifar10_unif_ratio_trend.png}
    \captionsetup{width=.95\linewidth}
    \caption{Ratio trend of the number of increased $\boldsymbol\omega^c_{nt}$, $\boldsymbol\omega^{n}_{tt}$, and $\boldsymbol\omega^n_{nt}$ under the CIFAR10 40\% uniform noise setting.
    %
    Around $65\%$ of $\boldsymbol\omega^c_{nt}$ and $\boldsymbol\omega^n_{nt}$ increase since they contain useful information.
    %
    Besides, less than $20\%$ of $\boldsymbol\omega^{n}_{tt}$ increase and thus more than $80\%$ of $\boldsymbol\omega^{n}_{tt}$ decrease since they contain harmful information.
    }
    \label{fig:cifar10_unif_ratio_trend}
\end{minipage}%
\begin{minipage}[t]{.5\textwidth}
  \centering
    \includegraphics[width=0.82\columnwidth]{figures/cifar10_imb_ratio_trend.png}
    \captionsetup{width=.95\linewidth}
    \caption{Ratio trend of the number of increased $\boldsymbol{\omega}_8$ on $C9$ instances under the Long-Tailed CIFAR10 $\mu=0.1$ setting.
    %
    Less than $10\%$ of $\boldsymbol{\omega}_8$ increase and thus more than $90\%$ decrease.
    %
    A small $\boldsymbol{\omega}_8$ strikes a balance between two kinds of information: "$C8$" and "not $C8$",
    which better handles class imbalance.
    }
    \label{fig:cifar10_imb_ratio_trend}
\end{minipage}
\end{figure}

\subsection{Class Imbalance Setting}
\label{sec:imbalance}

\begin{table}
	\centering
	\caption{Test accuracy on the long-tailed CIFAR10 and CIFAR100 with different imbalance ratios.}
	\label{tab:class_imbalance}
	\scalebox{0.84}{
	\begin{tabular}{ccccccc}
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		\multirow{2}*{Dataset} & \multicolumn{3}{c}{CIFAR10} &  \multicolumn{3}{c}{CIFAR100} \\
		\cmidrule(lr){2-4}
        \cmidrule(lr){5-7}

		 & \multicolumn{1}{c}{$\mu=1$} & \multicolumn{1}{c}{$\mu=0.1$}  & \multicolumn{1}{c}{$\mu=0.01$}  & \multicolumn{1}{c}{$\mu=1$}  & \multicolumn{1}{c}{$\mu=0.1$}  & \multicolumn{1}{c}{$\mu=0.01$}  \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		BaseModel & $92.73 \pm 0.37$ & $85.93 \pm 0.57$ & $69.77 \pm 1.13$ & $70.42 \pm 0.54$ & $56.25 \pm 0.49$ & {$37.79 \pm 0.82$} \\
		Fine-tuning  & $92.77 \pm 0.37$ & $82.60 \pm 0.49$ & $59.76 \pm 1.00$ & $70.52 \pm 0.57$ & $55.95 \pm 0.50$ & $37.10 \pm 0.87$ \\
		Focal  & $91.68 \pm 0.49$ & $84.57 \pm 0.83$ & $65.78 \pm 4.02$ & $68.48 \pm 0.38$ & $55.02 \pm 0.51$ & $37.43 \pm 1.00$ \\
		Balanced  & $92.80 \pm 0.47$ & $86.05 \pm 0.46$ & $63.63 \pm 3.60$ & $70.56 \pm 0.56$ & $55.02 \pm 0.80$ & $27.60 \pm 1.39$ \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		\specialrule{\cmidrulewidth}{0pt}{0pt}
		L2RW  & $89.70 \pm 0.50$ & $79.11 \pm 3.40$ & $51.15 \pm 7.13$ & $63.40 \pm 1.31$ & $46.28 \pm 4.51$ & $25.86 \pm 5.78$ \\
	    INSW  & $92.70 \pm 0.57$ & \uline{$86.31 \pm 0.28$} & \uline{$70.27 \pm 0.24$} & $70.52 \pm 0.39$ & $55.94 \pm 0.51$ & $37.67 \pm 0.59$ \\
		MWNet            & \textbf{92.95} $\pm$ \textbf{0.33} & $86.17 \pm 0.75$ & $62.70 \pm 1.76$ & \uline{$\textbf{70.64} \pm \textbf{0.31}$} & \uline{$56.49 \pm 1.52$} & \uline{$37.83 \pm 0.86$} \\
		\textbf{GDW}  & \uline{\textbf{92.94} $\pm$ \textbf{0.15}} & \textbf{86.77} $\pm$ \textbf{0.55} & \textbf{71.31} $\pm$ \textbf{1.03} & \textbf{70.65} $\pm$ \textbf{0.52} & \textbf{56.78} $\pm$ \textbf{0.52} & \textbf{37.94} $\pm$ \textbf{1.58} \\
		\specialrule{\cmidrulewidth}{0pt}{0pt}
	\end{tabular}
	}
\end{table}

\noindent \textbf{Setup and comparison methods.}
The imbalance factor $\mu \in (0, 1)$ of a dataset is defined as the number of instances in the smallest class divided by that of the largest~\cite{shuMetaWeightNetLearningExplicit2019a}.
Long-Tailed CIFAR \cite{krizhevskyLearningMultipleLayers2009} are created
by reducing the number of training instances per class according to an exponential function $n = n_i\mu^{i/(C-1)} $, where $i$ is the class index (0-indexed) and $n_i$ is the original number of training instances.
Comparison methods include: 1)~L2RW~\cite{renLearningReweightExamples2018}; 2)~INSW~\cite{huLearningDataManipulation2019a}; 3)~MWNet~\cite{shuMetaWeightNetLearningExplicit2019a}; 4)~BaseModel; 5)~Fine-tuning; 6)~Balanced~\cite{cuiClassBalancedLossBased2019}; 7)~Focal~\cite{linFocalLossDense2020}.

\noindent \textbf{Analysis.} As shown in Table \ref{tab:class_imbalance}, GDW performs best in nearly all settings and exceeds MWNet by $8.6\%$ when the imbalance ratio $\mu$ is $0.01$ in CIFAR10.
Besides, INSW achieves competitive performance at the cost of introducing a huge amount of learnable parameters (equal to the training dataset size $N$).
Furthermore, we find that BaseModel achieves competitive performance, but fine-tuning on the meta set hurts the model's performance.
We have tried different learning rates from $10^{-7}$ to $10^{-1}$ for fine-tuning, but the results are similar.
One explanation is that the balanced meta set worsens the model learned from the imbalanced training set.
These results align with the experimental results in \cite{huLearningDataManipulation2019a} which also deals with class imbalance.

Denote the smallest class as $C9$ and the second smallest class as $C8$ in Long-Tailed CIFAR10 with $\mu=0.1$.
Recall that $\boldsymbol \omega_j$ denotes the $j^{th}$ class-level weight.
For all $C9$ instances in an epoch, we calculate \textbf{the ratio of} the number of increased $\boldsymbol \omega_8$ \textbf{to} the number of all $\boldsymbol \omega_8$, and then visualize the ratio trend in Figure \ref{fig:cifar10_imb_ratio_trend}.
Since $C9$ is the smallest class, instance weighting methods upweight both $\boldsymbol \omega_8$ and $\boldsymbol \omega_9$ on a $C9$ instance.
Yet in Figure \ref{fig:cifar10_imb_ratio_trend}, less than $10\%$ of $\boldsymbol \omega_8$ increase and thus more than $90\%$
decrease.
This can be explained as follows.
There are two kinds of information in the long-tailed dataset regarded to $C8$: "$C8$" and "not $C8$".
Since $C8$ belongs to the minority class, the dataset is biased towards the "not $C8$" information.
Because $\boldsymbol \omega_8$ represents the importance of "not $C8$", a smaller $\boldsymbol \omega_8$ weakens the "not $C8$" information.
As a result, decreased $\boldsymbol \omega_8$ achieves a balance between two kinds of information: "$C8$" and "not $C8$", thus better handling class imbalance at the class level.
We have conducted further experiments on imbalanced settings to verify the effectiveness of GDW and see Appendix D for details.

\subsection{Real-world Setting}
\label{sec:real-world}

\noindent \textbf{Setup and training.} The Clothing1M dataset contains one million images from fourteen classes collected from the web~\cite{tongxiaoLearningMassiveNoisy2015}.
Labels are constructed from surrounding texts of images and thus contain some errors.
We use the ResNet-18 model pre-trained on ImageNet~\cite{dengImageNetLargescaleHierarchical2009} as the classifier.
The comparison methods are the same as those in the label noise setting since the main issue of Clothing1M is label noise~\cite{tongxiaoLearningMassiveNoisy2015}.
All methods are trained for $5$ epochs via SGD with a $0.9$ momentum, a $10^{-3}$ initial learning rate, a $10^{-3}$ weight decay, and a $128$ batchsize.
See Appendix \ref{section: Appendix_E} for details.

\begin{table}
  \caption{Test accuracy on Clothing1M.}
  \label{tab: Clothing1M}
  \scalebox{0.733}{
  \centering
  \begin{tabular}{c c c c c c c c c c c c}
    \toprule
    Method     & BaseModel  & Fine-tuning & Co-teaching & GLC & L2RW & INSW & MWNet & Soft-label & Gen-label & \textbf{GDW} \\
    \midrule
    Accuracy($\%$) & $65.02$& $67.68$& $68.13$ & $68.60$ & \uline{$68.80$} & $68.25$ &$68.46$ &$68.69$ &$67.64$ & \textbf{69.39}      \\
    \bottomrule
  \end{tabular}
  }
\end{table}

\noindent \textbf{Analysis.} As shown in Table \ref{tab: Clothing1M}, GDW achieves the best performance among all the comparison methods and outperforms MWNet by $0.93\%$.
In contrast to unsatisfying results in previous settings, L2RW performs quite well in this setting.
One possible explanation is that, compared with INSW and MWNet which update weights iteratively, L2RW obtains instance weights only based on current gradients.
As a result, L2RW can more quickly adapt to the model's state, but meanwhile suffers from unstable weights~\cite{shuMetaWeightNetLearningExplicit2019a}.
In previous settings, we train models from scratch, which need stable weights to stabilize training.
Therefore, INSW and MWNet generally achieve better performance than L2RW.
Whereas in this setting, we use the pre-trained ResNet-18 model which is already stable enough.
Thus L2RW performs better than INSW and MWNet.

\section{Conclusion}
Many instance weighting methods have recently been proposed to tackle label noise and class imbalance, but they cannot capture class-level information.
For better information use when handling the two issues, we propose GDW to generalize data weighting from instance level to class level by reweighting gradient flows.
Besides, to efficiently obtain class-level weights, we design a two-stage weight generation scheme which is embedded in a three-step bi-level optimization framework and leverages intermediate gradients to update class-level weights via a gradient descent step.
In this way, GDW achieves remarkable performance improvement in various settings.

The limitations of GDW are two-fold.
Firstly, the gradient manipulation is only applicable to single-label classification tasks.
When applied to multi-label tasks, the formulation of gradient manipulation need some modifications.
Secondly, GDW does not outperform comparison methods by a large margin in class imbalance settings despite the potential effectiveness analyzed in Section~\ref{sec:imbalance}.
One possible explanation is that better information utilization may not result in performance gain which also depends on various other factors.
\section{Acknowledgement}
We thank Prof. Hao Zhang from Tsinghua University for helpful suggestions. This research was supported in part by the MSR-Mila collaboration funding. Besides, this research was empowered in part by the computational support provided by Compute Canada (www.computecanada.ca).
\section{Derivation of D1}
\label{section: Appendx_A}
Denote the logit vector as $\boldsymbol x$, we have
\begin{align}
    \boldsymbol p_j =& \frac{e^{\boldsymbol x_j}}{\sum_k e^{\boldsymbol x_k}} \\
    \mathcal{L} =& -\sum_j  \boldsymbol y_j\log{\boldsymbol p_j}.
\end{align}

For the target~(label) position $t$ we have $\boldsymbol y_t=1$  and
\begin{equation}
    \frac{\partial \mathcal{L}}{\partial  \boldsymbol x_t} = \frac{\partial \mathcal{L}}{\partial \boldsymbol p_t}\frac{\partial \boldsymbol p_t}{\partial \boldsymbol x_t} = -\frac{\sum_k e^{\boldsymbol x_k}}{e^{\boldsymbol x_t}}\frac{e^{\boldsymbol x_t}\sum_k e^{\boldsymbol x_k}- e^{2\boldsymbol x_t}}{\left(\sum_k e^{\boldsymbol x_k}\right)^2} = \frac{e^{\boldsymbol x_t}}{\sum_k e^{\boldsymbol x_k}} - 1 = \boldsymbol p_t - \boldsymbol y_t.
\end{equation}

For any other position $j$~($j \neq t$), we have $\boldsymbol y_j = 0$ and
\begin{equation}
    \frac{\partial \mathcal{L}}{\partial \boldsymbol x_j} = \frac{\partial \mathcal{L}}{\partial \boldsymbol p_t}\frac{\partial \boldsymbol p_t}{\partial \boldsymbol x_j} = -\frac{\sum_k e^{\boldsymbol x_k}}{e^{\boldsymbol x_t}}\frac{-e^{\boldsymbol x_j+\boldsymbol x_t}}{\left(\sum_k e^{\boldsymbol x_k}\right)^2} = \frac{e^{\boldsymbol x_j}}{\sum_k e^{\boldsymbol x_k}} = \boldsymbol p_j - \boldsymbol y_j.
\end{equation}

Therefore, we can conclude that $\boldsymbol D_1$ = $\boldsymbol p - \boldsymbol y$.

\section{Zero-mean Constraint on Class-level Weights}
\subsection{Derivation}
\label{subsection: Appendix_B.1}
\begin{align}
    \label{eq: ana}
    \boldsymbol{\omega}_j(\boldsymbol{p}_j-\mathbf{y}_j) =& \boldsymbol{\omega}_j\boldsymbol{p}_j-\boldsymbol{\omega}_j\mathbf{y}_j\\
    =& \left(\sum_k \boldsymbol{\omega}_k\boldsymbol{p}_k\right)\frac{\boldsymbol{\omega}_j\boldsymbol{p}_j}{\sum_k \boldsymbol{\omega}_k\boldsymbol{p}_k}-\left(\sum_k \boldsymbol{\omega}_k\mathbf{y}_k\right)\frac{\boldsymbol{\omega}_j\mathbf{y}_j}{\sum_k \boldsymbol{\omega}_k\mathbf{y}_k}\\
    &\text{If $\mathbf{y}_j = 0$, the second term of (19) becomes $0$, therefore can be rewritten as $\boldsymbol{\omega}_t\mathbf{y}_j$} \nonumber\\
    =& \left(\sum_k \boldsymbol{\omega}_k\boldsymbol{p}_k\right){\boldsymbol{p}'_j}-\boldsymbol{\omega}_t\mathbf{y}_j\\
    =& \boldsymbol{\omega}_t\left({\boldsymbol{p}'_j} - {\mathbf{y}_j}\right) + \left(\sum_k{{\boldsymbol{\omega}_k} {\boldsymbol{p}_k}}-\boldsymbol{\omega}_t\right){\boldsymbol{p}'_j}.\label{app:eq:grad1'}
\end{align}

\subsection{Training without Zero-mean Constraint}
\label{subsection: Appendix_B.2}
\begin{equation} \label{app:eq2}
\begin{split}
f_{\boldsymbol \omega}\left(\nabla_{\boldsymbol \theta} \mathcal{L}\right) =& f_{\boldsymbol \omega}(\sum_j\frac{\partial \mathcal{L}_i}{\partial \boldsymbol{l}_j} \frac{\partial \boldsymbol{l}_j}{\partial \boldsymbol \theta}) \\
    =& \sum_j \boldsymbol{\omega}_j(\boldsymbol{p}_j-\mathbf{y}_j)\frac{\partial \boldsymbol{l}_j}{\partial \boldsymbol \theta} \\
    =& \boldsymbol{\omega}_{t} \sum_j({\boldsymbol{p}'_j}-\mathbf{y}_j)\frac{\partial \boldsymbol{l}_j}{\partial \boldsymbol \theta} + (\sum_k{{\boldsymbol \omega_k} {\boldsymbol{p}_k}}-\boldsymbol{\omega}_{t}) \sum_j {\boldsymbol{p}'_j}\frac{\partial \boldsymbol{l}_j}{\partial \boldsymbol \theta}.
\end{split}
\end{equation}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/zero_mean.png}
    \caption{Training curve without zero-mean constraint on CIFAR10 under 40\% uniform noise.}
    \label{fig:zero_mean}
\end{figure}

Without zero-mean constraint, the training becomes unstable.
We plot the training curve of the CIFAR10 40\% uniform noise setting in Figure~\ref{fig:zero_mean}.

\section{Label Noise Training Setting}
\label{section: Appendix_C}
Following the training setting of \cite{shuMetaWeightNetLearningExplicit2019a}, the classifier network is trained with SGD with a weight decay $5$e-$4$, an initial learning rate of $1$e-$1$ and a mini-batch size of $100$ for all methods.
We use the cosine learning rate decay schedule~\cite{loshchilovSGDRStochasticGradient2016} for a total of 80 epochs.
The MLP weighting network is trained with Adam \cite{kingmaAdamMethodStochastic2015} with a fixed learning rate 1e-3 and a weight decay 1e-4.
For GLC, we first train 40 epochs to estimate the label corruption matrix and then train another 40 epochs to evaluate its performance.
Since Co-teach uses two models, each model is trained for 40 epochs for a fair comparison.
We repeat every experiment 5 times with different random seeds (seed=1, 10, 100, 1000, 10000, respectively) for network initialization and label noise generation.
We report the average test accuracy over the last 5 epochs as the model's final performance.
We use one V100 GPU for all the experiments.

\section{Further Experiments on Imbalanced Setting}
\label{section: Appendix_D}
\subsection{Class-level Weight Analysis}
We conduct one more experiment under the imbalance setting to better verify the interpretability of GDW.
As shown in Table~\ref{tab: imb_C}, we report the ratio of the number of increased $\boldsymbol\omega_i$ after gradient update on $C_j$ instances in one epoch, where $C_0$ is the largest class and $C_9$ is the smallest class.

Note that $\boldsymbol \omega_i$ on $C_i$ contains the "is $C_i$" information in the dataset. As a result, $\boldsymbol \omega_i$ on $C_i$ should be large for small classes and small for large classes. As shown above, the ratio of increased $\boldsymbol \omega_i$ on $C_i$ (the diagonal elements) increases from $0.036$ to $0.935$ as $i$ increases from $0$ to $9$.

On the other hand, $\boldsymbol \omega_i$ on $C_j$ ($j \neq i$) contains the "not $C_i$" information in the dataset. If $i$ is a large class, $\boldsymbol \omega_i$ on $C_j$ ($j \neq i$) should be large and vice versa. For $\boldsymbol \omega_i$ ($i=0, 1, 2, 3, 4$), the ratio of increased $\boldsymbol \omega_i$ on $C_j$ ($j \neq i$) are generally larger than $0.5$, and for $\boldsymbol \omega_i$ ($i=5, 6, 7, 8, 9$), the ratio of increased $\boldsymbol \omega_i$ on $C_j$ ($j \neq i$) are generally less than $0.5$. These results align with our analysis on the interpretable information of gradient flows.

\begin{table}[htbp]
    \centering
    \caption{Ratio of increased class-level weights under the imbalance setting.}
    \label{tab: imb_C}
    \begin{tabular}{@{}ccccccccccc@{}}
    \toprule
weight/class & \(C_0\) & \(C_1\) & \(C_2\) & \(C_3\) & \(C_4\) & \(C_5\)
& \(C_6\) & \(C_7\) & \(C_8\) & \(C_9\) \\
\midrule
\(\boldsymbol\omega_0\) & 0.036 & 0.968 & 0.973 & 0.972 & 0.965 & 0.974 & 0.972 &
0.976 & 0.956 & 0.973 \\
\(\boldsymbol\omega_1\) & 0.887 & 0.095 & 0.912 & 0.929 & 0.907 & 0.927 & 0.911 &
0.922 & 0.910 & 0.920 \\
\(\boldsymbol\omega_2\) & 0.848 & 0.844 & 0.141 & 0.839 & 0.822 & 0.845 & 0.818 &
0.847 & 0.829 & 0.802 \\
\(\boldsymbol\omega_3\) & 0.585 & 0.608 & 0.552 & 0.405 & 0.569 & 0.541 & 0.561 &
0.559 & 0.617 & 0.596 \\
\(\boldsymbol\omega_4\) & 0.474 & 0.521 & 0.420 & 0.460 & 0.509 & 0.455 & 0.456 &
0.482 & 0.467 & 0.512 \\
\(\boldsymbol\omega_5\) & 0.291 & 0.261 & 0.288 & 0.252 & 0.309 & 0.701 & 0.303 &
0.267 & 0.297 & 0.257 \\
\(\boldsymbol\omega_6\) & 0.199 & 0.189 & 0.169 & 0.198 & 0.196 & 0.222 & 0.778 &
0.195 & 0.207 & 0.182 \\
\(\boldsymbol\omega_7\) & 0.117 & 0.117 & 0.105 & 0.084 & 0.115 & 0.079 & 0.126 &
0.920 & 0.133 & 0.090 \\
\(\boldsymbol\omega_8\) & 0.115 & 0.124 & 0.178 & 0.185 & 0.184 & 0.174 & 0.191 &
0.181 & 0.862 & 0.137 \\
\(\boldsymbol\omega_9\) & 0.043 & 0.050 & 0.064 & 0.061 & 0.074 & 0.062 & 0.097 &
0.069 & 0.040 & 0.935 \\
\bottomrule
    \end{tabular}
\end{table}
\subsection{Experiments on Places-LT}

\begin{table}[htbp]
    \centering
    \caption{Test accuracy on Places-LT.}
    \label{tab:mix_placeLT}
    \begin{tabular}{@{}ccccc@{}}
\toprule
Method & L2RW & INSW & MW-Net & \textbf{GDW} \\
\midrule
Accuracy (\%) & \(15.08\) & \(17.80\) & \(18.08\) & \textbf{19.17} \\
\bottomrule
    \end{tabular}

\end{table}
We have conducted experiments on the Places-LT dataset \cite{liuLargeScaleLongTailedRecognition2019} and compared GDW with other meta-learning-based methods.
For all methods, the weight decay is set to $0.001$ and the batchsize is set to $64$. We adopt a $0.01$ initial learning rate and a cosine learning rate decay policy for $10$ epochs. The weight decay is set to $0.001$. The backbone network is ResNet18 and we use the ImageNet pre-trained model for initialization.

As shown in Table~\ref{tab:mix_placeLT}, GDW achieves the best performance among all the comparison methods and outperforms MWNet by $1.09\%$. This improvement is larger than that of CIFAR100. The reason is that GDW can manipulate class-level information and thus performs better on the dataset with a larger number of classes ($365$ in Places-LT and $100$ in CIFAR100). Besides, we can observe that L2RW performs the worst and the reason may be that L2RW suffers from unstable weights~\cite{shuMetaWeightNetLearningExplicit2019a}.

\section{Real-world Training Setting}
\label{section: Appendix_E}
Similar to \cite{shuMetaWeightNetLearningExplicit2019a}, we use the $7$k validation set as the meta set and the origin test set to evaluate the classifier's final performance.
For GLC, we first train $2$ epochs to estimate the label corruption matrix and then train another $3$ epochs to evaluate its performance.
Since Co-teach uses two models, each model is trained for $3$ epochs for a fair comparison.

\section{Experiments on Mixed Setting}
\label{section: Appendix_F}
We conduct further experiments to verify the performance of GDW in the mixed setting, i.e. the coexistence of label noise and class imbalance.
Specifically, we compare GDW with the mostly second-best method MW-Net \cite{shuMetaWeightNetLearningExplicit2019a} under the mixed setting of uniform noise and class imbalance on CIFAR10 and CIFAR100.
As shown in Table~\ref{tab:mix}, GDW demonstrates great performance gain over MW-Net, which means GDW can simultaneously better tackle both problems.

\begin{table}[htbp]
    \centering
    \caption{Test accuracy under mixed settings.}
    \label{tab:mix}
    \begin{tabular}{@{}ccccc@{}}
    \toprule
    Dataset & Noise Ratio & Imb Factor & MW-Net & GDW \\
    \midrule
    CIFAR10 & \(0.40\) & \(0.10\) & \(71.54\) & \(76.30\) \\
    CIFAR10 &\(0.60\) &\(0.10\) & \(61.62\) & \(70.24\) \\
    CIFAR10 & \(0.40\) & \(0.01\) & \(48.04\) & \(48.53\) \\
    CIFAR10 & \(0.60\) & \(0.01\) & \(39.51\) & \(40.07\) \\
    CIFAR100 &\(0.40\) &\(0.10\) & \(36.10\) & \(38.20\) \\
    CIFAR100 &\(0.60\) &\(0.10\) & \(24.80\) & \(25.40\) \\
    CIFAR100 &\(0.40\)& \(0.01\) & \(21.26\) & \(22.07\) \\
    CIFAR100 &\(0.60\) &\(0.01\) & \(12.75\) & \(14.15\) \\
    \bottomrule
    \end{tabular}
\end{table}
\section{Checklist}

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    \answerYes{}
  \item Did you describe the limitations of your work?
    \answerYes{The proposed method can only be applied on classification tasks.}
  \item Did you discuss any potential negative societal impacts of your work?
    \answerNA{}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?
    \answerYes{}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerYes{See Section~\ref{method}.}
	\item Did you include complete proofs of all theoretical results?
    \answerYes{See Section~\ref{method}.}
\end{enumerate}

\item If you ran experiments...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    \answerYes{We only use public datasets and the code is in the supplementary materials.}
  \item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    \answerYes{Most of our settings follow \cite{shuMetaWeightNetLearningExplicit2019a} and other details are in the appendix.}
	\item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    \answerYes{We repeat all experiments on CIFAR10 and CIFAR100 with five different seeds and the mean and standard deviation are reported. For the Clothing1M dataset, we only run one experiment due to limited resources.}
	\item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    \answerYes{We use one V100 GPU. See the appendix for details.}
\end{enumerate}

\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerYes{For dataset, we cite the papers of CIFAR datasets and the Clothing1M dataset. For code, we cite \cite{heDeepResidualLearning2016}.}
  \item Did you mention the license of the assets?
    \answerNA{}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerNA{}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerNA{}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerNA{}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
    \answerNA{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerNA{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerNA{}
\end{enumerate}

\end{enumerate}

redpajama_set_name: RedPajamaArXiv


text:
Asia Sports
TravoZone is a woman-owned company that developed a unique AI-powered solution for the travel industry
New Delhi [India], November 30 (ANI/SRV): TravoZone, the new emerging travel company, comes into existence with the vision of "traveling together" and the mission of "empowering travelers."
It was founded to inspire and empower world travellers by providing extensive options and instant booking at competitive prices.
TravoZone is a woman-owned business inspired by founders Vijayshri Shrivastava and Sweta Srivastava, who believe that both domestic and international travel is what keeps individuals learning about themselves and the world.
In 2015, Vijayshri and Shweta encountered a poor experience while planning a family vacation with a professional travel company's support. The so-called high-rated professional travel company provided limited options with higher-than-average costs. Frustrated with the lack of options on flights and hotels with unexpected hidden costs, Vijayshri and Shweta were inspired to launch TravoZone.
TravoZone is an IATA-accredited travel company that offers travel-related services to the world market with a unique bouquet of products and services. The organisation provides travellers looking for their next destination with the technology to secure the best deals and convenient flight options and explore on-site destinations at their convenience.
Vijayshri shares, "Our mission is to empower travellers by equipping them with all available options at prices that fit within their budget." Travelling is an experience that Sweta and I both enjoy, and we enjoy doing it together, which is why we want to give other families and friends a chance to travel together.
TravoZone soon came up with an innovative AI-powered solution for the travel industry. Travelers, along with venture capitalists and angel investors, must keep an eye out for further information.
The technology to be developed by TravoZone will provide eager travellers the chance to diminish the costs associated with the consultants of professional travel agencies by equipping the customer with a powered flight booking option to search and book.
With the innovations and developments in artificial intelligence, Vijayshri and Shweta plan to bring those developments to travellers through TravoZone to provide customers searching for the best options to book their dream vacations.
Shweta said, "We will be backend agents for our customers, at a fraction of the cost associated with professional agencies." "We believe our customers will use these reduced costs to book more vacations each year through TravoZone." By utilising new technology, TravoZone will provide customers with travel options delivered to their handheld devices in record time.
TRAVOZONE is based in Thane, Maharashtra, India, is an IATA-accredited travel company, and can be reached at www.travozone.com, on Twitter, Facebook, or Instagram (@travozone).
TravoZone services and products include air ticketing, hotel and alternative accommodations bookings, bus ticketiholiday planning and packaging, ng, car hire, and ancillary travel requirements such as facilitating access to third-party travel insurance.
This story has been provided by SRV. ANI will not be responsible in any way for the content in this article. (ANI/SRV)
Get a daily dose of Bangladesh Sun news through our daily email, its complimentary and keeps you fully up to date with world and business news as well.
Publish news of your business, community or sports group, personnel appointments, major event and more by submitting a news release to Bangladesh Sun.
Haze in Dhaka/Shahjalal Intl
Bloemfontein [South Africa], January 30 (ANI): Following his side's five-wicket win over England in the second ODI, South Africa skipper ...
J-K: No opposition unity as key parties' leaders skip Bharat Jodo Yatra's concluding event in Srinagar
New Delhi [India], January 30 (ANI): As Congress' Bharat Jodo Yatra officially came to a conclusion in Jammu and Kashmir's ...
Congress leader Siddaramiah to miss Bharat Jodo Yatra today as Srinagar-bound flights cancelled due to bad weather
Bengaluru (Karnataka) [India], January 30 (ANI): Former Karnataka chief Minister and the Leader of Opposition in the Karnataka Assembly, Siddaramiah ...
Haryana CM visits Shefali Verma's home in Rohtak after T20 WC win
Rohtak (Haryana) [India], January 30 (ANI): Haryana Chief Minister Manohar Lal Khattar visited U19 T20 women's cricket team captain, Shefali ...
Bloemfontein [South Africa], January 30 (ANI): South Africa's late push for automatic qualification to the 2023 Men's Cricket World Cup ...
Lucknow (Uttar Pradesh) [India], January 30 (ANI): Indian bowling coach Paras Mhambrey revealed that spinner Yuzvendra Chahal was included in ...
© Copyright 1999-2023 Bangladesh Sun. All rights reserved.

redpajama_set_name: RedPajamaCommonCrawl


In the current economy we are in Lenders have tightened up their credit requirements and are taking the time to verify every piece of information potential borrowers submit. In fact when you submit an application for a mortgage be ready to run through the gamit and be ready for a frustrating ambiguous process that more than likely will not produce the results you are looking for. It is for this reason we see that many home sellers are using owner financing to sell their home and rightly so. After the note has been created many note holders (home sellers) are selling their notes as they have a need for CASH.
All notes are different, just like a finger print, and each one closes just a little bit differently. Some note transactions are easier to process than others and can close quickly, within a week. Others prove to be more challenging and produce many hurdles that need to be overcome before closing. This is one such transaction where the challenges were many but overcome and closed.
All in All not a bad note. The Balloon is a little short which speaks to the Time Value of Money (TVM,) meaning money received sooner is more valuable than money received later. The note seller wanted the home buyer to refinance by 3 years but did not have a clear exit strategy; I would have advised a longer balloon of 5 to 7 years given the current economy. Another positive aspect to this note is the property appraised at $346,000 which added to the equity of the note. The Loan To Value (LTV) was 68%, Note Balance / the Appraisal. This note has several positive aspects working for it.
Now, what made this note challenging? First, the note was in 2nd position and "wrapped" to the note holder's (home seller) existing mortgage. The 1st position note had a low interest rate and amortized over 30 years with a balance of just over $123,000, so it made it possible for the home owner to sell their property using a "Wrap" strategy and still make a couple hundred dollars a month, plus a nice down payment. In order to sell this note any investor would want the 1st position paid in full from part of the proceeds, ok no problem.
Credit proved to be a big obstacle on this note and cost the note holder several thousand dollars. The credit score of the payors were in the mid to low 500s, ouch! The initial offer with good credit was close to $200K, the offer we received with credit as is was $175. Now many investors on the secondary market will not look at notes unless the score is above 600, but given the right story and right investor a buyer was found and at a price that made sense to the note seller. The Texas Note Company has many such relationships with investors.
It would have been nice for the note seller to have the additional $25K but she was able to get an offer for her note that achieved all her objectives; pay off the 1st lien and still have enough cash to fulfill another project.
We have an offer that was accepted, $175K, and now we move into the due diligence and closing phase. Here at The Texas Note Company we use a North American Title Company to process and facilitate most all of our transactions. I can't tell you the value that a good Title Agent provides. It guarantees that the buyer, seller and payor are protected and the process runs unabated. I have used others in the past as well as attorneys and have gone through a couple of harrowing experiences.
Validate Note Payor payment history – all in order.
The steps above seems just a part of the process and it is, but each item was an expense that came out of the seller proceeds, they add up but did not deter the seller from going through with the transaction.
Here is a list of documents need to process this note sale.
We made it to closing. The buyers had forwarded the closing docs and the seller was ready to sign the bottom line. After all the signing was complete it was time for the seller to hand over the Deed of trust and the Original Promissory note. One problem, the promissory note she had was not the original it was a copy, Uh Oh! The seller did not go home with her check. She had to find the original, only problem she did not no where it was after looking everywhere. Fortunately , the buyer agreed to accept a copy resigned by the payor and note holder. I tell note holder all the time, THE PROMISORY NOTE IS THE MOST IMPORTANT NOTE DOCUMENT KEEP IT IN A SAFE PLACE! Losing the original note document is a REAL potential deal killer. Sound record keeping can mean thousands of dollars if you decide to carry back a note.


text:
Watch Leadbelly
"You can't bury a black legend like Leadbelly!"
Leadbelly is a 1976 movie directed by Gordon Parks. Roger E. Mosley stars as Leadbelly. Leadbelly was a larger-than-life musical figure who straddled the fence between blues and folk music. Leadbelly's real name was Huddie Ledbetter. The American folklorists John and Alan Lomax discovered Leadbelly in the Angola State Prison in Alabama. The movie is set in the 1940s, and the story is told in a series of flashbacks about his youth. The movie touches on his troubles with the law, his relationships with women and his musical development. This is a must see movie for fans of blues and roots music.
ScreenPix Subscribers on Apple TV
Own from $9.99
Leadbelly is a 1976 drama with a runtime of 2 hours and 6 minutes. It has received mostly positive reviews from critics and viewers, who have given it an IMDb score of 7.2.
Where to Watch Leadbelly
Leadbelly is available to watch, stream, download and buy on demand at Apple TV+, Apple TV, Amazon, Google Play, YouTube VOD and Vudu. Some platforms allow you to rent Leadbelly for a limited time or purchase the movie and download it to your device.
Roger E. Mosley
Paul Benjamin
Madge Sinclair
Watch Thousands of Shows and Movies
All the TV You Love
Ad Info - This movie may not be available on Hulu
Stream Movies Like Leadbelly
After We Fell
Kama Sutra: A Tale of Love
True to the Game 2
Also directed by Gordon Parks
Three the Hard Way
Thomasine & Bushrod
The Learning Tree
Shaft's Big Score
Moments Without Proper Names
The Super Cops
Aaron Loves Angela


This unique reference directory spotlights the entire Jewish Arts and Museum scene worldwide – from museums to public galleries, publishers through to the art trade!
Here you will find information on thousands of art museums, art and antique dealers, public galleries, universities and art associations, auctioneers, restorers, art publishers, art periodicals, art book dealers, and more.
Every entry contains data including institution or business names, addresses, phone and fax numbers, emails, websites, information about specialties, and names of owners, directors, or curators. Within each chapter, addresses are arranged alphabetically by country, alphabetically within each country section, and alphabetically according to city. All listings are in the native language of the country.
For curators, travellers, librarians, academics, lecturers at art schools, students, journalists ,art and antique dealers, auctioneers, book dealers, and anyone else involved in art either professionally or as a pastime, this Directory is your indispensable tool for navigating this vibrant art and museum world.
For those who aren't interested in buying your own personal copy, make sure that you have access to the directory through your local library.


And Bernie Sanders wasn't having it.
With a shower of confetti, Republicans chanted, cheered, and waved colorful, shiny pom-poms as Donald J. Trump formally accepted the nomination. The house was packed and energized as Trump discussed the wall, Hillary Clinton's policies, and upholding the law in a much more on-script tone than usual. As the amalgamation of red, white, and blue balloons rained down upon delegates dressed like Abraham Lincoln, women holding Trump dolls, and sequined cowgirl hats, Trump promised he would make America great again and the 2016 Republican National Convention came to a close.
Peter Thiel became the first openly gay man to address his sexual orientation on stage at the RNC, breaking down a barrierbetween the GOP and LGBTQ community. The Silicon Valley entrepreneur said, "Every American has a unique identity. I am proud to be gay," as the audience applauded — including the Trump family. There may have been cheers, but not everyone was convinced that the RNC is ready to walk the walk, citing the party platform's support of conversion therapy and the recent rash of discriminatory measures in various states supported by GOP lawmakers. Later, Trump said, "As your president, I will do everything in my power to protect our LGBTQ citizens from the violence and oppression of the hateful foreign ideology." Again, most weren't buying it.
As Trump offered open arms to those still feeling the Bern, the Bernie Sanders campaign got fired up. A photo of the former Democratic candidate for president watching Trump's speech on TV quickly circulated on social media, alongside biting, witty commentary from the Sanders Twitter account. As Trump said "his supporters will join our movement," Sanders called him a hypocrite and said his supporters did not stand for Trump's platform. He'll speak at the Democratic National Convention next week Monday.
4. "You Can't Always Get What You Want"... unless it's Trump.
After the Republicans' not-so-subtle divides took center stage this week, the party finished the RNC with a clear musical message: "You Can't Always Get What You Want." As the Rolling Stones tune played, balloons and confetti filled the arena. Trump walked off stage before the festivities ended, but delegates danced to the song through the end.
The RNC's use of another song — The Beatles' "Here Comes The Sun" — sparked outrage from the estate of songwriter George Harrison. On Twitter, the estate called the RNC's unauthorized use of his music offensive.
Feeling a vintage vibe? So did Trump's speechwriters when prepping the candidate's final remarks. The words are reminiscent of Nixon's RNC speech in 1968, which talked about crime in America, foreign conflict, and building the economy at home. But internet fact-checkers tried to remind those at home and on social media that things really aren't that bad.
And...Hillary still seems to be taking cues from Kim Kardashian and bringing her A-game to social media.
Teen Vogue will see you next week in Philadelphia for the Democratic National Convention where Hillary Clinton will become the first woman to ever receive the nomination from a major political party in America.


Ask A Cartoonist: Holiday Greetings!
We hope the winter season treats you well no matter what holidays you observe!
Happy Holidays and don't be a Grinch!
I had the music from the 1956 version of the King and I (Yul Brynner and Deborah Kerr) in mind when I wrote this strip...sort of like the whirlwind of cooking and preparation just beginning for Christmas dinner! Merry Christmas everyone!
John, Anne and the entire Lee Family would like to wish everyone the Happiest of Holidays!
Happy Holidays, readers! From Tina's Groove!
Wishing all the "Six Chix" fans (and everyone else, too!) a wonderful holiday!
Dethany sends her heartfelt holiday wishes with this strip from the Fastrack office party.
Dennis has a crisis of conscience in this Hank Ketcham Christmas Classic.
Happy Holidays from the Forth Family!
Zippy's Holiday Sentiments---have a happy one!
Snuffy an' th' whole Smif fambly wish you a very Merry Chis'mis an' a bodaciously blessed New Year!


Award-winning journalist Amy Goodman to moderate voting rights and Amendment 4 discussion in Melbourne on October 20.
Over the years, World Cafe has seen its share of guitar gods.
There might be no bluegrass music as we know it without Wade Mainer, who died Monday at his home in Flint, Mich., Sept. 12 at age 104.
Chuck Brown, the Godfather of Go-Go music, died in 2012. A year earlier, he'd stopped by World Cafe to discuss his 50-plus-year career and his last album.


Naza Group of Companies is a Malaysian enterprise conglomerate associated with motor buying and selling, automotive franchises, property development, food and beverage and resort management. Most people seemingly aren't conscious that Google and a handful of others are testing out using autonomous drive technology for bikes. Office provide shops and craft or scrapbooking stores carry magnetic alphabets you should use to create customized phrases or messages on a metal bar and then imprint in your clay. By far, the best approach to check a suspect gas pump relay — or another automotive relay — is to swap the suspect relay with a very good one. The specs are extremely suitable for operating engineering and simulation software program like SolidWorks and SolidEdge with excellent graphics rendering. So, these are all things to bear in mind if you happen to're looking for a laptop for mainstream engineering purposes. The amplifiers are directly related to an unstable provide of incessantly fluctuating voltage and present—the capacitor! WE CREATE A VARIETY OF MODERN AND SUSTAINABLE MERCHANDISE FOR AUTOMOTIVE MARKETS.
HP Pelzer Automotive Techniques meets the listed criteria for inclusion in OSHA's Severe Violators Enforcement Program , which focuses on recalcitrant employers that endanger workers by committing willful, repeat or failure-to-abate violations. Recouped Automotive Repair & Recouped Collision Center is a full service automotive and collision repair facility – we can handle all of your automotive needs at one handy location. The automotive engineering career can be apt for you when you have interests in engineering and designs.
For someone that has more expertise than I, please check with our fellow blogger, Kitties26 and her blog, Extra Nail Polish She has used automotive pigments efficiently in holographic and duochrome frankens. TI Automotive's Fuel Tank & Supply Methods Division options award-winning fuel tank merchandise and gasoline pump & module technology for gasoline, diesel, flex-gas and hybrid vehicle applications. AIAG Supply Chain Administration initiatives provide pointers, training and academic alternatives to understand and manage the complexity and scope of supply chain issues. An automotive engineer has to work in a group and talk with several types of people. Join our free newsletter under to receive up to date news on various new technology and trade tasks. Maybe it was the model of paint (Menards) or the circumstances, however the Chartreuse particularly took much more coats and nonetheless isn't as strong as I might hoped.


text:
Face-Off Between Oshiohmole And Ngige To Be Resolved Amicably – Lai Mohammed
More in Politics:
INEC's Database Shows We Defeated Buhari, APC With 1.6 Million Votes – PDP, Atiku March 20, 2019 11:13 am
Jimi Agbaje Dumps PDP As Party Accuses Him Of Bad Luck In Lagos March 20, 2019 11:07 am
Fani-Kayode Mocks Okorocha Over Son-In-law's Gov'ship Election Loss March 14, 2019 6:04 pm
The Minister of Information and Culture, Alhaji Lai Mohammed says the on-going face-off between National Chairman of APC, Adams Oshiomhole and Minister of Labour and Employment, Dr Chris Ngige will be resolved amicably.
The minister gave the assurance when he fielded questions from State House correspondents after the meeting of the Federal Executive Council (FEC) at the Presidential Villa, Abuja, on Wednesday.
The National Chairman of the All Progressives Congress, Comrade Adams Oshiohmole, had on July 23 threatened to expel the Minister of Labour and Employment, Chris Ngige, or any minister who failed to inaugurate members of board of parastatals under their supervision from the party.
However, Ngige, in a statement by the Director of Press, Ministry of Labour and Employment, Samuel Olowookere, said those that said he had not inaugurated the Board of the Nigeria Social Insurance Trust Fund (NSITF) because he was awarding contracts were ignorant of the Procurement Act.
The Minister of Information, who also spoke on Tuesday's defection of some members of the National Assembly from the ruling APC to Peoples' Democratic Party (PDP), said the issue was not discussed during the cabinet meeting.
He said that President Buhari had already stated that the defectors were at liberty to belong to any political party in line with democratic tenets.
He noted that the APC was in firm control at the National Assembly, in spite the defections at the two chambers of the Assembly.
"The governing party, the All Progressives Congress APC is still having the majority in both chambers of the National Assembly,'' he said.
Mohammed also reminded Nigerians on the danger of spreading fake news.


Posted by Rob McKenzie on March 3, 2017, 9:14 am, in reply to "Re: Vinyl Rattle"
Dan, a group of enthusiasts in California was so convinced of the 'sonic advantages' of direct-to- disc recordings that they formed their own label, Sheffield Lab Recordings, and made audiophile LPs by this method throughout the 1970s. This shows, incidentally, that the d-t-d approach wasn't consigned to history when magnetic tape was widely adopted. Sheffield Lab recorded both classical and non-classical titles. As well as being recorded d-t-d, sessions could be simultaneously recorded on tape. A number of the classical releases contained performances by the Los Angeles Philharmonic Orchestra conducted by Erich Leinsdorf. Some of these titles remain available, but currently only as CDs - not LPs.


This giveaway is hosted by Idols48 and Omise Japan.
This giveaway is hosted by Omise Japan and Idols48.
Do you have what it takes to win this contest hosted by Kiddie Jam? Read on for details.
This giveaway is hosted by whatoplay.
Do you have what it takes to win this contest hosted by papemelroti? Read on for details.
This giveaway is hosted by MaMeMiMommy.
This giveaway is hosted by Big-Bang Tech Philippines.


text:
Q: How do I fix this type inference in Typescript I have a typescript error I need to fix. This is declared in the script earlier:
protected suggestedServicesDict: { [id: number]: { [id: number]: { weight: number } } } = {}

then a for loop is used to attempt to loop through the collection
for (var categoryId in this.suggestedServicesDict) {

This results in the categoryId being inferred as a string not a number. Which later causes validation error because another function is expecting a number.
here later it is used in the suggestedCategory:
var suggestedCategory = { CategoryId: categoryId, Name: category.Name, AverageWeight: 0, Services: [] };

Here it is causing the suggestedCategory CategoryId to  be a string.
I don't seem to be able to successfully declare this as a number instead of allowing it to be inferred, I have tried a number of things, that have not worked.
Can any one tell me how to ensure that the categoryId is a number

A: While indexable types support numbers as keys, that only applies to adding/setting a value for a key:
let a = 3;
let o = {} as { [num: number]: string };

o[a] = "three";

But in javascript the keys are always strings, so the above code will actually create the following object:
{
    "3": "three"
}

If you know that your keys are string representation of numbers then you can use Number to get the number value:
for (let categoryId in this.suggestedServicesDict) {
    let idNumber = Number(categoryId);
    let value = this.suggestedServicesDict[idNumber]; // or [categoryId]
    ...
}

redpajama_set_name: RedPajamaStackExchange


Ms. XBrain can be deployed by packaging it as a war file and handing it to a servlet container (like Tomcat).
The current deployment is at http://femur.biostr.washington.edu:8080/MsDXBrain and when called, the Launch.jsp page should be the target.
A descendent of the root must be named "Patient"
All sites must be in labels called "Site", "Stimsite" or "Stimsite1"
All sites must have children named "right_coord" "sup_coord" and "ant_coord"
MindSeer has 2 modes that have advantages and disadvantages. Standalone is best when the data is local or the amount of data to download is small (or very reusable). Client/Server? works best when the data is in a large central repository that you don't wish to download. For visualizing on the average brain, we only need to download the data once. This makes it ideal for standalone as the server won't have any load from running the MindSeer server and the client has better performance. However this does not work for all individual subjects because the download would be too big. Because of this, the intermediate page launches either standalone or client/server depending on the patient selected.
For internal testing, you can use the Test Page. Simply type in a CSM query.
The immediately returned document is not the data.
Because of #1, I have to encode the results or the query into a URL (I opted for the query using Base64 and GZip).
To accommodate this, it takes the query as the post and then uses DXQP to get the results and process them when they are requested by MindSeer.
The following use cases summarize how this system works.
The XBrain User clicks on the 3D button and posts the XQuery to MsXBrain.
An intermediate page allows the user to select a patient.
The JNLP file is returned to the XBrain User.
Tomcat sends the file to the MindSeer File servlet.
MindSeer File requests the data using the encoded query.
The results are returned to MindSeer to display to the user.
Almost all of the real work for generating the map file is done in a single XQuery with Java just providing a cache and the front end.
It takes a query and calls DXQP from within the query.
The results are returned as Saxon objects so there is no performance hit.
We pull out all of the additional information for each site as meta-data.
The document is shipped to another Web Service to have the sites normalized.


text:
Listen to Juice WRLD's Justin Bieber-collaboration "Wandered to L.A."
Written by WinstonBurns
Dave Kotinsky/Dimitrios Kambouris/Getty
A collaboration between late Chicago rapper Juice WRLD and Justin Bieber has arrived. Listen to "Wandered to L.A." below. The track will appear on Juice WRLD's forthcoming album Fighting Demons, due on December 10 via Grade A Productions and Interscope.
Fighting Demons will be the second posthumous LP from Juice WRLD following his death in 2019. Legends Never Die was released last year with a third album in a trilogy to follow.
On December 16 HBO will broadcast Into the Abyss, a new documentary about Juice WRLD featuring previously unseen footage of Juice alongside unreleased music and in-depth testimony from the closest to the rapper. An animated movie based on his music is also in the works.


I won't deny there are things I miss about Singapore life: that I had an academic job with a really easy schedule, for one, and great students. That I was on the outskirts but still within bus and subway line service of an exciting city. That there were four major food cultures come together. But I really, really got into how the high temperature for every single day was 92 degrees Fahrenheit and the low about 74 and it just didn't get any colder than that.
I can take cold. It gets tolerably cold in New Jersey, although usually for only a couple days at a stretch. But I did do eight winters in Troy, New York, where the first flurries may come as early as Columbus Day and the last may come as late as Labor Day, and going a month without breaking freezing is not unheard-of for May, and to be really cozy in bed you need a couple layers of blankets and to go to the bedroom about a half-hour before you mean to sleep so you can set the box spring on fire.
So it's not like I'm facing the mid-Michigan winter without precedent here. And it hasn't been a bad winter, so far: a couple of snowstorms, none very large; a couple weeks of weather in the 20s, although leavened by a warm snap recently that got into the upper 50s as if that made sense for Michigan in January. But this weekend has seen the temperature drop like a rock; today it was about 18 Fahrenheit when bunny_hugger went outside to fill the squirrel and the birdsquirrel feeders, and Monday isn't looking to be any better.
As I say, I can take cold, but I'm not necessarily saying I approve of this.
Trivia: The early-baseball ceremony of the losers presenting the game ball to the winners was important enough that in 1872 the National Association of Amateur Base Ball Players issued specific instructions for ``Furnishing the Ball'.	Source:	 But Didn't We Have Fun? An Informal History Of Baseball's Pioneer Era, 1843 - 1870, Peter Morris.
Currently Reading:	 The Triumph Of Numbers: How Counting Shaped Modern Life, I B Cohen.