text:
\section{Introduction} \label{sec:intro}

Machine learning has a significant role in several high impact applications in the medical imaging domains, such as computer-aided diagnosis, image segmentation, registration and fusion, image-guided therapy, image annotation, and image database retrieval. Example applications include the classification of interstitial lung diseases based on computed tomography (CT) images~\cite{anthimopoulos2016lung}, the classification of tuberculosis manifestation based on X-ray images~\cite{cao2016improving}, the classification of neural progenitor cells from somatic cell source~\cite{jiang2015convolutional}, etc.

Although medical imaging analysis has witnessed impressive progressing recent years thanks to the development of large-scale labeled datasets \cite{dataDCCL,datasets_xray,dataTNSCUI},  it may result in major discrimination if not dealt with proper care on biases within such data~\cite{fabbrizzi2021survey}. The bias in the data causes generalizability degradation of the developed models, which has been identified as one of major limitations in deep learning applications in healthcare~\cite{topol2019high}. For example, in the use case of apical lesion detection on panoramic radiographs, models trained only on Germany images showed a (mean±SD) F1-score of 54.1±0.8\% on Germany data but 32.7±0.8\% on Indian data ($p<0.001$/t-test)~\cite{krois2021generalizability}.
\shijie{Recent studies have proposed several debiased models \cite{reddy2021benchmarking,barbu2019objectnet} but few practical applications in medical images, due to the unknown bias and lack of reliable test datasets.}

In the community, scientists and practitioners have been aware of the biases existed in their datasets, but it is not a trivial problem to identify biases from medical image datasets.  There are two major challenges.
Firstly, different from manually set biased attributes, like demographic properties(e.g., race, gender, etc.)~\cite{steed2021image}, or application specific properties \cite{bissoto2020debiasing,li2020shape}, biased attributes in medical image datasets have various types (i.e., too numerous to enumerate) and implicit representations (i.e., hard to model).  \cref{fig:biasexp} illustrates two cases. One of them is the strip might be caused by an accidental failure of the machine. Beside visible strips like in \cref{fig:biasexp}, in some cases, strips are invisible for human but can be recognized by neural networks. The other case is inflammation which appears in most of positive examples, but cannot be used as a feature in cancer diagnosis.
\begin{figure}
\centering
\subfigure[Strip]{\includegraphics[width=3cm,height=2cm]{Figures/Raw_Image/strip_example.jpg}}
\hspace{0.5cm}
\subfigure[Inflammation]{\includegraphics[width=3cm,height=2cm]{Figures/Raw_Image/inflammation_.jpg}}%
\caption{Bias Examples}\label{fig:biasexp}
\end{figure}

Secondly, in medical images, bias features are coupled with true decision features (i.e. the features of lesions). Take inflammation in \cref{fig:biasexp} as an example, inflammation symptom does not only appear in the negative samples, but also in the positive regions of the positive samples.  This is different from the previous study in person re-identification where biases are only in the background \cite{tian2018eliminating}.
As a result, we define the bias investigated in this study as data intrinsic bias because it is hidden in data.

To tackle the above problems, we first define the data intrinsic bias attribute as the discriminant direction of the latent space where positive and negative samples are distinguishable without involving any information from lesions.
Then, we propose a novel bias attribute identification framework for automatic bias discovery in medical datasets.
The framework contains two major components, KlotskiNet and Bias Discriminant Direction Analysis (BD\textsuperscript{2}A).
The objectives of proposed KlostkiNet is to build a mapping from the original space to the latent space. Specifically, different from ordinary classification tasks~\cite{campanella2019clinical,LI2019193,2019End}, KlostkiNet does not consider any tile with a lesion, but relies the tile with the maximum confidence to represent a whole sample in each training iteration.
Moreover, BD\textsuperscript{2}A is designed to determine a data intrinsic bias attribute by solving  an optimal eigenvalue task, and a theoretical solution is provided.

To sum up, the major contributions of this study are as follows:
\begin{itemize}
    \item We provide a formal definition on the data intrinsic bias.  We call biases ``data intrinsic" is because they are hidden in the data.
    \item We propose a novel bias attribute identification framework, which contains two key components, KlotskiNet and BD\textsuperscript{2}A. KlotskiNet is designed to build up the bias embedding space and BD\textsuperscript{2}A is designed  to discover the discriminant direction of the space, which is the data intrinsic bias.
    \item Experimental results on three datasets show the effectiveness of the bias attributes discovered by the framework. Compare with two baselines, the test set constructed by our framework can better \wanglj{expose the issues related to bias features in a model.}
    \item We also provide a methodology to generate debiased test sets as a byproduct the framework for further algorithm evaluation.
\end{itemize}

\section{Related Work}\label{sec:rel}

Datasets are of central importance to computer vision and more broadly machine learning.
As early as 2001, \cite{torralba2011unbiased} exposes the dataset bias problem in the general image recognition problem.
More recently,  \cite{fabbrizzi2021survey} provides a survey of biases in visual datasets, where categorizes biases into selection bias, framing bias and label bias.
However, in the medical image domain, it is not easy to classify biases into the above three types, because the biases come from a lot of different aspects, including machine, reagent, manipulation style of the doctor \cite{mohan2018mri}.
For example, \cite{wachinger2021detect} studies the confounding bias which is from unwanted variability associated with scanners and sites when pooling data in neuroimaging datasets.
Another example is a study of skin lesion detection using  ink markings/staining, and
patches applied to the patient skin as two of seven possible "culprit" artifacts for creating biases \cite{bissoto2020debiasing}.

Although researchers have been recognized biases in datasets, most studies are interested on manually set attributes which cause biases.
\cite{bissoto2020debiasing} identifies seven artifacts (e.g.,  dark corners (vignetting), hair, ink markings/staining, etc.) which are possible sources of biases in the skin lesion detection, and finds out the spurious correlations exploited by biased networks.
\cite{li2020shape} specifies that shape and texture are two prominent and complementary cues for recognizing objects and designs a debias method.
Furthermore, \cite{steed2021image} focuses on human-like biases from social psychology (e.g. racial, gender, etc.) and observes that the model learns biases as the way people does.

All the above studies expose bias issues in the dataset, but intrinsic biases are more important because they are unknown. %
A few  studies tackle the problem of bias attribute mining.  For example,
\cite{wachinger2021detect} applies Kolmogorov complexity to detect whether the confounded bias by providing the simplest factorization of the graphical model. However, this method can only be applied on tabular data.

Besides, \cite{khosla2012undoing} and \cite{lopez2019dataset} address the problem of mining biases from image datasets.  These methods are based on the assumption that each dataset carries biases that make the provenance of its images easily distinguishable and affect the ability of the models to generalise well. Thus, they compare different datasets to identify biases.
However, the assumption does not always exist in the medical image domain.  Firstly, it is possible that the bias distributions are same in multiple datasets (e.g., {two datasets have the same correlation between inflammation and cancer}), then these methods cannot be applied.  Secondly, due to multi-site, different geographic institutions have different positive phenotypes (i.e. {distributions} of true features  are not totally same), then the true features are identified as biases by the above methods. As a result, there is a need to mine intrinsic bias from medical image datasets.

In order to analyze, compare and evaluate the performance of algorithms, a debiased test dataset is necessary but not well addressed by previous studies.
The most related work is ObjectNet\cite{barbu2019objectnet}, which is a test-only dataset with 50,000 images in 313 ImageNet classes, captured by crowdsourcing, following special guidelines to reduce biases, in particular, randomizing backgrounds, rotations, and image viewpoints. This dataset controls biases in the stage of data collection, but it costs a lot of human efforts.
In this study, we provide a methodology to generate a debiased test dataset as a byproduct of data intrinsic bias mining for further algorithm evaluation.  %

\section{Problem Definition} \label{sec:prob}

Given a dataset $\mathcal{X}=\{(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_S,y_S)\}$, consists of $S$ training examples, $(\mathbf{x}_i,y_i)$, where $\mathbf{x}_i\in \mathbb{R}^M$ represents an $M$-dimension feature vector, and $y_i\in \{-1,1\}$ represents the label of the sample $i$. For a medical image dataset, the samples with the label $-1$ are called negative samples, denoted as $\mathcal{X}_N$, which do not contain any lesion. Meanwhile, the samples with the label $1$ are called positive samples, denoted as $\mathcal{X}_P$,  which contain lesions.

For a medical image $\mathbf{x}_i$, we can cut it into a set of $T$ tiles, denoted as  $\mathbf{R}_i=\{\mathbf{r}_{i,1},\ldots,\mathbf{r}_{i,T}\}$. Then, $\mathbf{R}_i$ can be divided into two subsets by whether containing foreground information: $\mathbf{R}_i^+$ and $\mathbf{R}_i^-$, where $\forall\ \mathbf{r}_{i,j}\in \mathbf{R}_i^+$, $\mathbf{r}_{i,j}$ contains foreground information of the example $\mathbf{x}_i$, and $\forall\ \mathbf{r}_{i,j}\in \mathbf{R}_i^-$, $\mathbf{r}_{i,j}$ only contains the background. It is noted that in a medical image, the lesion is the foreground. Then, for a positive sample $\mathbf{x}_i \in \mathcal{X}_p$, $\mathbf{R}_i=\mathbf{R}_i^+\cup\mathbf{R}_i^-$. Meanwhile, for a medical image $\mathbf{x}_i\in \mathcal{X}_N$, its corresponding foreground tile set is empty, i.e. $\mathbf{R}_i=\mathbf{R}_i^-$, because all tiles from a negative samples are without lesion.

In this study, we focus on data intrinsic biases which make backgrounds to distinguish positive and negative samples.
We set up a latent space with $K$-dimension and a map $g: \mathbb{R}^{M} \rightarrow \mathbb{R}^K$, i.e. $\mathbf{u}_{i}=g(\mathbf{x}_{i})$.
In order to get rid of the true feature from foregrounds, the embedding aims to distinguish positive and negative samples by only using background tiles.

We define a data intrinsic bias attribute as a direction $\phi$ in the latent space where the distributions of positive and negative samples are different in the one-dimensional subspace spanned by $\phi$.
Moreover, a data intrinsic bias feature is represented by a set of projected values by projecting samples on the subspace spanned by $\phi$, which is privileged with respect to one of labels.

Take a tabular data use case as an example to better understand data intrinsic bias attributes and bias features.  Given a job hunting scenario like~\cite{lahoti2019ifair}, where attributes are \textit{gender}, \textit{working experience}, and \textit{education experience}, and the \textit{salary} (high/low) is the label. Suppose in the dataset all workers with \textit{working experiences} over 10 years are labelled as high, then \textit{working experience} is a data intrinsic bias attribute, and \textit{working experience:$>10$} is the bias feature with respect to the high salary.

\section{Methodology} \label{sec:method}

\cref{fig:overview} provides an overview of our framework on mining data intrinsic bias attributes. In general, the framework contains two major components, KlotskiNet and Bias Discriminant Direction Analysis (BD\textsuperscript{2}A\xspace), where KlostkiNet is to build up the mapping $g$ which makes backgrounds to distinguish positive and negative samples and BD\textsuperscript{2}A\xspace~ is to obtain the discriminant direction of the embedding space.

The KlotskiNet is a multi-instance learning algorithm to extract the embedding vectors from raw images (\cref{subsec:net}). In the training phase, KlotskiNet performs supervised training on the labeled data $\mathcal{X}$ by removing all positive tiles. In the inference phase, given an image $\mathbf{x}\in \mathcal{X}$ which is represented as its corresponding background tile set $\mathbf{R}^-$, KlotskiNet first makes predictions on each tile in $\mathbf{R}^-$, then select the one with the largest confidence, and finally computes the output of the last convolutional layer on the selected tile as the embedding of $\mathbf{x}$, denoted as $\mathbf{u}$.

BD\textsuperscript{2}A\xspace~(\cref{subsec:bdv}) first formulates the definition of data intrinsic bias attribute identification task to a discriminant direction optimization problem with conjugated orthogonal constraints, and then provides a theoretical solution.

\subsection{KlotskiNet} \label{subsec:net}

We design a scheme called KlotskiNet to obtain a feature extractor to map an raw image to the latent space where positive and negative samples are distinguished by bias features.  \cref{algo:KlotskiNetTrain,algo:KlotskiNetInfer}  illustrate the training process and the feature extractor in the inference phase, respectively.

\cref{algo:KlotskiNetTrain} has two components, where Lines 1-8 is to obtain all negative tiles $\mathbf{R}^-_i$ by getting rid of the influence of foregrounds in the feature extractor, and Lines 9-17 is to training the network with the maximum confidence tile.
Specifically, Line 11 is to obtain the confidence of each tile $\mathbf{r}_{i,j}$ in $\mathbf{R}^-_i$, where the tuple $(q^p_j, q^n_j)$ representing the normalized confidences of $\mathbf{r}_{i,j}$ on positive label and negative label, respectively.
Lines 12-13 is to find out the index $j^*$ of the tile with the maximum confidence. We first get the indices of maximum positive and negative confidences, i.e. $j^*_p$ and $j^*_n$, respectively.  Then, to compare the maximum positive confidence and maximum negative confidence produces the index of the maximum confidence.   Finally,  Lines 14-15 illustrate to train the neural network with the cross-entropy loss calculated by the selected tile to distinguish positive and negative samples.

The reasons for selecting the tile with maximum confidence are twofold.
Firstly, comparing the maximum confidence tile with a random one, the maximum one has much more significant features to make a higher confidence. As we have removed all positive tiles, those features are potential bias features.
Secondly, we select the larger one between positive confidence and negative confidence.  Then, among the whole training set, the selected tiles contain both predicted positives and predicted negatives. Therefore, the model considers both positive and negative bias features.

\begin{algorithm}
  \caption{KlotskiNet: Training Phase}
  \label{algo:KlotskiNetTrain}
  \begin{algorithmic}[1]
  \REQUIRE Raw image set $\mathcal{X}=\{(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_S,y_S)\}$
  \ENSURE KlotskiNet $f(\mathbf{r};\theta)$
  \\\COMMENT{\# Remove positive tiles}
  \FOR{$i \in \{1,\ldots,S\}$}
    \STATE $\mathbf{R}_i \gets Split(\mathbf{x}_i)$
    \IF{$y_i == 1$}
        \STATE $\mathbf{R}_i^-\gets$ $\mathbf{R}_i\setminus\{\mathbf{r}_{i,j}|lesion\ in\ \mathbf{r}_{i,j}\}$
    \ELSE
        \STATE $\mathbf{R}_i^-\gets\mathbf{R}_i$
    \ENDIF
  \ENDFOR
  \\\COMMENT{\# Training}
  \WHILE{conditions to continue training}
    \FOR{$i \in \{1,\ldots,S\}$}
        \STATE $\{(q^p_j,q^n_j)\}\gets \{f(\mathbf{r}_{i,j};\theta)|\mathbf{r}_{i,j} \in \mathbf{R}_i^-\}$
        \STATE $j^*_p \gets argmax \{q^p_{j}\}$,  $j^*_n \gets argmax \{q^n_{j}\}$
        \STATE $j^* \gets j^*_p$ if $q^p_{j^*_p} > q^n_{j^*_n} $ otherwise $j^*_n$
        \STATE Calculate cross-entropy loss with $y_i$ and $(q^p_{j^*}, q^n_{j^*})$
        \STATE Back propagation to update $\theta$
    \ENDFOR
  \ENDWHILE
  \end{algorithmic}
\end{algorithm}

\cref{algo:KlotskiNetInfer} is the process of extracting the embedding feature to distinguish positive and negative samples by KlotskiNet. Similar as the training process in \cref{algo:KlotskiNetTrain}, Lines 1-6 remove all positive tiles.
It is noted that when we continue on BD\textsuperscript{2}A\xspace~ as \cref{subsec:bdv}, the training set are used to collect the samples in the embedding space, where labels are necessary because we want to identify bias attributes for positive and negative separately.
However, when the purpose is to check whether a given image containing a bias feature,  it is necessary to project the sample to the latent space by \cref{algo:KlotskiNetInfer}, but the label is not necessary.
Then, Lines 7-9 obtain the tile with the maximum confidence.   Finally, the embedding vector is generated by mapping the selected tile to the feature map of the last layer before the output layer.

\begin{algorithm}
  \caption{Feature Extractor by KlotskiNet}
  \label{algo:KlotskiNetInfer}
  \begin{algorithmic}[1]
  \REQUIRE A raw image with its label $(\mathbf{x}, y)$, KlotskiNet $f(\mathbf{r})$ and the mapping representing its last layer before output layer $f'(\mathbf{r})$
  \ENSURE Embedding $\mathbf{u}$ of the input $\mathbf{x}$
  \\\COMMENT{\# Remove positive tiles}
  \STATE $\mathbf{R} \gets Split(\mathbf{x})$
    \IF{$y == -1$ or without label}
        \STATE $\mathbf{R}^-\gets\mathbf{R}$
    \ELSE
         \STATE $\mathbf{R}^-\gets$ $\mathbf{R}\setminus\{\mathbf{r}_{j}|lesion\ in\ \mathbf{r}_{j}\}$
    \ENDIF
    \\\COMMENT{\# Obtain the embedding}
    \STATE $\{(q^p_j,q^n_j)\}\gets \{f(\mathbf{r}_{j};\theta)|\mathbf{r}_{j} \in \mathbf{R}^-\}$
    \STATE $j^*_p \gets argmax \{q^p_{j}\}$,  $j^*_n \gets argmax \{q^n_{j}\}$
    \STATE $j^* \gets j^*_p$ if $q^p_{j^*_p} > q^n_{j^*_n} $ otherwise $j^*_n$
    \STATE $\mathbf{u} \gets f'(\mathbf{r}_{j^*})$
  \end{algorithmic}
\end{algorithm}

\wanglj{To sum up, the key advantage of KloskitNet is to learn an image with erased positive tiles but without to involve any new bias related to 1) whether there is an erased region; 2) location or shape of the erased region. This is because KlotskiNet breaks the image into tiles, and leverages multi-instance learning to select the representative tile as the model input. The model  is only trained by the selected tile (i.e. with the max confidence) at a time, but does not aware which region is.  That is to say, the model makes use of the whole image while avoiding to contain any information from the erased region.}

\subsection{BD\textsuperscript{2}A\xspace} \label{subsec:bdv}

As defined in \cref{sec:prob}, a data intrinsic bias attribute as a direction $\phi$ in the latent space obtained by \cref{algo:KlotskiNetTrain} where the distributions of positive and negative samples are different in the one-dimensional subspace spanned by $\phi$. Since it is hard to formulate the distribution differences by empirical data, we leverage sample moments to represent the characteristics of the distribution.  In this study, we address on the second moment following linear discriminant analysis(LDA)~\cite{martinez2001pca}.  However, we cannot apply LDA directly because LDA provides an overall space discriminant direction but cannot generate the discriminant direction with respect to a specific label.

Suppose $\mathcal{U}$ is produced by applying \cref{algo:KlotskiNetInfer} on the training set $\mathcal{X}$,  based on the above intuition that the direction is related to the label, we divide $\mathcal{U}$ into two subsets based on the label, which are $\mathcal{U}_p$ and $\mathcal{U}_n$. Suppose $\bar{\mathbf{u}}_p$, and $\bar{\mathbf{u}}_n$ are the mean vectors of the positive and negative embeddings, respectively.

In the following discussion, we use the way to find the positive bias attributes to illustrate the method.  It is noted that we can obtain the negative bias attributes in the same way by only switching the footmarks $p$ and $n$.

The positive label scatter matrix $S_p$ and the scatter of negatives centered by the mean of positive samples, which represents how negative samples are far away from positives, i.e., $S_{pn}$ are defined as:
\begin{eqnarray}
S_{p}&=&\mathbb{E}\{(\mathbf{u}_p-\bar{\mathbf{u}}_p)^T(\mathbf{u}_p-\bar{\mathbf{u}}_p)\} \\
S_{pn}&=&\mathbb{E}\{(\mathbf{u}_n-\bar{\mathbf{u}}_p)^T(\mathbf{u}_n-\bar{\mathbf{u}}_p)\}
\end{eqnarray}
The criterion function can be defined as follows:
\begin{equation}
    L(\phi) = \frac{\phi^T S_{pn} \phi}{\phi^T S_p \phi} \label{con:object_func}
\end{equation}
where $\phi$ is an arbitrary vector in $\mathbb{R}^K$.

The vector $\phi_1$ corresponding to maximum of $L(\phi)$ is the optimal discriminant direction.  This vector $\phi_1$ is the eigenvector corresponding to maximum eigenvalue of the following eigenequation:
\begin{equation}\label{con:r_vector_eigen}
    S_{pn}{\phi_1} = \lambda S_p {\phi_1}
\end{equation}

As introduced in \cref{sec:intro}, biases have various types, thus, the optimal discriminant direction is not enough. In the following, we figure out a way to compute the $k$-th optimal direction.  Suppose $k$ directions $\phi_1,\phi_2,\ldots,\phi_k$ $(1 \leq k \leq K)$ are obtained, then we can make the following transformation from $\mathbb{R}^K$ to $\mathbb{R}^k$ to obtain the feature corresponding to each direction:
\begin{equation}\label{con:transform}
    \mathbf{v} = \Phi_k^T
    \mathbf{u}
\end{equation}
where $\Phi_k = [\phi_1^T,\phi_2^T,\ldots,\phi_k^T]$.

In order to obtain uncorrelated features, following \cite{UDVpaper}, a series of conjugated orthogonality constraints to \cref{con:r_vector_eigen}, which are $\forall\ i \leq k$, then:
\begin{equation}\label{con:constrains}
    \phi_{k+1}^T S_p \phi_i = 0
\end{equation}
That is to say, suppose a positive sample embedding $\mathbf{u}_p$ is transformed to $\mathbf{v}_p$ with \cref{con:transform}, where $\mathbf{v}_p=[v_{p,1},\dots,v_{p,i}, \dots,v_{p,j}, \dots, v_{p,k}]^T$, then for any two features $v_{p,i}$ and $v_{p,j}$, they are statistically uncorrelated based on \cref{con:constrains}.

Finally, we can compute the $\phi_{k+1}$ according to the follow theorem:
\begin{thm}
The uncorrelated discriminant attribute $\phi_{k+1}$ is the eigenvector correspoding to maximum eigenvalue of the following eigenequation:
\begin{equation}
    [
        I -
        S_p \Phi_k^T
       \Phi_k
    ]
    S_{pn} \phi_{k+1}
    = \lambda S_p \phi_{k+1} \label{con:sol}
\end{equation}
where $I$ is identity matrix.
\end{thm}

\begin{pof}
Let $\phi_{k+1}$ satisfy the following constrain related to its normalization form as a complementary of \cref{con:constrains}:
\begin{equation}\label{con:const_normal}
    \phi_{k+1}^T S_p \phi_{k+1} = 1
\end{equation}
Transform \cref{con:object_func} by considering all constrains from \cref{con:constrains} and \cref{con:const_normal} as
\begin{eqnarray}\label{con:lagrange}
L(\phi_{k+1}) &=& \phi_{k+1}^T S_{pn} \phi_{k+1} - \lambda (\phi_{k+1}^T S_p \phi_{k+1}-1)\nonumber \\
                &-& \sum_{i=1}^k \mu_i \phi_{k+1}^T S_p \phi_{i}
\end{eqnarray}
where $\lambda$ and $\mu_i$ ($i=1,\ldots,k$) are Lagrange multipliers.  The optimal point is performed by setting the partial
derivative of $L(\phi_{k+1})$ with respect to $\phi_{k+1}$ as zero, which is:
\begin{equation}\label{con:deri}
    2S_{pn}\phi_{k+1} - 2\lambda S_p \phi_{k+1} - \sum_{i=1}^k \mu_i S_p \phi_{i} = 0
\end{equation}
Multiply \cref{con:deri} by $\phi_{i}^T$ ($i=1,\ldots,k$), then we can obtain
\begin{equation}\label{con:mu}
    \mu_i = 2\phi_i^T S_{pn} \phi_{k+1}
\end{equation}
Denote $M = [\mu_1,\ldots,\mu_k]= {\Phi}_k S_{pn} \phi_{k+1}$.
Put \cref{con:mu} into \cref{con:deri}, then we have
\begin{eqnarray}
    S_{pn}\phi_{k+1} - \lambda S_p \phi_{k+1} - S_p\Phi_k^T M = 0 \\
     S_{pn}\phi_{k+1}  - S_p\Phi_k^T {\Phi}_k S_{pn} \phi_{k+1} = \lambda S_p\phi_{k+1}
\end{eqnarray}
Therefore, \cref{con:sol} is obtained.

\end{pof}

\section{Experiments} \label{sec:exp}
To validate our method, we first introduce our experiment settings in \cref{subsec:setting}. Then, \cref{subsec:ovelperf} shows the overall performance of bias attributes identified by the framework.
Finally, we evaluate the performance of individual bias attributes quantitatively (\cref{subsec:ablation}) and qualitatively (\cref{subsec:casestudy}).
\wanglj{In addition, an extra dataset, more case studies as well as the parameter analysis} will be shown in Appendix. %
\subsection{Experiment Settings}\label{subsec:setting}
\subsubsection{Datasets} \label{subsec:data}

We conduct our experiments on two popular medical image datasets: Deep Cervical Cytological Lesions(DCCL)~\cite{dataDCCL} and Thyroid Nodule Segmentation and Classification in Ultrasound Images 2020(TNSCUI)~\cite{dataTNSCUI}.
\shijie{To verify the potential usage of our framework on the general image domain, we also test on CI-MNIST~\cite{reddy2021benchmarking}.  More detailed information on CI-MNIST are in Appendix. }

\begin{table}[]
\centering
\caption{Datasets Overview}\label{tab:datasets}
\begin{tabular}{@{}lrrrrr@{}}
\toprule
           & Train\# & Val\#   & Test\#  & Total\# & Accuracy \\ \midrule
DCCL       & 5,919  & 1,905  & 2,979  & 10,803 & 0.869           \\
TNSCUI & 2,186  & 729   & 729   & 3,644  & 0.750           \\
CI-MNIST  & 50,000 & 10,000 & 10,000 & 70,000 & 0.897           \\ \bottomrule
\end{tabular}
\end{table}

Specifically, it is worthy noted that the last row ``Accuracy'' in \cref{tab:datasets}, which represents the accuracy of the test set on KlotskiNet\xspace.  As illustrated in \cref{algo:KlotskiNetTrain}, there is no positive tile involving to train KlotskiNet\xspace.  Suppose there is no bias attribute in data, the network should work as a random generator (i.e., accuracy is around 0.5) because all input is noise which does not contain any information of the lesion/foreground. However, in all of our three datasets, the accuracy of KlotskiNet\xspace is above 0.5, which indicates the existence of biases.

\subsubsection{Baseline Methods} \label{subsec:baseline}

\wanglj{\textit{\textbf{Color:}} To compare with our method, we first set up a baseline representing manually bias identification method. In previous studies \cite{2015Structure,2019Fast}, researchers have found that normalizing unwanted color variations aids model performance in computational pathology, thus we use average  RGB values of each image as the baseline attributes to compare with our framework.
}\wanglj{
\textit{\textbf{ImageNet-C(IC):}} \cite{imagenetc} is a common robustness evaluation dataset, which has been used for debiased method evaluation in previous studies\cite{li2020shape,useimegenetc1}.
In this paper, we apply the generation method from \cite{imagenetc} on medical image datasets to compare with our framework. Three possible corruptions in medical images are selected for experiments: brightness(B), contrast(C), and jpeg compression(J).
}
\subsubsection{Evaluation Metric} \label{subsec:metric}
Following \cite{reddy2021benchmarking,barbu2019objectnet}, we use the drop of performance to evaluate the effectiveness of the framework.

In the evaluation, we first identify biased attributes by our method and baselines. Then, we \textit{project} the test set $\mathcal{X}_{test}$ on an attribute, \textit{select} samples with an absolute projected value in the top-$\theta$ percentile as the sample with bias feature, and \textit{put} them into the biased set, where $\theta$ is the parameter controlling the projected value to be recognized as the bias feature. With this approach, we can separate biased set with respect to the attribute from the rest, which will be used in \cref{subsec:ablation,subsec:casestudy}.  When to evaluate the overall performance(\cref{subsec:ovelperf}), the above \textit{project, select and put} operations iterate with top-$k$ optimal directions, where $k$ is a parameter to be discussed in Appendix too.  Finally, we obtain the biased set $\mathcal{X}_{biased}$. %

\wanglj{In order to evaluate the performance drop on the real prediction model, we have implemented a multiple instance model(\mil) of decision support systems as \cite{campanella2019clinical}.}
The performance drop is defined by the performance metric of MIL model on $\mathcal{X}_{test}\setminus\mathcal{X}_{biased}$ minus that on $\mathcal{X}_{biased}$. The larger performance drop, the more biased information leveraged by the model.

The performance metrics we will use include accuracy, precision, recall and area under the ROC curve(ROC-AUC). %
In particular, as a positive bias attribute affects negative samples, when checking the performance of an individual positive attribute, the recall and precision are calculated with respect to negative samples, and vice versa.

\begin{table*}[!htb]
\centering
\caption{Overall Performance
}%
   \label{tab:overall}
\centering
\begin{tabular}{@{}llllllll@{}}
\toprule
Datasets & Method    & \begin{tabular}[c]{@{}l@{}}Drop of\\      Accuracy\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Positive\\      Precision\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Positive\\      Recall\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Negative\\      Precision\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Negative\\      Recall\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      ROC-AUC\end{tabular} \\ \midrule
         & IC-B\cite{imagenetc}      & 0.124                                                           & -0.026                                                                           & 0.213                                                                         & -0.042                                                                           & 0.129                                                                         & 0.043                                                          \\
         & IC-BJ\cite{imagenetc}     & 0.174                                                           & 0.201                                                                            & 0.011                                                                         & 0.447                                                                            & 0.105                                                                         & 0.205                                                          \\
DCCL     & IC-BJC\cite{imagenetc}    & 0.198                                                           & 0.123                                                                            & 0.209                                                                         & 0.167                                                                            & 0.176                                                                         & 0.220                                                          \\
         & Color\cite{2015Structure,2019Fast}   & -0.050                                                          & -0.011                                                                           & -0.078                                                                        & -0.087                                                                           & -0.010                                                                        & -0.039                                                         \\
         & Our & \textbf{0.561}                                                  & \textbf{0.454}                                                                   & \textbf{0.584}                                                                & \textbf{0.573}                                                                   & \textbf{0.480}                                                                & \textbf{0.624}                                                 \\ \midrule
         & IC-B\cite{imagenetc}      & -0.023                                                          & 0.021                                                                            & -0.150                                                                        & 0.149                                                                            & -0.051                                                                        & -0.018                                                         \\
         & IC-BJ\cite{imagenetc}     & 0.032                                                           & 0.080                                                                            & -0.252                                                                        & \textbf{0.418}                                                                   & -0.043                                                                        & 0.051                                                          \\
TNSCUI   & IC-BJC\cite{imagenetc}    & 0.015                                                           & 0.045                                                                            & -0.112                                                                        & 0.188                                                                            & -0.021                                                                        & 0.081                                                          \\
         & Color\cite{2015Structure,2019Fast}   & 0.008                                                           & 0.012                                                                            & -0.015                                                                        & 0.039                                                                            & 0.000                                                                         & 0.018                                                          \\
         & Our & \textbf{0.311}                                                  & \textbf{0.330}                                                                   & \textbf{0.256}                                                                & 0.331                                                                            & \textbf{0.300}                                                                & \textbf{0.355}                                                 \\ \bottomrule
\end{tabular}
\end{table*}

\subsubsection{Implementation Details} \label{subsec:details}
\textit{\textbf{Network architecture.}} For the KlotskiNet training in DCCL and TNSCUI datasets, we use a pre-trained ResNet-50 \cite{resnet} as the backbone, and replace the top layers with a fully-connected layer and  a binary classification softmax layer.

\noindent
\textit{\textbf{Image split.}}
In image splitting(Line 2 in \cref{algo:KlotskiNetTrain,algo:KlotskiNetInfer}), for an image from DCCL and TNSCUI, we cut it into $5 \times 3$ tiles (each tile has $400 \times 400$ pixels).

Specifically, in DCCL and TNSCUI, lesions are foreground as well as well annotated. For a tile containing the lesion, it is recognized as a positive tile.

\noindent
\textit{\textbf{Hyper-parameters.} }
All networks are trained in an end-to-end fashion. Specifically, we utilize stochastic gradient descent with a learning rate 0.001 for DCCL as well as 0.0025 for TNSCUI.
The batch size is set to 32. We use Keras \cite{chollet2015keras} for all our experiments.
Both in Color and our method, $\theta = 12\%$. For IC, the corruption severity is set to 1, which ensures true features are not lost.

\subsection{Overall Performance}\label{subsec:ovelperf}

As shown in \cref{tab:overall}, we observe significant performance drop on our method, which indicates that the biased samples contain strong priors information that the rest samples do not have. Furthermore, this shows that the bias attributes extracted by our method are available in a real prediction model.

\wanglj{Besides, we compare our framework with two baseline methods. Although the drop of negative precision in TNSCUI\xspace is worse than IC-BJ, in most cases, our method outperforms.
Interestingly, we also observe the two baseline methods perform a negatively drop of performance on some scenarios, which indicates these methods are not sufficiently robust to bias validation.}
For color, we investigate the reason for this phenomenon is some of the three channels are not bias attribute at all, top-$\theta$ percentile samples are just outliers and hard to fit. As a result, we suggest more subtle adjustments can be applied on colors as a following studies of \cite{2015Structure,2019Fast}.
\wanglj{For IC, we investigate the corruptions are not necessarily well debiased, but change the distribution of bias features.
For example, low brightness of an image is a negative bias,  darkening of the image makes the model more biased to be negative, leading to a recall drop while a precision raise. As a result, we suggest apply our frameworks to identify the bias
for targeted corruption.
}

\subsection{Performance of Individual Attribute} \label{subsec:ablation}
In this subsection, we show the statistics on performance of individual attributes. Besides to use KlotskiNet\xspace only, we set Principal Component Analysis(PCA) as a baseline of BD\textsuperscript{2}A\xspace.  In the following discussions, we use K to represent KlotskiNet\xspace, K+P to represent KlotskiNet\xspace with PCA.  We collect the performance drop on each attribute and the statistics are shown in \cref{fig:BoxDCCL,fig:BoxTNSCUI} for DCCL and TNSCUI, respectively. CI-MNIST's is in Appendix.

As shown in \cref{fig:BoxDCCL,fig:BoxTNSCUI}, for KlotskiNet(K) only, we select biased samples based on each component of hidden layer outputs as a direction, the results become unpredictable.
This is due to only valid attributes can be used to select biased samples, while it is difficult to ensure that each output component is valid.  As a result, the selection is randomness, some outliers hard to classify are picked, and finally the result has a reverse performance drop.

Moreover, PCA(K+P) works better than KlotskiNet(K) only, because it helps to transform the space with respect to the direction by optimizing the within-label variance. That is to say, PCA reduces the randomness in selecting the samples. However, as PCA does not consider the distribution of the other label, the performance drop is not as good as ours, especially, the variance of K+P is larger than ours.

\subsection{Case Studies} \label{subsec:casestudy}

In this subsection, we apply the class activation mapping(CAM)~\cite{gradcam} to visualize the key features recognized by models. Besides KlotskiNet\xspace and \mil, we train a new \mil model with a debiased training set with respected to a given attribute.  The method to generate the debiased training set is to apply the \textit{project, select and put} operations in \cref{subsec:metric} on the training set to identify biased training samples with $\theta$ as 5\%. We call the model as Debaised. In addition, note that not every bias attribute has a semantics meaning as shown in the cases, because attributes are determined as directions in the high-dimensional space.

The cases are shown in \cref{fig:caseDCCL,fig:caseTNSCUI} for DCCL and TNSCUI\xspace respectively.  In each figure, besides the two raw test images as well as the corresponding images with the heatmap generated from models: KlotskiNet\xspace, \mil and Debiased (i.e. (b)-(e)), we also demonstrate the distributions of the projected values on the given attribute from positive test sets and negative test sets (i.e (a)).

Particularly, as shown in \cref{fig:caseDCCL} (b-e), both KlotskiNet and \mil focus on colored stripes while the important features of Debiased are most on cells. %
Similar phenomenon is shown in \cref{fig:caseTNSCUI} (b-e),  both KlotskiNet and \mil focus on the white vertical line in the edge of images, while Debiased almost ignores it. However, there is another bias existing in the two images we selected, which is the white dot at the bottom of the images.  As the bias attribute we selected is to describe the vertical line, and constrained as \cref{con:constrains}, the white dot becomes of the important feature in the Debaised model only ignored the vertical line as shown in \cref{fig:caseTNSCUI} (e).

\begin{figure}[t]
   \centering
   \includegraphics[width=0.75\linewidth]{Figures/Result/visu/dccl_cam.JPG}
    \caption{Case Study on DCCL: (a) histogram of projected values; (b) raw images; (c-e) heatmaps from corresponding models}
   \label{fig:caseDCCL}
\end{figure}

\begin{figure}[t]
   \centering
   \includegraphics[width=0.75\linewidth]{Figures/Result/visu/tnscui_cam.JPG}
   \caption{Case Study on TNSCUI\xspace: (a) histogram of projected values; (b) raw images; (c-e) heatmaps from corresponding models}
   \label{fig:caseTNSCUI}
\end{figure}

\begin{figure*}[t]
   \centering
   \includegraphics[width=0.75\linewidth]{Figures/Result/Box/DCCL_boxplot.jpg}
   \caption{Statistics on Performance of Individual Attributes with DCCL
   }
   \label{fig:BoxDCCL}
\end{figure*}

\begin{figure*}[t]
   \centering
   \includegraphics[width=0.75\linewidth]{Figures/Result/Box/TNSCUI_boxplot.jpg}
   \caption{Statistics on  Performance of Individual Attributes with TNSCUI %
   }
   \label{fig:BoxTNSCUI}
\end{figure*}

\section{Conclusion}
This study tackles the issues on automatic bias identification on medical image analysis. We first define the data intrinsic bias attribute and then propose a novel bias identification framework. The framework contains two major components, where KlostkiNet is to build the mapping which makes backgrounds to distinguish positive and negative samples and BD\textsuperscript{2}A\xspace provides a theoretical solution on determining bias attributes. Extensive experiments show the framework is effective because performance drops without the selected biased samples in a practice model.

\section{Appendix}
\subsection{Datasets}
In this section, we provide details of the datasets and settings used in \cref{sec:exp}.

\begin{table}[]
\centering
\caption{Datasets Overview}\label{tab:datasets}
\begin{tabular}{@{}lrrrrr@{}}
\toprule
           & Train\# & Val\#   & Test\#  & Total\# & Accuracy \\ \midrule
DCCL       & 5,919  & 1,905  & 2,979  & 10,803 & 0.869           \\
TNSCUI & 2,186  & 729   & 729   & 3,644  & 0.750           \\
CI-MNIST  & 50,000 & 10,000 & 10,000 & 70,000 & 0.897           \\ \bottomrule
\end{tabular}
\end{table}

\textbf{DCCL:}
Deep Cervical Cytological Lesions (DCCL) is the largest cervical cytology dataset \cite{dataDCCL}.
We use the provided train, validation, and test splits (5919, 1905, and 2979 images respectively).
The dataset is annotated by six pathologists with eight years of experience on the average on cell level labels: 27,972 lesion cells labeled based on The 2014 Bethesda System.

\textbf{TNSCUI:}
Thyroid Nodule Segmentation and Classification in Ultrasound Images 2020 (referred to as TNSCUI) is the biggest public dataset of thyroid nodule ultrasound images \cite{dataTNSCUI}.
Each ultrasound image is provided with a bounding box of the nodule and its annotated class (benign or malignant).
We select a subset of images for a train, validation and test
split of 2186, 729, and 729 images respectively.

\textbf{CI-MNIST:}
CI-MNIST is a recent published variant of MNIST dataset with introduced different types of biased features \cite{reddy2021benchmarking}.
For an input image , the label  indicates positive or negative, respectively, given that  is even or odd. The train, validation and test splits has 50000, 10000, and 10000 images respectively.
The pipeline of KlotskiNet in CI-MNIST is different from the above mentioned medical datasets. Any tiles containing a part of digits are removed, no matter in a negative or positive sample. To extract the region containing the digit, we implement a simple segmentation algorithm.

\subsection{Experiment of CI-MNIST}

For the CI-MNIST task, we use three convolution layers with a kernal size $3\times 3$ and the numbers of kernels are 64, 256 and 2,048 followed by a global average pooling layer as the backbone.
In image splitting, for CI-MNIST, we cut it into $4 \times 4$ tiles (each tile has $8 \times 8$ pixels). We regard the region with the digit as the foreground where pixel values of all channels are not zero.

\subsection{Hyper-Parameters Analysis} \label{subsec:parameter}
\subsubsection{Dimension of Subspace $k$}
As described in \cref{subsec:bdv}, $\Phi$ can be determined by iterative \cref{con:sol}.
Obviously, the dimension of $\Phi$, is limited by the number of real bias features and the dimension of embedding $\mathcal{U}$ which we set as 2048 in \cref{sec:exp}.
As shown in \cref{fig:lossCurveDCCL,fig:lossCurveTNSCUI}, with the increase of $k$, the value of criterion function $L$ defined in \cref{con:object_func} decreases gradually. We observe that beyond a certain value of $k$, the contribution of the $\phi_k$ is limited and stable, which means it does not contribute on distinguishing positives and negatives. In this study, $k=50$.

In addition, since bias for CI-MNIST datasets is generated of single factors, the maximum of $k$ is 1.

\begin{figure}
\centering
\subfigure[Negative]{\includegraphics[width=0.45\linewidth]{Figures/Result/Curve/Negative_feature_spnsp_dccl_klotski.png}}
\hspace{0.5cm}
\subfigure[Positive]{\includegraphics[width=0.45\linewidth]{Figures/Result/Curve/Positive_feature_spnsp_dccl_klotski.png}}
\caption{$L(\Phi) - k$ curve of DCCL }\label{fig:lossCurveDCCL}
\end{figure}

\begin{figure}
\centering
\subfigure[Negative]{\includegraphics[width=0.45\linewidth]{Figures/Result/Curve/Negative_feature_spnsp_thy_klotski.png}}
\hspace{0.5cm}
\subfigure[Positive]{\includegraphics[width=0.45\linewidth]{Figures/Result/Curve/Positive_feature_spnsp_thy_klotski.png}}
\caption{$L(\Phi) - k$ curve of TNSCUI}\label{fig:lossCurveTNSCUI}
\end{figure}

\subsubsection{Threshold on Bias Sample Selection $\theta$}
As described in \cref{subsec:metric}, $\theta$ is a parameter of data cleansing method, which is used in our method as well as the baseline Color. As shown in \cref{fig:thetaAndDropRate} and \cref{fig:thetaAndDropPreformance}, With the increase of $\theta$, the drop of performance is improved (debiased performance raised) while more data is deleted from the datasets (the available data amount reduced). The trade-off between the debiased performance and the available data amount depends on the actual application.
More $\theta$ extension table of overall performance are shown in the \cref{tab:perext}.
\begin{table}[]
\centering
\caption{Overall Performance Extension Table}\label{tab:perext}
\begin{tabular}{@{}llllllll@{}}
\toprule
       &              & \begin{tabular}[c]{@{}l@{}}Drop of\\      Accuracy\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Positive\\      Precision\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Positive\\      Recall\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Negative\\      Precision\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      Negative\\      Recall\end{tabular} & \begin{tabular}[c]{@{}l@{}}Drop of\\      ROC-AUC\end{tabular} \\ \midrule
DCCL   & IC-B         & 0.124                                                           & -0.026                                                                           & 0.213                                                                         & -0.042                                                                           & 0.129                                                                         & 0.043                                                          \\
       & IC-BJ        & 0.174                                                           & 0.201                                                                            & 0.011                                                                         & 0.447                                                                            & 0.105                                                                         & 0.205                                                          \\
       & IC-BJC       & 0.198                                                           & 0.123                                                                            & 0.209                                                                         & 0.167                                                                            & 0.176                                                                         & 0.220                                                          \\
       & Color($\theta=0.02$)   & -0.009                                                          & -0.003                                                                           & -0.014                                                                        & -0.014                                                                           & -0.002                                                                        & -0.013                                                         \\
       & Color($\theta=0.06$)   & -0.029                                                          & -0.008                                                                           & -0.045                                                                        & -0.046                                                                           & -0.005                                                                        & -0.028                                                         \\
       & Color($\theta=0.12$)   & -0.050                                                          & -0.011                                                                           & -0.078                                                                        & -0.087                                                                           & -0.010                                                                        & -0.039                                                         \\
       & Our($\theta=0.02$) & 0.095                                                           & 0.062                                                                            & 0.119                                                                         & 0.112                                                                            & 0.061                                                                         & 0.078                                                          \\
       & Our($\theta=0.06$) & 0.299                                                           & 0.205                                                                            & 0.330                                                                         & 0.345                                                                            & 0.235                                                                         & 0.282                                                          \\
       & Our($\theta=0.12$) & \textbf{0.561}                                                  & \textbf{0.454}                                                                   & \textbf{0.584}                                                                & \textbf{0.573}                                                                   & \textbf{0.480}                                                                & \textbf{0.624}                                                 \\ \midrule
TNSCUI & IC-B         & -0.023                                                          & 0.021                                                                            & -0.150                                                                        & 0.149                                                                            & -0.051                                                                        & -0.018                                                         \\
       & IC-BJ        & 0.032                                                           & 0.080                                                                            & -0.252                                                                        & \textbf{0.418}                                                                   & -0.043                                                                        & 0.051                                                          \\
       & IC-BJC       & 0.015                                                           & 0.045                                                                            & -0.112                                                                        & 0.188                                                                            & -0.021                                                                        & 0.081                                                          \\
       & Color($\theta=0.02$)   & 0.001                                                           & 0.003                                                                            & -0.004                                                                        & 0.008                                                                            & 0.000                                                                         & 0.003                                                          \\
       & Color($\theta=0.06$)   & 0.005                                                           & 0.009                                                                            & -0.009                                                                        & 0.024                                                                            & 0.001                                                                         & 0.008                                                          \\
       & Color($\theta=0.12$)   & 0.008                                                           & 0.012                                                                            & -0.015                                                                        & 0.039                                                                            & 0.000                                                                         & 0.018                                                          \\
       & Our($\theta=0.02$) & 0.056                                                           & 0.041                                                                            & 0.028                                                                         & 0.105                                                                            & 0.035                                                                         & 0.102                                                          \\
       & Our($\theta=0.06$) & 0.115                                                           & 0.058                                                                            & 0.070                                                                         & 0.219                                                                            & 0.064                                                                         & 0.207                                                          \\
       & Our($\theta=0.12$) & \textbf{0.311}                                                  & \textbf{0.330}                                                                   & \textbf{0.256}                                                                & 0.331                                                                            & \textbf{0.300}                                                                & \textbf{0.355}                                                 \\ \bottomrule
\end{tabular}
\end{table}

\begin{figure}
\centering
\subfigure[DCCL]{\includegraphics[width=0.45\linewidth]{Figures/appendix/dccl_theta_drop.png}}
\hspace{0.5cm}
\subfigure[TNSCUI]{\includegraphics[width=0.45\linewidth]{Figures/appendix/thy_theta_drop.png}}
\caption{$\theta - drop$ curve of DCCL and TNSCUI }\label{fig:thetaAndDropRate}
\end{figure}

\begin{figure}
\centering
\subfigure[DCCL]{\includegraphics[width=0.7\linewidth]{Figures/appendix/theta_and_drop_dccl.jpg}}
\hspace{0.5cm}
\subfigure[TNSCUI]{\includegraphics[width=0.7\linewidth]{Figures/appendix/theta_and_drop_thy.jpg}}
\caption{$\theta - performance \quad drop$ curve of TNSCUI}\label{fig:thetaAndDropPreformance}
\end{figure}

\subsubsection{Corruption Severity}
As described in the \cite{imagenetc}, each type of corruption has five levels of severity. In this study, corruption severity level = 1.
Corruption severity level $>1$ may cause a reduction of the key evidence related to the target. As shown in \cref{fig:corruption}, a small punctate echogenic foci can be clearly seen in \cref{fig:corruption} (a)\&(c) arrow points area, while the arrow points area in  \cref{fig:corruption} (b)\&(d) seems to be a solid area. This changes could seriously affect radiologist' diagnosis.

\begin{figure}
\centering
\subfigure[Sample 1 origin]{\includegraphics[width=0.3\linewidth]{Figures/appendix/sample/9006o.jpg}}
\subfigure[Sample 1 corrupted]{\includegraphics[width=0.3\linewidth]{Figures/appendix/sample/9006c.jpg}}

\subfigure[Sample 2 origin]{\includegraphics[width=0.3\linewidth]{Figures/appendix/sample/2516o.jpg}}
\subfigure[Sample 2 corrupted]{\includegraphics[width=0.3\linewidth]{Figures/appendix/sample/2516c.jpg}}
\caption{Corruption Severity}\label{fig:corruption}
\end{figure}

\subsection{Individual Performance of CI-MNIST}
As mentioned in \cref{subsec:parameter},
CI-MNIST datasets is generated of single biased factors, so we only leverage the first component  as the bias attribute from  PCA and BD\textsuperscript{2}A\xspace, respectively.
 As indicated in \cref{fig:boxMNIST}, the BD\textsuperscript{2}A\xspace does not unfold an advantage by comparing with PCA, possibly because of the bias is too simplistic and monolithic in CI-MNIST.

\begin{figure*}
\centering
\subfigure[Negative]{\includegraphics[width=0.9\linewidth]{Figures/Result/Box/MNIST_Negative_attribute.jpg}}
\hspace{0.5cm}
\subfigure[Positive]{\includegraphics[width=0.9\linewidth]{Figures/Result/Box/MNIST_Positive_attribute.jpg}}
\caption{Statistics on  Performance of Individual Attributes with CI-MNIST}\label{fig:boxMNIST}
\end{figure*}

\begin{table}[]
\centering
\caption{Debiased datasets file names}\label{tab:filenames}
\begin{tabular}{@{}lll@{}}
\toprule
Datasets & $\theta$ & List File Name             \\ \midrule
TNSCUI   & 0.02  & debiased\_TNSCUI\_theta002 \\
TNSCUI   & 0.06  & debiased\_TNSCUI\_theta006 \\
TNSCUI   & 0.12  & debiased\_TNSCUI\_theta012 \\
DCCL     & 0.02  & debiased\_DCCL\_theta002   \\
DCCL     & 0.06  & debiased\_DCCL\_theta006   \\
DCCL     & 0.12  & debiased\_DCCL\_theta012   \\ \bottomrule
\end{tabular}
\end{table}

redpajama_set_name: RedPajamaArXiv


Online travel agencies shaped up the travel landscape during the first months of 2017, being part of many travellers' plans. Whatever need a traveller is searching for, an online travel agency is capable of answering those and serving as a travel resource, planner, and agent at the same time.
Below are ten online travel agencies that had been making waves recently, all offering distinct services and features any traveller will find useful.
Priceline offers the following services: hotels, cars, packages, flights, and cruises. On this site, as with the rest, users can manage their trips and avail of deeper discounts via Express Deals on hotel deals and get to discover further deals on other travel needs. What sets them apart is their Name Your Own Price, where travellers bid on hotels and cars for up to 60% off on hotels and up to 40% off on rental cars.
Priceline features insider look into travellers' fun excursions, featuring vacation deals or packages and special deals on hotels and more.
You have options to choose from more than 440 airlines to fly with JustFly. JustFly caters to a mostly United States demographic, and is emerging as one of the most popular online travel agencies thanks to their noticeably cheap prices for travel services flights, hotels, cars. Should you wish to cancel, JustFly introduces an extended cancellation policy and further insures travellers with instant e-ticket delivery like Canada contemporary FlightHub.
JustFly sells the lowest-priced air tickets in an efficient user-friendly environment, constantly pushing their technology to the boundaries up and ahead the rest of the online travel agencies.
FlightHub offers Canada's cheap travel deals, giving Canadians a place to get the more deals on travel services for the best prices. This world-famous online travel agency, flagship brand, and Canada's go-to travel website offers flights, cars, hotels, and even travel insurance.
They work around the clock to serve all their customers, and feature instant e-ticket delivery that eliminates people waiting around for an email confirmation, 24/7 customer support, instant reminders on your travel needs, and constantly updated deals.
Expedia, founded and based in Washington, United States of America, is one of the biggest travel companies of the world. Added to the traditional searching and booking for flights, hotels, flight+hotel, and cars, users can check out Things to Do and Discover. According to their site, Expedia searches more than 321,000 hotels worldwide, and you can save big when you book a flight and hotel at the same time.
Another feature is their Price Guarantee, or you're refunded twice the difference. Also, they have Last Minute Deals open for booking at all times at the greatest prices.
This site has flights, hotels, cars, cruises, vacation packages, along with activities, event tickets, and deals. Cheaptickets is a subsidiary of Expedia.
Search for combination packages on their search engine is clearly signposted. True to their name, users can save bigger when they book flights and hotels at the same time. Furthermore, with CheapCash, travellers can earn cash when they book on select flights with Cheaptickets, to be redeemed towards eligible hotels within 30 days.
Through Travelocity, you can reserve from more than 140,000 hotels and 130 airlines. In common with the rest of the travel websites, you can get hotels, car services, flights, cruises, vacation packages, vacation rentals, travel deals, as well as check out things to do and discover. One reviewer in particular noted their somewhat steeper prices, but Travelocity makes up for this with rewards.
Their end can change or update your travel plans for no extra fee and price match, which further make up for their difference in price.
With Hotwire, people can search for hotels, cars, flights, and vacations. They have a mobile app for smartphones and tablets for faster easier booking and amazing deals. Book two or more travel services in a bundle for great savings. Whatever vacation you have in mind, Hotwire's got people covered with insider knowledge and a mission to help travellers get more out of travel with less cost.
Check out their deals for cars and hotels organised by place. Hotwire also crafted their Hot Rate deals for travellers' deepest discounts.
This is TripAdvisor company, taking after the latter's website appearance. In a pretty straightforward manner, users are met with options for flights, hotels, vacation packages, cars, and cruises. Booking a flight and hotel will give you added savings. Booking with Bookingbuddy comes at no booking fee, and has a Best Price Guarantee.
Searching through their search engine is clear, offering you the option to be alerted on changes in your route.
Get affordable fare on CheapOair for flights, hotels, cars, cruises, and vacations. Though they offer the usual services like Best Price Guarantee and 24/7 customer support, their deals really are cheaper than most. You can browse through their travel deals for ideas of where to go ahead to next or view the best prices falling under a number of categories such as the following: last minute flights, domestic flights, international flights, top 25 deals.
Reviews from CheapOair customers can give visitors an insider look or idea about the service.
On Opodo, people can search for flights, flight+hotel, cars, and hotels. You can check out your most recent searches at the upper tab of Opodo's website, while you can also check out their latest deals and their best prices or destinations.
Those who sign up to their app, on the other hand, can get instant and free alerts on all their flights.
Taking travel agencies online was a great move, and had impacted the tasks and details one can accomplish.
What can you say about our list of top 10 online travel agencies for 2017? Do you agree with the online travel agencies we featured, have favorites of your own, or simply want to share your experiences? If you have anything to add, feel free to leave your comments here below!
Nice write up, I find myself agreeing with most of your first impressions. really cool and informative.
I started using travel websites a few months ago, and find that they really help me save, especially on impt. flights.


and a home that can best meet that child's needs.
Our licensing and training departments work hand-in-hand together, to ensure a smooth and expedited process.
At Wesley House, we have created an innovative approach to foster care that puts the children in our community first.
Get involved today and find out the many different ways that you can help to change a child's future for the better.
Every day in Florida children are removed from their homes because their parents are not able to provide the love and care they need – sometimes because they didn't learn good parenting skills; sometimes because of substance abuse, mental illness, and domestic violence; sometimes because of all of the above.
As a foster parent, you can give a child the love and security they need. You can play a significant role in helping families heal and reunite and, when that is not possible, you can help children transition to a new home and a new family.
While children are in foster care, Wesley House provides Independent Living courses in two stages, one for ages 13 through 15 and a second for ages 16 through 17, to teach them skills to help them graduate successfully to independence.The Independent Living courses assist foster youth to develop practical skills in budgeting and money management, resume preparation, job networking, banking, grocery shopping and nutritional meal reparation. Committed business and community leaders share their expertise with our foster children during these sessions.


The team at Fidgets are experts in making your children's hair cut the best experience possible. Whether it be a tidy up trim or a whole new look the staff at Fidgets will do their best to create the look you're after.
The team are well qualified, with many years of experience in Barbering, Styling, Cutting and Colouring so you definitely know you're in safe hands.
Here at Fidgets we know your first hair cut can be a little daunting but we are here to help and reassure.
Our staff are experienced in working with children and young adults with additional needs and our team are Autism friendly.
So, whoever you are, come to Fidgets its more than just a hair cut its a fun experience for the whole family.
Bobbie joined our team in April 2013 and works full time as a stylist, she is also a trained barber. Bobbie is very patient and understanding with fidgets and wrigglers who don't like having their haircut, she sings songs and nursery rhymes to try and calm the little ones to make their experience as stress free and enjoyable as possible – whether it be in a character car, on a parents knee or whilst playing with toys! Bobbie will go above and beyond to ensure your child leaves with a full haircut.
Jenny joined our team in April 2013 and is our Level 3 senior stylist who also went on to do her Level 4 in hairdressing management. As well as being a trained barber Jenny also specializes in hair extensions and colour correction making her a popular choice with the mums and dads who book in with her. Jenny is very experienced with little fidgets who find it hard to sit still, she is always relaxed and is good at calming and reassuring little ones.
Gemma joined our team in 2014 and works part time in the salon. As well as being a level 3 stylist, she is also a trained barber. Gemma is kind, caring and the children love her soothing voice. Gemma often uses her skills to create fantastic styles for girls competing in dance competitions up and down the country, she is super fast and precise at plaiting and braiding – making her a firm favorite for little girls wanting hair like princess Elsa! She has also created beautiful hair for numerous wedding parties. Being a mum of two girls herself, she is very patient and understanding.
Laura is the newest addition to the Fidgets team and joined us in March 2019, she has had plenty of experience cutting children's hair and is a level 3 qualified hairstylist and barber. Laura is very calm and understanding and is great at keeping little ones entertained. Laura has a little girl so is great at doing plaits for all the little princesses that come into the salon.
Kelly has been with us since February 2018. She is our head party organizer and host. Having two little fidgets of her own Kelly knows how to entertain little ones and is well liked by all of the children that visit our salon. She is bubbly, friendly and fun to be around, we think Kelly is a great addition yo our Fidgets family.
Courtney has been with us since March 2017 and is our Level 2 junior apprentice. She has already qualified in barbering and is keen to complete her Level 2 in ladies hair dressing. She is the youngest member of the Fidgets family aiming to progress and eventually become a permanent barber at Fidgets – as she loves working with children! Courtney also co-hosts all of our fantastic Fidgets party events.


If you're a hearing health care provider please fill out the information below. Find out more about how the Brightridge Network can help refer people to your practice. We will email network guidelines and then follow up with a phone call to answer any questions you may have. Thank you for your interest in being a part of the Brightridge Network.


text:
J Best Books
poetry, articles and science fiction
Moppers Anonymous
Poetry Vol 1
Giants of the North
Childe Harold: Pocket Edition
Don Juan: Pocket Edition
I accept Crypto currency!
Selling Poetry Online?
Selling a collection of poetry online, at first thought, can seem a rather overwhelming task. We live in an age of TV shows at our fingertips, an internet full of YouTube clips and more information than we could ever want, need or care about. We're trained to skim read as we search for the answers we seek among the infinite pages of words. Clicking a thumb up or thumb down can be the difference between finding our audience and getting buried – very quickly – beneath the mountain of 1s and 0s. How, exactly, does a poet survive selling poetry on this fickle yet unavoidable battlefield called the internet?
(2018 Edit) Maximizing profits from e-books:
To learn how I keep up to 99% revenue from each e-book sale using other services alongside Amazon, check out my new Manual here: Maximizing eBook sales profits
Typical Search Results for Self Published Poetry Collections
With approximately 5,000 eBooks and 160,000 web-pages being launch each day, trying to stand out among them can seem like a daunting task. There is good news for those passionate about the written word, however. While things like Search Engine Optimization, reviews, and blog interviews are invaluable to poetry sales online, there's also an external force that is sweeping the online market.
This massive influx of new daily content brings with it waves of frustration, impatience and disappointed readers. The ability for anyone to post content to the internet has quickly created an ocean of garbage, and people are getting tired of wading through its stench. Although social media updates may have their place, an emoticon is no substitute for the emotion a poem can induce. A cat video, whilst mildly entertaining, cannot provoke thought, nor provide a lasting effect on its viewer. People are no longer satiated by bite-size tweets, no matter how many they consume, and the want for honest, good quality content is seeping back into general society. The fad is dying, novelty is wearing off, and now is the time to get your poetry online.
Many unheard of poets are releasing their works on forums and blogs- many kindle collections are popping up and are priced between $2 and completely free. With so much poetry available, there's no reason that new gems shouldn't be getting discovered daily. How many have you discovered? As a writer, I have learned the hard way that there are no secret tips to just sell poetry to the poet's community. You and I, we are the community, and with that comes a responsibility. If we don't want poetry to die in this digital age, then we need to keep it alive. How can we expect others to read poetry collections if we, the poets, won't do the same ourselves?
Unless we want poetry to stop with history's heavyweights, then this supportive network is crucial to our art. If a masterpiece is written in the forest, but no one's around to read it…
Positive communication between readers and other poets can only have a positive effect on the community and on our works. Even if you are selling your poetry online, interaction is paramount. Read poetry, review poetry, join forums (such as The Poet Sanctuary and Reddit's writing subs) and talk to poets and poetry readers. Breathe life into what you love, and it will in turn breathe life back into you and your written words. It's not a quick fix solution, it's hard work, but if all readers and writers share this load, there's no reason that our poetry can't thrive on the spindles of the internet.
Jonathon Best is a writer and poet from Perth, Western Australia. You can read his poetry free at jbestbooks.com.
His poetry collection: Dreams, magic and life is now available at Amazon.
Jon Best
Jonathon Best is a writer and poet from Perth, Western Australia. You can read his poetry free at jbestbooks.com
amazon, ebook, poetry, selling, writing
Jungle Yard
Recent Posts on: J Best Books
Elephant's Tear
On edge of dry creek's inlet, A shadow plods on by – The rider but a silhouette against an orange sky. His garments were in fading, With tassels ready frayed, Though none of this could quite be seen amidst the coming shade. For, blooming in the distance, Was man and mount's friend, one. The only […]
A dark power is sweeping across the land.
Our lone hero hears of a strange disturbance in the forests, and travels north to lend aid.
What he finds there is more than he can comprehend, but is it all as it seems?
A tale of adventure and magic, told in verse…
Shadow Creatures
A yawn, a stretch,
The vampires breathe
And look towards the moon.
The red of blood
and smell of fear-
Their instincts are in tune…
Statue and Stone
The beast that drags is belly raw
And eats with rounded teeth.
Oh, to be as the statue, grand –
To rule atop this wretched land;
Alas, it crawls beneath.
The walls, a mess of colour, always changing, always true,
Though no one sees the messages left hidden in plain view,
Then once again the sunlight fades and litter floods the shore-
And drags behind the world's unseen who paint the murals new…
Click to follow this writer and receive notifications of new posts.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: Unexpected SerializationException when using IRequiresRequestStream We have a request Dto which is defined as this:
public class AddDocumentByUploadBinaryRequest: AddDocumentRequest, IRequiresRequestStream
{
    public string FileName { get; set; }

    public Stream RequestStream { get; set; }
}

And AddDocumentRequest class definition is:
public class AddDocumentRequest: IReturn<ResponseBase>
{
    public Guid Id { get; set; }

    public string[] Tags { get; set; }

    public Guid? LotId { get; set; }

    public Guid? ContactId { get; set; }

    // never need it to attach document to existing bill
    //public Guid? BillId { get; set; }

    public Guid? FolioId { get; set; }

    public Guid? JobTaskId { get; set; }

    public Guid? TaskId { get; set; }

    public Guid? TenantInvoiceId { get; set; }

    public Guid? InspectionTaskId { get; set; }

    public Guid? InspectionReportAreaId { get; set; }

    public Guid? InMessageId { get; set; }

    public DocumentTypes? DocumentType { get; set; }

    public EventLinkTypes? DocumentArea {   get; set; }

    public Guid? MessageId { get; set; }

    public Guid? LinkId {
        get {
            switch (DocumentArea) {
                //case EventLinkTypes.Bill:
                //  return BillId;
                case EventLinkTypes.Contact:
                    return ContactId;
                case EventLinkTypes.Inspection:
                    return InspectionTaskId;
                case EventLinkTypes.InspectionReport:
                    return InspectionTaskId;
                case EventLinkTypes.InspectionReportArea:
                    return InspectionReportAreaId;
                case EventLinkTypes.Job:
                    return JobTaskId;
                case EventLinkTypes.Lot:
                    return LotId;
                case EventLinkTypes.Task:
                    return TaskId;
                case EventLinkTypes.Message:
                    return MessageId;
                default:
                    return FolioId;
            }
        }
    }
}

The API works most of the time but occassionally, we get "Unable to bind to request" error with the details like this:

Unable to bind to request 'AddDocumentByUploadBinaryRequest'; Error code: SerializationException; Field name: ���iD�����; Message: '�Z~�l�_$�6j�Ǒ�Xƾ��ԯ㵂���;W;/�����k� 6��;ׯG$�O�`�V�j��a�Z�[S幉>�>����[An��O
  X�n$�<�p�H��GN��[���c�|5�#��

?�' is an Invalid value for '���iD�����'
The captured error message suggests that the binary body was unexpectedly deserialized and failed.
The client device always uses Content-Type: application/octet-stream header. I have read relevant source code of ServiceStack but I couldn't think of why the binary body would still go into deserialization.
Any ideas would be appreaciated.

redpajama_set_name: RedPajamaStackExchange


Improved flood storage. The redesigned golf course will store water during rainstorms. We will also move and widen the Frank and Poet Drain corridor, helping to protect upstream residences and businesses from storm water backups that have become more frequent.
Updated and modernized golf course. We will completely redesign and rebuild the southern 18 holes of Highlands which, at 40, no longer provides the amenities today's golfers want. The new layout will include new water features and landscaping for a park-like setting.
Wildlife habitat. Landscaping along the relocated Frank and Poet Drain will screen the RLP and provide wildlife habitat.
Continued revenue and lower taxes. The RLP will contribute to the city general fund for at least 12 more years, keeping property taxes low while maintaining the same high level of city services, including fire, police, and library.
Clean energy. The new disposal area will generate additional landfill gas. A private company buys it from us and converts it into clean energy.
In June, 2016 the Riverview City Council passed a resolution (host community agreement) agreeing to host the expanded landfill. (The current landfill is composed of seven segments, or cells. The expansion is sometimes referred to as Cell 8.) In the next step, the city will create a conceptual plan for the development, to include the overall golf course renovations along with the Cell 8 limits and grading, for city council approval.
The city can use the approved conceptual plan to start applying for permits and to develop more detailed plans. The permitting process could take as long as two years.
In addition to the city, the Michigan Department of Environmental Quality (MDEQ), Wayne County Department of Public Services (WCDPS) and the US Army Corps of Engineers must review, comment on and approve the proposed plans before construction can begin.
If everything goes smoothly we could complete the permit process in the second quarter of 2018. Construction could begin on the golf course renovation and relocation of the Frank and Poet drain late that year, followed by the first phase of Cell 8.
The redesigned golf course will store storm water in two ways: In the Frank and Poet Drain itself and in the layout and grading of the golf course. The flood storage will reduce the risk of major storm events to residents of Colonial Village, Pheasant Run, The Glens and The Forest neighborhoods.
To add floodplain storage, we will widen the drain corridor and create low areas adjacent to the drain. As the water level in the drain rises, it will flow into the storage areas, instead of backing up into upstream neighborhoods. These storage areas will slowly drain out within 24 to 48 hours. The slight inconvenience to golfers will be offset by the benefits to homeowners. The new drain will also meander through the golf course, adding to existing water hazards, challenging players while providing habitat for wildlife and plants.
Storm water runoff from the landfill itself will most likely collect in two new sedimentation ponds located adjacent to the Frank and Poet Drain, on the landfill property. Wayne County storm water ordinances require that the ponds filter the runoff and discharge it at a controlled rate into the drain. The ponds will also contribute to the aesthetics of the landfill/golf course interface, and provide additional wildlife habitat.
The landfill is designed to ensure that only runoff from covered areas is allowed into the ponds and thus into the drain. During construction and operation, any storm water coming into contact with waste ("leachate") is carefully controlled and sent to the sanitary sewer.
Residents of The Forest subdivision will be most affected once construction begins. Residents on Coachwood Road can expect to see and hear construction equipment, the loss of trees and other vegetation and loss of green views for at least one season. The golf course will salvage as many old-growth trees as possible and add replacement trees.
Construction of the Frank and Poet Drain will take place simultaneous with the golf course renovation. The course and the drain will be allowed to revegetate and become established before play resumes.
The new portion of the RLP comes last and people really won't see much of it because of its distance from The Forest and the barrier of the golf course, trees, and the reconfigured drain. As we build it, we'll cover the exterior slopes with soil and plant grass seed. This is both a regulatory requirement and good environmental management practice. It will be several years after construction begins before waste placement in the new cell is even visible.
We will sell some of the landfill gas generated from Cell 8 to a private company. If there's more, the city could use it for other natural-gas powered projects, keeping any revenue that results.
Construction of the gas collection system will coincide with the landfill's construction. The design will also include conceptual plans for an energy project (based on input from the city), as well as backup destructive capacity (flares) if the energy project fails at any time. These systems are highly regulated and the city has an excellent compliance record on the existing RLP.
Leachate is the liquid waste produced as trash decomposes. It is non-hazardous and generally can be disposed as industrial wastewater, treated at the wastewater plant. It is continuously collected and removed, following strict standards. The base liner for Cell 8 will contain the most current control mechanisms to remove the leachate from the cell and dispose of it. We will coordinate with the wastewater treatment plant during permitting and construction to include the new leachate source in the site's industrial discharge permit at lowest cost. We screen incoming waste to prevent contamination but still monitor the leachate to ensure compliance with the permit requirements.
We estimate Cell 8 will add approximately 15 years (15 million tons) to the Riverview Land Preserve's capacity, but it could be more if decomposition rates and recycling increase and we improve methods of trash placement and compaction.


text:
package synergynetframework.appsystem.contentsystem.items;

import java.net.URL;
import java.util.HashMap;
import java.util.LinkedHashMap;

import synergynetframework.appsystem.contentsystem.ContentSystem;
import synergynetframework.appsystem.contentsystem.items.implementation.interfaces.IRoundFrameImplementation;
import synergynetframework.appsystem.contentsystem.items.utils.ImageInfo;

/**
 * The Class RoundFrame.
 */
public class RoundFrame extends RoundContentItem implements
		IRoundFrameImplementation {

	/** The Constant serialVersionUID. */
	private static final long serialVersionUID = -9030421443289334813L;

	/** The URL to image map. */
	protected LinkedHashMap<URL, ImageInfo> URLToImageMap = new LinkedHashMap<URL, ImageInfo>();

	/**
	 * Instantiates a new round frame.
	 *
	 * @param contentSystem
	 *            the content system
	 * @param name
	 *            the name
	 */
	public RoundFrame(ContentSystem contentSystem, String name) {
		super(contentSystem, name);

		// frameImplementation =
		// (IFrameImplementation)this.contentItemImplementation;
	}

	/*
	 * (non-Javadoc)
	 * @see
	 * synergynetframework.appsystem.contentsystem.items.implementation.interfaces
	 * .IRoundFrameImplementation#drawImage(java.net.URL)
	 */
	public void drawImage(URL imageResource) {
		if (imageResource != null) {
			URLToImageMap.put(imageResource, new ImageInfo(imageResource,
					getBorderSize(), getBorderSize(), ((int) (2 * getRadius()))
							- (2 * getBorderSize()), ((int) (2 * getRadius()))
							- (2 * getBorderSize())));
			((IRoundFrameImplementation) this.contentItemImplementation)
					.drawImage(imageResource);
		}
	}

	/*
	 * (non-Javadoc)
	 * @see
	 * synergynetframework.appsystem.contentsystem.items.implementation.interfaces
	 * .IRoundFrameImplementation#drawImage(java.net.URL, int, int, int, int)
	 */
	public void drawImage(URL imageResource, int x, int y, int width, int height) {
		if (imageResource != null) {
			URLToImageMap.put(imageResource, new ImageInfo(imageResource, x, y,
					width, height));
			((IRoundFrameImplementation) this.contentItemImplementation)
					.drawImage(imageResource, x, y, width, height);
		}
	}

	/**
	 * Gets the image resources.
	 *
	 * @return the image resources
	 */
	public LinkedHashMap<URL, ImageInfo> getImageResources() {
		return URLToImageMap;
	}

	/*
	 * (non-Javadoc)
	 * @see
	 * synergynetframework.appsystem.contentsystem.items.implementation.interfaces
	 * .IRoundFrameImplementation#getImages()
	 */
	public HashMap<URL, ImageInfo> getImages() {
		return URLToImageMap;
	}

	/*
	 * (non-Javadoc)
	 * @see
	 * synergynetframework.appsystem.contentsystem.items.implementation.interfaces
	 * .IRoundFrameImplementation#removeAllImages()
	 */
	@Override
	public void removeAllImages() {
		URLToImageMap.clear();
		((IRoundFrameImplementation) this.contentItemImplementation)
				.removeAllImages();
	}

	/*
	 * (non-Javadoc)
	 * @see
	 * synergynetframework.appsystem.contentsystem.items.implementation.interfaces
	 * .IRoundFrameImplementation#removeImage(java.net.URL)
	 */
	public void removeImage(URL imageResource) {
		URLToImageMap.remove(imageResource);
		((IRoundFrameImplementation) this.contentItemImplementation)
				.removeImage(imageResource);
	}
}

redpajama_set_name: RedPajamaGithub


The Lower Colorado River Authority extended lake lowerings by five days after recent rains prompted the partial opening of a floodgate on Tom Miller Dam.
The LCRA began the drawdown of lakes LBJ and Marble Falls on Dec. 30 but paused it Jan. 2. Originally, the LCRA estimated lakes LBJ and Marble Falls to reach their lowered level by Jan. 6 and 9, respectively.
As of Jan. 4, the LCRA expected Lake LBJ to reach its lowered level by Jan. 9 and Lake Marble Falls on Jan. 12. Those dates could change if the area receives additional rain, according to the LCRA.
The pause also has affected the refill dates. The LCRA originally expected to begin refilling the lakes on Feb. 19. Now, the refill is scheduled to begin Feb. 24 and conclude Feb. 28.
Lake LBJ is about 2 feet below its normal operating range at the moment and, during the lake drawdowns, will be 4 feet lower than normal.
Lake Marble Falls is about 3 feet below its normal operating range and, when fully lowered, will be about 7 feet lower.
The water released from both lakes is about 26,000 acre-feet of water, which will raise the level of Lake Travis about 1.3 feet.
The lakes are being lowered so landowners can perform maintenance and repairs to lakeside property. Property owners recovering from the October flood also will be able to more easily remove debris, dredge, and work on other infrastructure during the drawdowns.
Full lake lowering information, including permitting requirements, are available on the LCRA's Lake Lowerings website.


text:
What is employee confidentiality policy?
The Employee shall protect the Confidential Information by using the same degree of care, but no less than reasonable care, to prevent the unauthorized use, dissemination or publication of the Confidential Information as the Employee uses to protect its own Confidential Information.
What will be your policy on confidentiality?
Employees Should Remember to: Keep confidential information locked at all times. Shred confidential documents once they are no longer needed. View confidential information only on secure devices. Disclose confidential information to other employees only when it is necessary and authorized.
What is confidential information policy?
It is the responsibility of every employee to respect and maintain the security and confidentiality of Confidential Information. A violation of this policy may result in disciplinary action. "Confidential Information" includes information in any form, such as written documents or records, or electronic data.
How should an employee handle confidential information?
Ensuring that confidential information is always locked away at night, and not left unattended during the day; Password-protecting sensitive computer files; Marking confidential information clearly as such, and ensuring that paper copies are shredded before disposal; and.
What is the purpose of a confidentiality policy?
The purpose of the Confidentiality Policy is to ensure that all staff, members, volunteers and users understand the Organisations requirements in relation to the disclosure of personal data and confidential information.
How do you ensure confidential information is protected?
7 Practical tips for protecting confidential information
Know who you are disclosing information to.
Clearly label all confidential information as "confidential".
Use passwords and encrypted files for electronic documents.
Provide initial and ongoing advice to individuals.
What are five 5 ways of maintaining confidentiality?
5 important ways to maintain patient confidentiality
Create thorough policies and confidentiality agreements.
Provide regular training.
Make sure all information is stored on secure systems.
No mobile phones.
Think about printing.
What is considered a breach of confidential information?
A breach of confidentiality is when data or private information is disclosed to a third party without the data owner's consent. In many professions, protecting confidential information is essential for maintaining trust and ongoing business with your clients.
What are the main points of confidentiality policy?
The principle of confidentiality is about privacy and respecting someone's wishes. It means that professionals shouldn't share personal details about someone with others, unless that person has said they can or it's absolutely necessary.
How can an employer protect their confidential information?
share or otherwise use any of your business' confidential information.
Ensure Employees Return All Confidential Information in Their Possession.
Protect Your Digital Files Via Encryption.
Remedies.
What is considered confidential information in the workplace?
Confidential information handled in a workplace includes things like client records, business planning and forecasting, employee records, and information collected in the course of research and development.
Confidentiality policies that specifically prohibit employee discussions of terms and conditions of employment (i.e., wages, hours or workplace complaints) or that employees would reasonably understand to prohibit such discussions may be deemed unlawful.
Can an employer require you to keep your salary confidential?
That implies that in an employment contract, an employer cannot include provisions stating that salary is to be kept confidential. Likewise, employers cannot hold employees liable for punitive measures if they share information on salary or bonuses.
https://www.youtube.com/watch?v=i_BK3ppBW4Q
What is employee confidentiality policy? The Employee shall protect the Confidential Information by using the same degree of care, but no less than reasonable care, to prevent the unauthorized use, dissemination or publication of the Confidential Information as the Employee uses to protect its own Confidential Information. What will be your policy on confidentiality? Employees…
Does CIN 3 have symptoms?
Where to buy Black Cement Air Jordan 3?


While the best defense against data loss is redundant and real-time backup, we understand that sometimes data loss sneaks right up on you. Whether your vacation pictures didn't make it safely from your camera to your computer or a bumbling roommate deleted the paper you've been working all week on, having emergency data recovery tools handy is crucial to getting your data back before it's gone forever. Earlier this week we asked you to share your favorite data recovery tools with us. We tallied up your favorites and now we're back with the nominees for best free data recovery tool.
TestDisk is a powerful open-source tool for recovering your data. Not only can TestDisk perform basic file recovery like undeleting accidentally deleted files from FAT, NTFS, and ext2 file systems, but it comes with a host of additional functionality. With TestDisk you can recover your boot sector from a backup, rebuild your boot sector, fix FAT tables, fix your MFT, locate the ext2/ext3 backup SuperBlock, copy deleted files from partitions to recovery media, and find lost partitions in dozens of formats to help you locate your lost data. It's a command line tool, so there's no GUI, but the menus and the documentation in the wiki should get you started without much trouble.
Recuva is a user-friendly Windows-based tool. When you run Recuva, you can resurrect missing files using either the file-recovery wizard or the application's manual mode. The file-recovery wizard is handy when you're sure your data is gone but you're not quite sure where it went or how to get it back. The wizard lets you narrow your search type to pictures, music, documents, video, or all files, and you can set the search location to everywhere on your computer, removable media only, in My Documents, the Recycle Bin, or a specified location. If you don't need the wizard you can jump right into manual mode and get to work searching where you know the file should be. Recuva uses a green/yellow/red light system to indicate how probable the recovery of your files will be, and when available, it can provide previews image files available for recovery. Recuva also includes a tool to securely wipe files you find, handy if you're attempting a file recovery just to ensure the files are actually dead and gone.
PhotoRec is a companion program to TestDisk, also included in this Hive Five. Like TestDisk, this app is also devoid of a GUI, but likewise is quite powerful at file recovery. We're including PhotoRec independently of TestDisk because many users distinctly prefer PhotoRec as a safer alternative when deep disk recovery isn't necessary. This recovery tool won't mess with your partitions or help you rebuild your master boot record; it will, however, dive into your disks in a safe, read-only mode and ignore partitions and file systems in an effort to seek out your missing files. PhotoRec focuses on file types, is operating-system agnostic, and despite its name, isn't relegated to just photos. Overall, PhotoRec is a powerful tool for quickly and safely copying your deleted files to another disk.
Restoration is a tiny, no-frills, portable recovery tool. You can use it in all versions of Windows and Windows file systems. It lacks some of the advanced functionality of other nominees but does have basic file-name search and the ability to sort by file parameters such as size and filename. Despite its tiny size, it performed just as well as the other nominees when tasked with restoring files from our test disks. Restoration weighs in at a mere 406k and would make a great addition to any Windows-based USB toolkit.
Undelete Plus used to be commercial software but has gone on a lengthy "limited time offer" freeware run. This file recovery app works on all versions of Windows and incarnations of the FAT and NTFS file systems. Like Recuva, Undelete Plus assigns a recovery probability to files it finds based on how damaged the file is. You can sort files by type, set filters based on time and size to avoid sifting through every deleted file on your disk, and keep folder structures intact when you perform your recovery.
Now that you've had a chance to look over the contenders for best free data recovery tool, it's time to cast your vote and see who wins the crown.
If you have a file-recovery tip, technique, or tool you're dying to share. sound off in the comments and help your fellow readers get their data back.