\section{Additional model information}
\label{sec:model-appendix}
Additional information about the models used in our work and their training datasets can be found in \Cref{tab:model-pretraining-data,tab:datasets}. We can observe that VisualBERT has been trained on a data amount that is quite small compared to those of CLIP-BERT and LXMERT.

It is also worth noting on the different backbones of the models. CLIP-BERT is a single-stream multimodal model with a CLIP backbone for visual processing. LXMERT is a dual-stream multimodal model with a Faster R-CNN detector backbone. While VisualBERT is a single-stream model that also utilizes Faster R-CNN detector backbone. Since CLIP has been trained on the immense WIT dataset, the backbone data sizes differ greatly between CLIP-BERT and the other multimodal models.

\begin{table*}
    \centering
    \begin{tabular}{l|l l l|l|l}
    \hline
    Model & Text & Visual text & Images+Text & Backbone & Training objectives \\
    \hline
    BERT & 80M &  &   & & MLM, NSP\\
    \hline
    CLIP-BERT-implicit & 80M &  & 4.7M & 400M & MLM \\
    CLIP-BERT-explicit & 80M &  & 4.7M & 400M & MLM \\
    BERT-CLIP-BERT-train & 80M & 4.7M &  & & MLM  \\
    \hline
    LXMERT &  &  & 9.2M & 0.1M & MLM, RFR, DLC, \\
    & & & & & ITM, IQA\\
    BERT-LXMERT-train & 80M & 9.0M & & & MLM \\
    BERT-LXMERT-train-scratch &  & 9.0M & & & MLM \\
    \hline
    VisualBERT & 80M &  & 1.7M & 0.1M & MLM, ITM \\
    \end{tabular}
    \caption{An overview of the pre-trained models, the sizes of their training datasets and their pre-training objectives. The sizes are measured in number of training samples. The backbone column indicates the training data sizes for the image processing backbones of the models. For the training objectives, ITM refers to Image-Text Matching, RFR to RoI-Feature Regression, DLC to Detected Label Classification, MVM to Masked Visual Modeling and IQA to image QA.}
    \label{tab:model-pretraining-data}
\end{table*}

\begin{table*}[]
    \centering
    \begin{tabular}{l|l|l|l|l}
        Dataset & Data sources & \# of text & \# of images \\
        \hline
        CLIP-BERT V+L & MS COCO, SBU Captions, VG-QA, CC & 4.72M & 2.91M \\
        LXMERT V+L & MS COCO, VG, VQA, GQA, VG-QA & 9.18M & 0.18M \\
        VisualBERT V+L & MS COCO, VQA & 1.27M & 0.12M
    \end{tabular}
    \caption{The vision-language datasets on which the multimodal models originally were trained. More information about the datasets can be found in the articles that introduced the models.}
    \label{tab:datasets}
\end{table*}

\section{Additional information on Visual Property Norms}\label{sec:normdata-info}
Information about the different segments and number of entries per segment in the Visual Property Norms can be seen in \Cref{tab:pf-segments}.

\begin{table}
    \centering
    \begin{tabular}{r|r|r r r r}
        PF & entries & \emph{has} & \emph{has a} & \emph{made of} & \emph{is}\\
        \hline
        2 & 6,541 & 1,675 & 1,190 & 1,176 & 2,500 \\
        5 & 3641 & 1,016 & 642 & 760 & 1,223\\
        10 & 2001 & 583 & 347 & 509 & 562 \\
        20 & 613 & 169 & 88 & 209 & 147\\
        30 & 27 & 5 & 2 & 10 & 10
    \end{tabular}
    \caption{The data segments segmented based on production frequencies together with their number of entries. The entries are calculated as the number of feature-concept-label entries, where there can be several features belonging to the same feature and concept. The PF column indicates the production frequency threshold for each segment, all features with a production frequency higher or equal to this threshold are included in the segment. We also list the number of labels per feature relation type.}
    \label{tab:pf-segments}
\end{table}

\section{Additional results on Visual Property Norms}
\label{sec:normdata-appendix}

Additional model results on the Visual Property Norms can be found here.

\Cref{fig:score-per-relation} indicates model performance per feature relation across the production frequency segments. We can observe how the models show the best performance for the \emph{is made of} relation, which arguably can be associated more with visual perceptual properties.

\Cref{fig:score-per-query} shows model score per query template across all production frequency segments, indicating that CLIP-BERT-implicit benefits from being more robust to different query templates. Additionally, these results indicate that BERT-LXMERT-train would have the best overall score on Visual Property Norms if the queries containing ``q: a'' were to be removed.

Lastly, \Cref{fig:score-normdata-colors} contains the results of the models on the color part of Visual Property Norms which has been filtered to only contain queries with gold labels describing colors. Here, we see some indications of a better performance of CLIP-BERT-explicit for colors. Potentially, the imagination capacity of this model is more helpful for queries with answers relating to more basic visual properties, such as color.

\begin{figure*}
    \centering
    \input{images/score_per_relation.pgf}
    \caption{The model scores on Visual Property Norms per feature relation. The error bars indicate the standard deviation of the model performance over the different query templates. The score has been calculated by masking the vocabulary of the models to only contain the possible answers of the task.}
    \label{fig:score-per-relation}
\end{figure*}

\begin{figure*}
    \centering
    \input{images/per_query.pgf}
    \caption{The score for each model on Visual Property Norms per query template. The score has been calculated by masking the vocabulary of the models to only contain the possible answers of the task.}
    \label{fig:score-per-query}
\end{figure*}

\begin{figure*}
    \centering
    \input{images/all_only_colors.pgf}
    \caption{The score for each model per production frequency segment on Visual Property Norms that has been filtered to only contain samples for which the correct answer is one or more out of 11 possible colors. The score has been calculated by masking the vocabulary of the models to only contain the possible answers of the task.}
    \label{fig:score-normdata-colors}
\end{figure*}
\section{Introduction}

Language models (LMs) trained on large amounts of textual data have shown great performance on several textual tasks \citep{devlin2018bert,gpt3}. However, recent work has illuminated limitations with text-only training of LMs. These limitations arise from a lack of meaning \citep{bender-koller-2020-climbing} and experience \citep{bisk-etal-2020-experience}, together with the problem of reporting bias \citep{gordon2013reporting}. Multimodal training has been identified as one way to create models that do not suffer from the aforementioned limitations \citep{paik-etal-2021-world,huang2021makes}. While several multimodal models have been developed \citep{tan-bansal-2019-lxmert,li2019visualbert,li2020oscar}, few evaluation methods exist that can tell us whether multimodal training mitigates text-only training limits.
\begin{figure}[t]
    \centering
    \includegraphics[scale=0.18]{images/overview.png}
    \caption{We introduce the two evaluation tasks Memory Colors and Visual Property Norms for measuring visual commonsense knowledge in a LM.}
    \label{fig:overview}
\end{figure}

If we wish to successfully create multimodal LMs that learn from more than text, we need a way to evaluate them for what we expect them to have learned from their multimodal training.

One hypothesis is that multimodal training should aid LMs in learning commonsense knowledge \citep{zhang-etal-2021-need}. There are several text-only evaluation tasks that aim to measure the commonsense knowledge in LMs \citep{zellers-etal-2019-hellaswag,Zhou_Zhang_Cui_Huang_2020}, but none of them focus explicitly on the commonsense knowledge learned through training on more than text.

In this work, we focus on models trained on images and text, denoted \emph{vision-and-language models}. We reason that if there is any additional information to be learnt from a visual modality it should firstly be basic visual commonsense knowledge. That is, visual conceptual knowledge that is viewed as commonsense by humans, and thus not attainable from text alone due to reporting bias.

We propose a simple method for measuring the visual commonsense knowledge of a model using two zero-shot masked language text-only tasks, depicted in \Cref{fig:overview}. The first task is the Memory Colors evaluation task \citep{norlund2021} and the second we create based on the visual features in the Centre for Speech, Language and the Brain (CSLB) concept property norms dataset \citep{devereux2014centre}. We refer to the latter task as the Visual Property Norms evaluation task. We complement our work with the results of four vision-and-language models and four baselines on these two tasks.

\section{Evaluation Tasks}

Our aim is to evaluate models for visual commonsense knowledge. To do this we make use of the existing Memory Colors evaluation task described in \cref{sec:memory-colors}, and introduce a new evaluation task, Visual Property Norms in \cref{sec:visual-property-norms}. Memory Colors is smaller than Visual Property Norms and specifically focuses on visual information related to the color of different concepts, so it is potentially easier. We include both tasks to get a performance curve over increasing difficulty.

Common for both tasks is that they contain queries in English relating to visual properties of tangible concepts and that these queries are based on the knowledge of multiple human participants. Therefore, the tasks can be considered to evaluate a basic aspect of visual commonsense knowledge

Also common for both tasks is that they use textual templates containing a \texttt{[MASK]} token to be predicted by a model in a cloze-style fashion, similarly to the method used by \citet{kassner-schutze-2020-negated} and \citet{petroni-etal-2019-language}. The advantages with querying the models in this fashion is that most LMs\footnote{Excluding autoregressive LMs.} already have been exposed to this type of query format, including most multimodal models. We can then evaluate any model in a masked language modelling fashion on these tasks without additional training or having to make model-specific adaptations, enabling easy evaluation for researchers who wish to use these evaluation tasks.

This form of cloze-style evaluation is also referred to as \emph{prompt-based retrieval}. The reliability of this method has recently been questioned by \citet{jiang2020know} and \citet{cao-etal-2021-knowledgeable} due to the query format sensitivity of LMs. To alleviate this issue, we evaluate the models using several different prompts for each of the two tasks.

\subsection{Memory Colors} \label{sec:memory-colors}

The Memory Colors evaluation task is a text-only zero-shot cloze test in English that evaluates a model for its knowledge of memory colors. It queries a model for the color of 109 typical objects using 13 different query templates. The task has been created with the help of 11 human participants, so to some extent it encodes human visual commonsense knowledge limited to colors. Some examples of queries can be seen in \Cref{fig:overview}.

We use the same evaluation metric as specified by \citet{norlund2021}, i.e. the accuracy score after masking the model output for the 11 possible colors black, blue, brown, green, grey, orange, pink, purple, red, white and yellow.

\subsection{Visual Property Norms}\label{sec:visual-property-norms}

We also introduce a new cloze task in English to evaluate for visual commonsense knowledge, denoted Visual Property Norms. It is the largest query-based pure-language evaluation task capable of evaluating LMs for visual commonsense knowledge, containing 6,541 visual conceptual features produced by human participants.

We base it on the CSLB concept property norms dataset \citep{devereux2014centre} that contains the conceptual knowledge of 30 human participants for each of 541 concrete objects, with 123 participants in total. This knowledge is represented as a set of features per object, for which each feature is specified with a production frequency (PF). The PF describes how many of 30 participants produced that feature, so a feature with a high PF can be considered to be more apparent to the participants, since more came to think of it. All features are also categorized as either \emph{encyclopaedic}, \emph{functional}, \emph{other perceptual}, \emph{taxonomic} or \emph{visual perceptual}. \Cref{tab:normdata_examples} contains some examples of visual perceptual features in the dataset.
\begin{table}[h]
    \centering
    \begin{tabular}{l|l|l|r}
        Concept & Relation & Feature & PF \\
        \hline
        Cherry & has a & stalk & 17 \\
        Fern & is & green & 29 \\
        Hair & is & thin & 22 \\
        Plum & has & flesh & 9 \\
    \end{tabular}
    \caption{Some concepts and their visual perceptual features in the concept property norms dataset.}
    \label{tab:normdata_examples}
\end{table}

We create our evaluation task from the concept property norms dataset in a set of steps. Firstly, since our goal is to measure visual commonsense knowledge, we only make use of the \emph{visual perceptual} features. Since we wish to perform cloze tests through masked language modelling, only feature alternatives describable by one wordpiece from the BERT base uncased tokenizer are included.

Furthermore, we only include the four most common feature relations in the task. These are \emph{has}, \emph{has a}, \emph{made of} and \emph{is}. We then part the data into five different segments based on production frequency. This is done by thresholding the features for each concept such that only features with a PF above the set threshold for a certain data segment are included as gold labels in that segment.
The segments and their PF thresholds are listed in the appendix.

Lastly, we create queries from the concepts in each data segment using 8 different query templates, seen in the appendix. Some examples of Visual Property Norms queries can be seen in \Cref{fig:overview}.

Similarly to \citet{Weir-et-al:2020} we use the mean average precision (mAP) as our evaluation metric, since there may be multiple correct answers for each query in our evaluation data. We calculate this score for each concept and relation, per query template and production frequency segment. We then get a final score for each production frequency segment by taking the average score over all query templates and concepts per segment.
This metric is measured over a vocabulary that has been masked to only contain the 614 possible answer alternatives in the Visual Property Norms evaluation data.

\begin{figure*}[t]
    \begin{center}
    \begin{subfigure}[b]{0.9\textwidth}
    \caption{Memory Colors}
    \label{fig:memory-colors-results}
    \hspace{-0.8cm}\input{images/memory_colors_results.pgf}
    \end{subfigure}
    \begin{subfigure}[b]{0.9\textwidth}
    \caption{Visual Property Norms}
    \label{fig:normdata-results}
    \hspace{-0.8cm}\input{images/all.pgf}
    \end{subfigure}
    \end{center}
    \caption{The model accuracy on Memory Colors and model scores on Visual Property Norms per production frequency segment. The multimodal model results are depicted with warmer colors, and the unimodal model results are depicted in cooler colors. The error bars indicate the standard deviation of the model performance over the different query templates. The score has been calculated by masking the vocabulary of the models to only contain the possible answers of the task.}
    \label{fig:results}
\end{figure*}

\section{Models}

We evaluate four multimodal pre-trained models for their visual commonsense knowledge. These are CLIP-BERT both with and without imagination\footnote{The explicit version has the ability to ``imagine'' visual features when queried with text.}\citep{norlund2021}, a LXMERT base uncased \citep{tan-bansal-2019-lxmert} and VisualBERT
\citep{li2019visualbert}. We also evaluate four unimodal baseline models. These are a BERT base uncased pre-trained on English Wikipedia and BookCorpus, a BERT base uncased further trained on the pure-text part of the CLIP-BERT training data (BERT-CLIP-BERT-train) and two BERT base uncased models trained on the pure-text part of the LXMERT training data, one from scratch and one initialized from pre-trained BERT weights (BERT-LXMERT-train-scratch and BERT-LXMERT-train).

All models are to some extent based on the BERT base architecture and consequently share the same vocabulary and tokenizer. They are also of similar sizes with $\sim 110$M trainable weights, the exception being LXMERT with $\sim 230$M trainable weights.
Additional information about the models can be found in the appendix.

\vspace{-0.2cm}
\paragraph{Adapting the models for pure-text queries} The majority of current multimodal models have not been developed to be queried only with text. In this case, both CLIP-BERT and VisualBERT should work well with only removing their visual features input, since they are single-stream models. However, LXMERT is a dual-stream model that requires a visual feature input.
We handle the removal of visual information by simply removing the visual processing chain in LXMERT, making the language input the only input given to the Cross-Modality Encoder in the model. This would not work if we still wanted to use the model in a multimodal fashion, but we can make this adaption since we are only interested in querying the model for visual commonsense knowledge via language.

\section{Results}

The results of the models on our two evaluation tasks can be seen in \Cref{fig:results}. We format the analysis of the results around a set of questions.

\textbf{Do the multimodal models display more memory colors knowledge?}
  The multimodal CLIP-BERT-explicit model has the best performance on this task. So to some extent, yes. But it is worth noting that the unimodal BERT model trained on LXMERT training data is second best on the task, outperforming both LXMERT and VisualBERT, indicating a small multimodal advantage.

 \textbf{Is performance on Memory Colors indicative of performance on Visual Property Norms?}
 The ranking visible in \Cref{fig:memory-colors-results} does not entirely differ from that in \Cref{fig:normdata-results}.
 The main exception being CLIP-BERT-explicit, which has the best performance on Memory Colors, but is outperformed by most other models on Visual Property Norms. We perform a closer analysis of how these results compare by extracting Visual Property Norm results for colors in the appendix.

\textbf{Do the models perform better when evaluated on more apparent concept features?}
We can observe how the model performance unanimously increases with increased production frequency threshold in \Cref{fig:normdata-results}. Thus, it appears as though the models agree more with concept features that can be regarded as more apparent.

\textbf{Do the multimodal models contain more visual commonsense knowledge?}
The results in \Cref{fig:normdata-results} do not really indicate clear advantage of either unimodal or multimodal models. The multimodal model CLIP-BERT-implicit may generally have the best performance on the task, but the unimodal models trained on visual text data do not differ much in performance. For example, the unimodal BERT-LXMERT-train performs almost on par with CLIP-BERT-implicit.

This conclusion is similar to that of \citet{yun-etal-2021-vision-language}, who also compared vision-and-language models to text-only models trained on captions. They found that the models have similar performance with respect to their internal linguistic representations for general tasks.

These results do not mean that the idea of having models learn language from more than text has failed. They do however indicate that there is more work to be done on developing models that use multimodal pretraining to improve on their natural language understanding.

However, we cannot exclude the possibility in our work that the multimodal models suffer in performance due to a lack of visual feature input.
Future work investigating this would be valuable.

\textbf{Are the models sensitive to how they are queried?}
Prevalent for all models is that their performance varies greatly with how they are queried.  BERT-LXMERT-train may have the best performance on Visual Property Norms if queried differently. We evaluate the model performances depending on query template in the appendix. This highlights the importance of querying the models with different prompts, since the models may perform dissimilarily depending on prompt due to the degree of prompt-dataset fitness, as reported by \citet{cao-etal-2021-knowledgeable}.

\textbf{Does fine-tuning on visual language develop visual commonsense knowledge?}
In both \Cref{fig:memory-colors-results,fig:normdata-results} it is visible that unimodal model performance greatly improves with fine-tuning on visual text corpora. Potential explanations for this are that the models become more attuned to the task with fine-tuning, or that corpora from VQA and image captioning do not suffer as much from reporting bias compared to more common corpora. Thus, text that has been curated to explicitly contain visual information may suffice as a replacement for images.

\section{Related Work}

\citet{Weir-et-al:2020} also use the CSLB concept property norms to probe LMs for commonsense knowledge. Our work differs from theirs in that we focus on visual commonsense knowledge and evaluate several multimodal models for whether their multimodal training grants them additional visual commonsense knowledge.

\citet{norlund2021} also query a multimodal model for visual commonsense knowledge but with a focus on memory colors. \citet{paik-etal-2021-world} present similar work but with more focus on probing and reporting bias. In our work, we include general visual commonsense knowledge concepts and evaluate several multimodal models.

Additionally, \citet{iki-aizawa-2021-effect} evaluate several vision-and-language models on GLUE, to investigate the effect of an additional visual modality on the general linguistic capabilities of a model. Our work differs in that we evaluate the models specifically for visual commonsense knowledge.

Other tasks that have been developed to evaluate the performance of vision-and-language models are Visual Question Answering (VQA) tasks and Visual Commonsense Reasning (VCR) tasks \citep{balanced_vqa_v2, hudson2018gqa,zellers2019vcr}. Our work differs from these in that we evaluate for visual knowledge in models without conditioning on an image, to investigate whether the linguistic capabilities of a model improve from training on more than text. In the aforementioned tasks, the text prompts are always conditioned on an image provided with the prompt, obstructing equal comparisons with text-only models.

\section{Limitations}
Our work is limited to a subset of vision-and-language models, so the results found may not translate to all such model types. Also, since our evaluation utilizes prompt-based retrieval, its measurement accuracy depends on how well this method works for LMs.
Additionally, as previously mentioned, we do not investigate how well the multimodal models adapt to a unimodal input. Thus, our results depend on whether the models were functioning adequately with our method of adapting them to a unimodal input.

\section{Ethical Considerations}
Our work should not have any direct ethical implications, since we mainly introduce evaluation tasks and evaluate different models on them. We do however investigate visual conceptual perceptions based on data from a potentially small group of people whose world-view may be culturally different from that of other individuals. This means that we may encourage knowledge that benefits some people more than others.
Similar issues are discussed by \newcite{liu2021visually}.
Our investigation is limited to English-language models and datasets, limiting the generality of our conclusions.

\section{Conclusions}
We introduce new evaluation methods for measuring the visual commonsense knowledge in LMs and evaluate a number of multimodal LMs on these benchmarks. We find that there are no significant differences in performance between models trained on pure text and models trained on images and text. Most prominently, we find that a unimodal LM trained on image captions and VQA queries can attain a visual commonsense knowledge on par with that of a multimodal model.

We also confirm the results by \citet{jiang2020know} and \citet{cao-etal-2021-knowledgeable}, that LMs are sensitive to query format even when querying for commonsense knowledge. This casts some doubts on what is really measured in a model for a cloze task and whether we can reason about LMs as having knowledge. An interesting future step would be to investigate this further and see if it would be more applicable to use e.g. probing or some other evaluation method

Nonetheless, this is a first step towards measuring the visual commonsense knowledge in multimodal as well as unimodal LMs. We hope that the evaluation tasks introduced here may aid other researchers in their aim to create models that learn language from more than text.

\section*{Acknowledgements}

We thank the anonymous reviewers for their valuable feedback and knowledge sharing.

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Additionally, the computations were enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC), partially funded by the Swedish Research Council through grant agreement no. 2018-05973.


\section{Introduction}
The Ising model was originated by Lenz in 1920. Later, it was
investigated by his student Ising in 1925 \cite{Ising1}. The model
is very important in the investigation of phase transition since
it unifies the study of phase transitions in systems as diverse as
gas--liquids, ferromagnets, binary alloys, and so
on\cite{Stanley1,Hu1,Zhu1,Keskin1,Aouini1}. The model has an exact
solution in one-dimension (1D) with the prediction of the absence
of phase transition at finite temperature
\cite{Huang1,Baxter1,book1,Beale1,McCOY1}. The model has been
studied using several theoretical approaches, such as high and
low-temperature series expansions, Monte Carlo simulations,
renormalization group methods, and perturbative field theoretical
methods
\cite{Hasenbusch1,Jasch1,Gupta1,Heringa1,Salman1,Campostrini1,Butera1}.
A more extensive list of methods and references can be found in
the exhaustive review article by Pelissetto and Vicari
\cite{Pelissetto1}. Almost seventy years have passed since
Onsager's \cite{Onsager1} celebrated solution of the Ising model
free energy in 1944, followed by Yang's \cite{Yang1} proof of
Onsager's result for the spontaneous magnetization in 1952, there
is, however, no known solution for the 3D model
\cite{Zhang1,Wu41,Perk51}. Therefore, Onsager and Yang's works of
the square lattice Ising model is still one of the very few
exactly solved models \cite{Yang831,Perk11,Perk21}, it now serves
as a proving ground for new theories, approximations, and
numerical algorithms. Before introducing the calculation of the
average magnetization expressions for the body-centered cubic
lattice, we think that it is relevant to mention shortly the
important achievements of the subject.

The proof of the existence of the critical point by Kramers and
Wannier \cite{Kramers1} with the method of a dual transformation
was an important development in the investigation of phase
transition. Later, the qualitative calculation of Kramers and
Wannier was proven analytically by Onsager and \cite{Newell1,
Kaufman1}. The spontaneous magnetization of the Ising model on
rectangular lattice was first calculated by Onsager, he has not,
however, published his derivation \cite{Keh1}. The average
magnetization expression of the square lattice Ising model,
$\langle\sigma\rangle=[1-\sinh(2K)^{-4}]^{\frac{1}{8}}$ was first
published by Yang \cite{Yang19521}. The method used in Yang's
derivation is too cumbersome, he recalls it as the longest
calculation in his career \cite{201}. Later, a less complicated
method, but is still too hard to recover, has been proposed to
study the 2D Ising model \cite{261,271,281,331,341, Syozi1}. It is
also important to mention that average magnetization relations
were obtained for the Ising model on honeycomb lattice
\cite{Naya1} by Naya and by \cite{Potts1} for the triangular
lattice. Now let us introduce the main steps that will be used in
the calculation of the average magnetization of the body-centered
cubic lattice.  For the sake of confirmation of the validity and
relevance of the method that will be used in this paper, we first
want to mention some important features of the odd spins
correlation function since they will be one important part of the
calculation of the average magnetization of body-centered cubic
Ising lattice in his paper. Unfortunately, there is no large
available literature except for some papers on three spin
correlation functions. The three spin correlation function of the
2D Ising model \cite{1141,Baxtery1} was first considered by Baxter
for three spins surrounding a triangle by using the Pfaffian
method. A simpler derivation of the three spin correlation of the
honeycomb lattice was also given by Enting \cite{Enting1}. We
think that it is important to mention that there were some other
important studies on the subject of the three spin correlation
functions \cite{1041,1051,1061,Tanaka1}. The common physical
properties of the three spin correlation function obtained in
these works is that: the three spin correlation function
manifestly assumes the same critical coupling strength and the
same critical exponent as the order parameter. As we will see in
the next section, these physical properties are relevant as well
as necessary to describe the three spin correlation function. We
can, therefore, use these physical properties to propose a
relevant mathematical functional form for the odd spin correlation
functions. These points will be elucidated more through this
paper. In the next section, we will try to obtain the desired
average magnetization expression for the body-centered cubic
lattice with the same analytic method used in our previous paper
\cite{kaya21,kaya31} and the previously derived average
magnetization interrelation by us \cite{kaya1}. The obtained
expression will be compared to the already available simulation
result \cite{Hakan1}. In the same section, we will discuss the
relevance of the obtained average magnetization relation and we
will also present some concluding remarks about the obtained
relation.

\section{The average magnetization calculation of body-centered cubic Ising lattice}

The previously derived formula \cite{kaya1} can be arranged in the absence of external magnetic field for the
body-centered cubic lattice as,
\begin{equation}
\langle\sigma_{0,i}\rangle=
\langle\tanh[ K(\sigma_{1,i}+\sigma_{2,i}+\cdots\sigma_{8,i} )]\rangle.
\end{equation}
Here $\sigma_{0,i}$ denotes the
central spin at the $i^{th}$ site while $\sigma_{l,i}$,
$l=1,2,\cdots,8$, are the nearest neighbor spins around its central
spin,and $K$ is the coupling strength. Apparently from now on, caring the index $i$ is not necessary.

Expressing the tangent hyperbolic function with the following equivalent mathematical form by the help of Fig. 1 as,
\begin{eqnarray*}
&&\tanh[ K(\sigma_{1}+\sigma_{2}+\cdots +\sigma_{8})]=A_{1}\{\sigma_{1}+\sigma_{2}+\cdots+\sigma_{8} \}+
{} \\ && A_{2}\{\sigma_{1}\sigma_{2}\sigma_{3}+\cdots\}+A_{3}\{\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}\sigma_{5}+\cdots\}
+A_{4}\{\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}\sigma_{5}\sigma_{6}\sigma_{7}+\cdots\}.
\end{eqnarray*}

It is worthwhile to mention that in the parentheses of this equation indicated as $\{\cdots\}$ coming after
$A_{1}$, $A_{2}$, $A_{3}$, and $A_{4}$, there are $8, 56, 56, 8$ terms respectively.
\begin{figure*}[!hbt]
\begin{center}$
\begin{array}{cc}
\scalebox{0.7}{\includegraphics{bc14.eps}}
\end{array}$
\end{center}
\caption{The body centered cubic lattice structure, the eight spins surrounding the central spin $\sigma_{0}.$}
\end{figure*}

Writing the relation $\tanh[ K(\sigma_{1}\!+\!\sigma_{2}\!+\cdots \!+\sigma_{8})]$ for different orientations of
the eight spin degrees of freedom, one can only obtain four linearly independent equation as
\begin{eqnarray*}
&&\tanh(8K)=8A_{1}+56A_{2}+56A_{3}+8A_{4},
{} \nonumber \\ && \tanh(6K)=6A_{1}+14A_{2}-14A_{3}-6A_{4},
{} \nonumber \\ && \tanh(4K)=4A_{1}-4A_{2}-4A_{3}+4A_{4},
{} \nonumber\\ && \tanh(2K)=2A_{1}-6A_{2}+6A_{3}-2A_{4}.
\end{eqnarray*}

The solution of this system of linear equations, after some algebra the solution for $A_{1}$, $A_{2}$, $A_{3}$,
and $A_{4}$, can be obtained readily,
\begin{eqnarray*}
&&A_{1}(K)=\frac{1}{128}[14\tanh(2K)+14\tanh(4K)+6\tanh(6K)+\tanh(8K)],
{} \nonumber \\ &&A_{2}(K)= \frac{1}{128}[-6\tanh(2K)-2\tanh(4K)+2\tanh(6K)+\tanh(8K)],
{} \nonumber \\ && A_{3}(K)=\frac{1}{128}[6\tanh(2K)-2\tanh(4K)-2\tanh(6K)+\tanh(8K)],
{} \nonumber\\ && A_{4}(K)= \frac{1}{128}[-14\tanh(2K)+14\tanh(4K)-6\tanh(6K)+\tanh(8K)],
\end{eqnarray*}
by these expressions.

Now, substituting the equivalent mathematical form of tangent hyperbolic function into Eq. (1) and followingly
taking the average of both sides, Eq. (1) turns out to be
\begin{eqnarray}
&&\langle\sigma\rangle=8A_{1}\langle\sigma\rangle+A_{2}[ 24\langle\sigma_{1}\sigma_{2}\sigma_{3}\rangle+
24\langle\sigma_{1}\sigma_{2}\sigma_{7}\rangle+ 8\langle\sigma_{1}\sigma_{3}\sigma_{6}\rangle]
{} \nonumber \\ && +A_{3}[ 24\langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}\sigma_{5}\rangle+
24\langle\sigma_{1}\sigma_{2}\sigma_{7}\sigma_{8}\sigma_{3}\rangle+8\langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{5}
\sigma_{7}\rangle]{} \nonumber \\ &&+8A_{4}\langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}\sigma_{5}
\sigma_{6}\sigma_{7}\rangle.
\end{eqnarray}

Now let us introduce some physical interpretation of the odd spins correlation functions appearing in this equation.
To this end, taking into account the critical behavior of the order parameter $\langle\sigma\rangle$ might be helpful.
Recalling the singularity of the order parameter at the critical point, one can easily see that the odd spins
correlation function can be expressed as a function of the order parameter. On the other hand, it is not easy or not
even possible to propose an exact relation for the odd spins correlations, but taking into account general behavior
of the critical properties of the order parameter, one can conjecture or propose heuristic functional forms for the
odd spins correlation functions with the following relations as,
\begin{eqnarray*}
&&\langle\sigma_{1}\sigma_{2}\sigma_{3}\rangle=a_{2,1}\langle\sigma\rangle+(1-a_{2,1})
\langle\sigma\rangle^{\frac{1+\beta}{\beta}},
{} \nonumber \\ &&\langle\sigma_{1}\sigma_{2}\sigma_{7}\rangle=a_{2,2}\langle\sigma\rangle+
(1-a_{2,2})\langle\sigma\rangle^{\frac{1+\beta}{\beta}},
{}\nonumber \\ && \langle\sigma_{1}\sigma_{3}\sigma_{6}\rangle=a_{2,3}\langle\sigma\rangle+
(1-a_{2,3})\langle\sigma\rangle^{\frac{1+\beta}{\beta}},
{} \nonumber \\ && \langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}\sigma_{5}\rangle=a_{3,1}\langle\sigma\rangle+
(1-a_{3,1})\langle\sigma\rangle^{\frac{1+\beta}{\beta}},
{} \nonumber \\ && \langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{7}\sigma_{8}\rangle=a_{3,2}\langle\sigma\rangle+
(1-a_{3,2})\langle\sigma\rangle^{\frac{1+\beta}{\beta}},
{} \nonumber \\ && \langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{5}\sigma_{7}\rangle=a_{3,3}\langle\sigma\rangle+
(1-a_{3,3})\langle\sigma\rangle^{\frac{1+\beta}{\beta}},
{} \nonumber \\ && \langle\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}\sigma_{5}\sigma_{6}\sigma_{7}\rangle=
a_{4,1}\langle\sigma\rangle+(1-a_{4,1})\langle\sigma\rangle^{\frac{1+\beta}{\beta}}.
\end{eqnarray*}
Substituting these conjectured odd spins correlation functions into Eq. (2), it leads to
\begin{figure*}[!hbt]
\begin{center}$
\begin{array}{cc}
\scalebox{1.2}{\includegraphics{BCC6.eps}}
\end{array}$
\end{center}
\caption{The plot of $A_{1}$, $A_{2}$, $A_{3}$, and $A_{4}$ with respect to $K$. }
\end{figure*}
\begin{eqnarray*}
&&\!\!\!\!\!\!\!\!\!\!\textstyle\langle\sigma\rangle=\Big\{ 8A_{1}\!+\!24A_{2}\Big[a_{2,1}\!+\!a_{2,2}\!+
\!\frac{a_{2,3}}{3}\Big]\!+\!24A_{3}\Big[a_{3,1}\!+\!a_{3,2}\!+\!\frac{a_{3,3}}{3}\Big]\!+\!8A_{4}a_{4,1}\Big\}
\langle\sigma\rangle {}\nonumber \\ &&\textstyle+\Big\{24A_{2}\Big[\frac{7}{3}-(a_{2,1}+a_{2,2}+\frac{a_{2,3}}{3})\Big]+
24A_{3}\Big[\frac{7}{3}-(a_{3,1}+a_{3,2}+\frac{a_{3,3}}{3})\Big] {}\nonumber \\ &&+8A_{4}(1-a_{4,1}) \Big\}
\langle\sigma\rangle^{\frac{1+\beta}{\beta}}.
\end{eqnarray*}
In the above equation, there are seven unknown parameters which are needed to be obtained.

To this end, if we define more proper parameters in terms of the coefficients as, $z_{1}=a_{2,1}+a_{2,2}+
\frac{a_{2,3}}{3}$ and $z_{2}=a_{3,1}+a_{3,2}+\frac{a_{3,3}}{3}$. And assuming intuitively that $z_1$ and
$z_2$ have approximately equal values, one can define $a_{4,1}$ in terms of $z_{1}$ and $z_{2}$ as $a_{4,1}=
\frac{3}{7}\frac{z_{1}+z_{2}}{2}$. This final approximation can be elucidated if we consider Fig. 2. From this
figure one can see that $A_{4}$ assumes values a lot less than all the others, $A_{1}$, $A_{2}$ and $A_{3}$.
Meaning that any rough approximate values of $a_{4,1}$ can only change the final result very slightly (or even
the effect of it is unnoticeable). Thus the number of unknown parameters are reduced to two unknown, $z_{1}$ and $z_{2}$.

To obtain these two unknowns, one can exploit the physical properties of the average magnetization. For example,
if $K$ goes to infinity, $\langle\sigma\rangle$ must goes to one. Unfortunately, this property of average magnetization
can not produce a relation between $z_{1}$ and $z_{2}$.

Using the critical behavior of the average magnetization, one can use the property of $\langle\sigma\rangle$ at
the critical point: it assumes the value of zero if $K<K_{c}$, and it assumes values different from zero if $K>K_{c}$.

Using this final remark and assuming $z_{1}\simeq z_{2}$, and
denoting both of them with $z $, one can obtain the value of z as
$z=0.8238$ if the critical value of $K$ for body-centered cubic
lattice $K_{c}=0.1573$ is considered. After this short analysis,
we obtain the following relation for the average magnetization of
body centered cubic Ising lattice as,
\begin{equation}
\langle\sigma\rangle=\Big[\frac{1-8A_{1}-24(A_{2}+A_{3}+\frac{1}{7}A_{4})}{24(A_{2}+A_{3})(\frac{7}{3}-z)+
8A_{4}(1-\frac{3}{7}z)}\Big]^{\beta},
\end{equation}
where the value of $\beta$ can be taken as equal to $0.325$ for all the 3D Ising lattice.

We have plotted Fig. 3 for the obtained average magnetization expression presented with Eq. (3). In this figure,
the red data points indicates the simulation results of \cite{Hakan1} and dashed curve shows the result of Eq. (3).
\begin{figure*}[!hbt]
\begin{center}$
\begin{array}{cc}
\scalebox{1.1}{\includegraphics{BCC9.eps}}
\end{array}$
\end{center}
\caption{The plot of the average magnetization of the body centered cubic Ising lattice with respect to $K$. }
\end{figure*}

As seen in Fig. (3), the agreement of the obtained expression for average magnetization of body-centered cubic
lattice in this paper and the simulation data is almost perfect. Even, it is almost impossible to see the differences
in this figure.

Therefore, we can claim that the heuristically obtained average magnetization relation is given by Eq. (3) is almost
exact. This means that the conjectures used here in this paper have been previously applied to the square, honeycomb,
triangular, and simple cubic lattices in references \cite{kaya21,kaya31} for the odd spins correlation functions are
relevant and also valuable.

\section{Conclusion and Discussion}
In this paper, the calculation of the average magnetization of the body-centered cubic Ising lattice was considered by
the method which was developed previously in a series of papers of us. Indeed, we have obtained an average magnetization
relation for this lattice structure, and we observed that the obtained expression is in good agreement with the
simulation data. Recalling, the almost exact agreement of the previously obtained result for honeycomb, square,
triangular, and simple cubic structure average magnetization relation and the average magnetization relation obtained
in this current paper for the body-centered cubic lattice.

We can claim that the conjectured mathematical forms for the odd spins correlation functions are relevant and also
important since they lead to an analytical calculation of the average magnetization of Ising lattices. Of course,
the method is inevitably heuristic. However, we think that any relevant analytical calculation method developed for
the calculation of the average magnetization expressions of Ising lattices can be considered valuable and helpful.

In other words, in this paper, we obtain the average magnetization relation of the body-centered cubic Ising lattice
with very simple and tractable mathematical and physical approaches. We think that this point is also important if
we recall the mathematical procedure applied to obtain the square lattice average magnetization by Yang.


\section{Analysis}
\textbf{CL-Reg and Cor-Reg render the feature space more isotropic.}
\input{tables/table_analysis_isotropyChange_English}
As shown in Table~\ref{table_analysis_isotropy_change_English}, both regularizers manage to make the feature space more isotropic, compared to IntentBERT-ReImp where only supervised pre-training is employed. In addition, Cor-Reg achieves similar or better isotropy compared to CL-Reg, which is consistent with Cor-Reg's relative superiority in performance. Interestingly, when the two regularizers are adopted simultaneously, the isotropy is not further improved, but a better performance is observed.

As for cross-lingual scenario, we summarize the results in Table~\ref{table_analysis_isotropy_change_CrossLingual}. Similar to experiments in English, both regularizers render the semantic space more isotropic. However, CL-Reg is less effective than Cor-Reg, for example, in Thai and Chinese, although they achieve similar performance. This observation indicates that the effectiveness of CL-Reg may not be ascribed to the improved isotropy for some target languages, which partially explains its complementarity with Cor-Reg. As shown in the table, when they are employed simultaneously, both the isotropy and the performance are improved.

\textbf{Moderate isotropy is helpful.}
To investigate the relation of isotropy and performance, we tune the weight on Cor-Reg to yield different degrees of isotropy and examine the corresponding performance. As shown in Fig.~\ref{figure: acc_iso_curve}, a common pattern is observed for most datasets -- the best performance is achieved only when the isotropy is moderate.
This observation indicates an appropriate trade-off between supervised pre-training and the isotropization that is achieved by the proposed method.
\input{figures/figure_acc_iso_curve}

\subsection{Correlation matrix is advantageous over covariance matrix}
In the design of Cor-Reg, correlation matrix, rather than covariance is employed to characterize isotropy, although the latter contains more information -- variance. We argue that the design choice is advantageous. We conduct experiments using covariance matrix. Particularly, the non-diagonal elements are pushed towards $0$, while the diagonal elements towards $1$, $0.5$ and the mean value, respectively, which are denoted by Cov-Reg-1, Cov-Reg-0.5 and Cov-Reg-mean in Table~\ref{table: analysis, coefficient matrix, or covariance matrix}. As shown in Table~\ref{table: analysis, coefficient matrix, or covariance matrix}, all these configurations under-perform Cor-Reg.
\input{tables/table_analysis_correlation_covariance}

\textbf{The proposed methods are complementary with batch normalization.} Batch normalization~\cite{ICML-2015-IoffeS} is an off-the-shelf effective method to mitigate the anisotropy via normalizing each dimension into a scalar with zero mean and unit variance.
As shown in Table~\ref{table: analysis, batch normalization}, our methods outperform batch normalization, and combining them yields even better performance.
\input{tables/table_analysis_batchNorm}
\section{Experiments}
To validate the effectiveness of the approach, we conduct extensive experiments.

\subsection{Experimental Setup}
\label{subsection:cross domain}
\textbf{Datasets.} To perform supervised pre-training, we follow ~\citeauthor{zhang-etal-2021-effectiveness-pre} to use the  OOS dataset~\cite{larson2019evaluation} which contains diverse semantics of $10$ domains.
Also following ~\citeauthor{zhang-etal-2021-effectiveness-pre}, we exclude the domains ``Banking'' and ``Credit Cards''
since they are similar in semantics to one of the test dataset BANKING77. We then use $6$ domains for training and $2$ for validation, as shown in Table~\ref{table: domain split of OOS}. For evaluation, we employ three datasets:
\textbf{BANKING77}~\cite{casanueva2020efficient} is an intent detection dataset for banking service.
\textbf{HINT3}~\cite{arora2020hint3} covers $3$ domains, ``Mattress Products Retail'', ``Fitness Supplements Retail'', and ``Online Gaming''. \textbf{HWU64}~\cite{DBLP:conf/iwsds/LiuESR19} is a large-scale dataset containing $21$ domains. Dataset statistics are summarized in Table~\ref{table: Dataset statistics}.
\input{tables/table_OOS_domainSplit}
\input{tables/table_dataset_statistics}

\textbf{Our Method.}
Our method
can be applied to fine-tune any PLM. We conduct experiments on two popular PLMs, BERT~\cite{devlin2018bert} and RoBERTa~\cite{liu2019roberta}. For both of them, the embedding of $[CLS]$ is used as the utterance representation in Eq.~\ref{equation: linear layer classifier}. We employ logistic regression as the classifier. We select the hyperparameters $\lambda, \lambda_1, \lambda_2$, and $\tau$ by validation. The best hyperparameters are provided in Table~\ref{table: best hyperparameters}.
\input{tables/table_hyperParams_best}

\input{tables/table_mainResult_BERT}
\input{tables/table_mainResult_RoBERTa}

\textbf{Baselines.} We compare our method to the following baselines. First, for BERT-based methods, \textbf{BERT-Freeze} freezes BERT; \textbf{CONVBERT}~\cite{mehri2020dialoglue}, \textbf{TOD-BERT}~\cite{wu2020tod}, and \textbf{DNNC-BERT}~\cite{zhang-etal-2020-discriminative} further pre-train BERT on conversational corpus or natural language inference tasks.
\textbf{USE-ConveRT}~\cite{henderson-etal-2020-convert, casanueva2020efficient} is a transformer-based dual-encoder pre-trained on conversational corpus.
\textbf{CPFT-BERT} is the re-implemented version of CPFT~\cite{zhang-etal-2021-shot}, by further pre-training BERT in an unsupervised manner with mask-based contrastive learning and masked language modeling on the same training data as ours.
\textbf{IntentBERT}~\cite{zhang-etal-2021-effectiveness-pre} further pre-trains BERT via supervised pre-training described in Section~\ref{section_methodoloy_supervised_pre-training}.
To guarantee a fair comparison, we provide \textbf{IntentBERT-ReImp}, the re-implemented version of IntentBERT, which uses the same random seed, training data, and validation data as our methods. Second, for RoBERTa-based baselines, \textbf{RoBERTa-Freeze} freezes the model. \textbf{WikiHowRoBERTa}~\cite{zhang2020intent} further pre-trains RoBERTa on synthesized intent detection data.
\textbf{DNNC-RoBERTa} and \textbf{CPFT-RoBERTa} are similar to DNNC-BERT and CPFT-BERT except the PLM.
\textbf{IntentRoBERTa} is the re-implemented version of IntentBERT based on RoBERTa, with uses the same random seed, training data, and validation data as our method.
Finally, to show the superiority of the joint fine-tuning and isotropization, we compare our method against whitening transformation~\cite{su2021whitening}. \textbf{BERT-White} and \textbf{RoBERTa-White} apply the transformation to BERT and RoBERTa, respectively. \textbf{IntentBERT-White} and \textbf{IntentRoBERTa-White} apply the transformation to IntentBERT-ReImp and IntentRoBERTa, respectively.

All baselines use logistic regression as classifier except DNNC-BERT and DNNC-RoBERTa, wherein we follow the original work\footnote{https://github.com/salesforce/DNNC-few-shot-intent} to
train a pairwise encoder for nearest neighbor classification.

\textbf{Training Details.} We use PyTorch library and Python
to build our model.
We employ Hugging Face implementation\footnote{https://github.com/huggingface/transformers} of \emph{bert-base-uncased} and \emph{roberta-base}. We use Adam~\cite{kingma2014adam} as the optimizer with learning rate of $2e-05$ and weight decay of $1e-03$. The model is trained with Nvidia RTX 3090 GPUs. The training is early stopped if no improvement in validation accuracy is observed for $100$ steps. The same set of random seeds, $\left\{1, 2, 3, 4, 5\right\}$, is used for IntentBERT-ReImp, IntentRoBERTa, and our method.

\textbf{Evaluation.}
The baselines and our method are evaluated on $C$-way $K$-shot tasks. For each task, we randomly sample $C$ classes and $K$ examples per class. The $C\times K$ labeled examples are used to train the logistic regression classifier. Note that we do not further fine-tune the PLM using the labeled data of the task.
We then sample another $5$ examples per class as queries. Fig.~\ref{figure: idea_illustration} gives an example with $C=2$ and $K=1$. We report the averaged accuracy of $500$ tasks randomly sampled from $\mathcal{D}_{\text{target}}$.

\subsection{Main Results}
\label{main_results}
The main results are provided in Table~\ref{table: main_result_effectiveness_of_our_methods_BERT}~(BERT-based) and Table~\ref{table: main_result_effectiveness_of_our_methods_RoBERTa}~(RoBERTa-based). The following observations can be made. First,
our proposed regularized supervised pre-training, with either CL-Reg or Cor-Reg, consistently outperforms all the baselines by a notable margin in most cases, indicating the effectiveness of our method.
Our method also outperforms whitening transformation, demonstrating the superiority of the proposed joint fine-tuning and isotropization framework. Second, Cor-Reg slightly outperforms CL-Reg in most cases, showing the advantage of
enforcing isotropy explicitly with the correlation matrix. Finally, CL-Reg and Cor-Reg show a complementary effect in many cases, especially on BANKING77. The above observations are consistent for both BERT and RoBERTa. It can be also seen that higher performance is often attained with RoBERTa.

\input{tables/table_analysis_isotropyChange_English}
\label{section: analysis, better isotropy}
The observed improvement in performance comes with an improvement in isotropy.
We report the change in isotropy by the proposed regularizers in Table~\ref{table_analysis_isotropy_change_English}. It can be seen that both regularizers and their combination make the feature space more isotropic compared to  IntentBERT-ReImp that  only uses supervised pre-training. In addition, in general, Cor-Reg can achieve better isotropy than CL-Reg.

\subsection{Ablation Study and Analysis}
\label{section: analysis}
\textbf{Moderate isotropy is helpful.}
To investigate the relation between the isotropy of the feature space and the performance of few-shot intent detection, we tune the weight parameter $\lambda$ of Cor-Reg to
increase the isotropy and examine the performance. As shown in Fig.~\ref{figure: acc_iso_curve}, a common pattern is observed: the best performance is achieved when the isotropy is moderate.
This observation indicates that it is important to find an appropriate trade-off between learning intent detection skills and learning an insotropic feature space. In our method, we select the appropriate $\lambda$ by validation.
\input{figures/figure_acc_iso_curve}

\textbf{Correlation matrix is better than covariance matrix as regularizer.}
\label{section: analysis, compared to covariance}
In the design of Cor-Reg (Section~\ref{section-method-cor-reg}), we use the correlation matrix, rather than the covariance matrix, to characterize isotropy, although the latter contains more information -- variance. The reason is that it is difficult to determine the proper scale of the variances.
Here, we conduct experiments using the covariance matrix, by pushing the non-diagonal elements (covariances) towards $0$ and the diagonal elements (variances) towards $1$, $0.5$, or the mean value, which are denoted by Cov-Reg-1, Cov-Reg-0.5, and Cov-Reg-mean respectively in Table~\ref{table: analysis, coefficient matrix, or covariance matrix}. It can be seen that all the variants perform worse than Cor-Reg.
\input{tables/table_analysis_correlation_covariance}

\textbf{Our method is complementary with batch normalization.} Batch normalization~\cite{ICML-2015-IoffeS} can potentially mitigate the anisotropy problem via normalizing each dimension
with unit variance.
We find that combining our method with batch normalization yields better performance, as shown in Table~\ref{table: analysis, batch normalization}.
\input{tables/table_analysis_batchNorm}

\textbf{The performance gain is not from the reduction in model variance.} Regularization techniques such as L1 regularization~\cite{tibshirani1996regression} and L2 regularization~\cite{hoerl1970ridge} are often used to improve model performance by reducing model variance. Here, we show that the performance gain of our method is ascribed to the improved isotropy~(Table~\ref{table_analysis_isotropy_change_English}) rather than the reduction in model variance. To this end, we compare our method against L2 regularization with a wide range of weights, and it is observed that reducing model variance cannot achieve comparable performance to our method, as shown in Fig.~\ref{figure: analysis_l2reg}.
\input{figures/figure_analysis_L2Reg}

\input{figures/figure_computational_overhead}
\textbf{The computational overhead is small.} To analyze the computational overheads incurred by CL-Reg and Cor-Reg, we decompose the duration of one epoch of our method using the two regularizers jointly. As shown in Fig.~\ref{figure:computational overhead}, the overheads of CL-Reg and Cor-Reg are small,
only taking up a small portion of the time.

\section{Conclusion}
\label{sec:conclusion}
In this work, we have identified and analyzed the anisotropy of the feature space of a PLM fine-tuned on intent detection tasks. Further, we have proposed a joint training framework and designed two regularizers based on contrastive learning and correlation matrix respectively to increase the insotropy of the feature space during fine-tuning, which leads to notably improved performance on few-shot intent detection. Our findings and solutions may have broader implications for solving other natural language understanding tasks with PLM-based models.

\section*{Acknowledgments}
We would like to thank the anonymous reviewers for their valuable comments. This research was supported by the grants of HK ITF UIM/377 and PolyU DaSAIL project P0030935 funded by RGC.
\section{Introduction}
Intent detection is a core module of task-oriented dialogue systems. Training a well-performing intent classifier with only a few annotations, i.e., few-shot intent detection, is of great practical value. Recently, this problem has attracted considerable attention~\cite{vulic2021convfit, zhang-etal-2021-shot, dopierre2020few} but remains a challenge.

To tackle few-shot intent detection, earlier works employ induction network~\cite{geng2019few}, generation-based methods~\cite{xia2020composed}, metric learning~\cite{nguyen2020dynamic}, and self-training~\cite{dopierre2020few}, to design sophisticated algorithms.
Recently, pre-trained language models (PLMs) have emerged as a simple yet promising solution to a wide spectrum of natural language processing (NLP) tasks, triggering the surge of PLM-based solutions for few-shot intent detection~\cite{wu2020tod, zhang-etal-2021-effectiveness-pre, zhang-etal-2021-shot, vulic2021convfit, zhang-etal-2021-shot}, which typically fine-tune PLMs on conversation data.

A PLM-based fine-tuning method~\cite{zhang-etal-2021-effectiveness-pre}, called IntentBERT, utilizes a small amount of labeled utterances from public intent datasets to fine-tune PLMs with a standard classification task, which is referred to as \emph{supervised pre-training}. Despite its simplicity, supervised pre-training has been shown extremely useful for few-shot intent detection even when the target data and the data used for fine-tuning are very different in semantics. However, as will be shown in Section~\ref{section: pilot experiments, fine-tuning -> anisotropy}, IntentBERT suffers from severe anisotropy, an undesirable property of PLMs~\cite{gao2019representation, ethayarajh2019contextual, li2020sentence}.

\input{figures/figure_idea}

Anisotropy is a geometric property that semantic vectors fall into a narrow cone. It has been identified as
a crucial factor for the sub-optimal performance of PLMs on a variety of downstream tasks~\cite{gao2019representation, arora-etal-2016-latent, cai2020isotropy, ethayarajh2019contextual}, which is also known as the representation degeneration problem~\cite{gao2019representation}. Fortunately, isotropization techniques can be applied to adjust the embedding space and yield significant performance improvement in many tasks~\cite{su2021whitening, rajaee2021cluster}.

Hence, this paper aims to answer the question:
\begin{itemize}
    \item Can we improve supervised pre-training via \emph{isotropization} for few-shot intent detection?
\end{itemize}
Many isotropization techniques have been developed based on transformation~\cite{su2021whitening, huang-etal-2021-whiteningbert-easy}, contrastive learning~\cite{gao2021simcse}, and top principal components elimination~\cite{DBLP:conf/iclr/MuV18}. However, these methods are designed for off-the-shelf PLMs. When applied on PLMs that have been fine-tuned on some NLP task such as semantic textual similarity or intent classification, they may introduce an adverse effect, as observed in \citet{rajaee2021isotropy} and our pilot experiments.

In this work, we propose to regularize supervised pre-training with isotropic regularizers.
As shown in Fig.~\ref{figure: idea_illustration}, we devise two regularizers, a contrastive-learning-based regularizer (CL-Reg) and a correlation-matrix-based regularizer (Cor-Reg), each of which can increase the isotropy of the feature space  during supervised training. Our empirical study shows that the regularizers can significantly improve the performance of standard supervised training,
and better performance can often be achieved when they are combined.

The contributions of this work are three-fold:
\begin{itemize}
    \item We present the first study on the isotropy property of PLMs for few-shot intent detection, shedding light on the interaction of supervised pre-training and isotropization.
    \item We improve supervised pre-training by devising two simple yet effective regularizers to
    increase the isotropy of the feature space.
    \item We conduct a  comprehensive evaluation and analysis to validate the effectiveness of the proposed approach.
\end{itemize}

\section{Method}
\input{figures/figure_method_2regularizers}
\label{section_methodoloy}
The pilot experiments reveal the anisotropy of a PLM fine-tuned on intent detection tasks and the challenge of applying isotropization techiniques on the fine-tuned model. In this section, we propose a joint fine-tuning and isotropization framework. Specifically, we propose two regularizers to make the feature space more isotropic during fine-tuning. Before presenting our method, we first introduce supervised pre-training.

\subsection{Supervised Pre-training for Few-shot Intent Detection}
\label{section_methodoloy_supervised_pre-training}
Few-shot intent detection targets to train a good intent classifier with only a few labeled data $\mathcal{D}_{\text{target}}=\{(x_i,y_i)\}_{N_t}$, where $N_t$ is the number of labeled samples in the target dataset, $x_i$ denotes the $i_\text{th}$ utterance, and $y_i$ is the label.

To tackle the problem, \citet{zhang-etal-2021-effectiveness-pre} propose to learn intent detection skills (fine-tune a PLM) on a small subset of public intent detection benchmarks by supervised pre-training. Denote by $\mathcal{D}_{\text{source}}=\{(x_i,y_i)\}_{N_s}$ the source data used for pre-training, where $N_s$ is the number of examples. The fine-tuned PLM can be directly used on the target dataset. It has been shown that this method can work well  when
the label spaces of $\mathcal{D}_{\text{source}}$ and $\mathcal{D}_{\text{target}}$ are disjoint.

Specifically, the pre-training is conducted by attaching a linear layer (as the classifier) on top of the utterance representation generated by the PLM:
\begin{equation}
    \begin{split}
        \text{p}(y|\mathbf{h}_i) = \text{softmax}\left(\mathbf{W}\mathbf{h}_i + \mathbf{b}\right) \in \mathbb{R}^L,
    \end{split}
    \label{equation: linear layer classifier}
\end{equation}
where $\mathbf{h}_i \in \mathbb{R}^{d}$ is the representation of the $i_\text{th}$ utterance in $\mathcal{D}_{\text{source}}$, $\mathbf{W} \in \mathbb{R}^{L \times d}$ and $\mathbf{b} \in \mathbb{R}^{L}$ are the parameters of the linear layer, and $L$ is the number of classes. The model parameters $\theta=\left\{\phi, \mathbf{W}, \mathbf{b}\right\}$, with $\phi$ being the parameters of the PLM, are trained on $\mathcal{D}_{\text{source}}$ with a cross-entropy loss:
\begin{equation}

        \theta = \argmin_{\theta} \mathcal{L}_{\text{ce}}\left(\mathcal{D}_{\text{source}};\theta\right).

    \label{equation: transfer learning task}
\end{equation}
After supervised pre-training, the linear layer is removed, and the PLM can be immediately used as a feature extractor for few-shot intent classification on target data. As shown in \citet{zhang-etal-2021-effectiveness-pre}, a parametric classifier such as logistic regression can be trained with only a few labeled samples to achieve good performance.

However, our analysis in Section~\ref{section: pilot experiments, fine-tuning -> anisotropy} shows the limitation of supervised pre-training, which yields a anisotropic feature space.

\subsection{Regularizing Supervised Pre-training with Isotropization}
To mitigate the anisotropy of the PLM fine-tuned by supervised pre-training, we propose a joint training objective by adding a regularization term  $\mathcal{L}_{\text{reg}}$ for isotropization:
\begin{equation}
    \mathcal{L}=\mathcal{L}_{\text{ce}}(\mathcal{D}_{\text{source}};\theta)+\lambda \mathcal{L}_{\text{reg}}(\mathcal{D}_{\text{source}};\theta),
    \label{equation: loss, general isotropy regularizer}
\end{equation}
where $\lambda$ is a weight parameter. The aim is to learn intent detection skills while maintaining an appropriate degree of isotropy. We devise two different regularizers introduced as follows.

\textbf{Contrastive-learning-based Regularizer.} Inspired by the recent success of contrastive learning in mitigating anisotropy~\cite{yan-etal-2021-consert, gao2021simcse}, we employ the dropout-based contrastive learning loss used in \citet{gao2021simcse} as the regularizer:
\begin{equation}
    \mathcal{L}_{\text{reg}}=-\frac{1}{N_b}\sum_i^{N_b}\text{log}\frac{e^{\text{sim}(\mathbf{h}_i, \mathbf{h}_{i}^{+})/\tau}}{\sum_{j=1}^{N_b}e^{\text{sim}(\mathbf{h}_i, \mathbf{h}_{j}^{+})/\tau}}.
    \label{equation: regularizer contrastive}
\end{equation}
In particular, $\mathbf{h}_i \in \mathbb{R}^{d}$ and $\mathbf{h}_i^+ \in \mathbb{R}^{d}$ are two different representations of utterance $x_i$ generated by the PLM with built-in standard dropout~\cite{JMLR:v15:srivastava14a}, i.e., $x_i$ is passed to the PLM twice with different dropout masks to produce $\mathbf{h}_i$ and $\mathbf{h}_i^+$. $\text{sim}(\mathbf{h}_1, \mathbf{h}_2)$ denotes the cosine similarity between $\mathbf{h}_1$ and $\mathbf{h}_2$. $\tau$ is the temperature parameter. $N_b$ is the batch size.
Since $\mathbf{h}_i$ and $\mathbf{h}_i^+$ represent the same utterance, they form a positive pair. Similarly, $\mathbf{h}_i$ and $\mathbf{h}_j^+$ form a negative pair, since they represent  different utterances. An example is given in Fig.~\ref{subFigure: figure: method_2reg_CLReg}. By minimizing the contrastive loss, positive pairs are pulled together while negative pairs are pushed away, which in theory enforces an isotropic feature space~\cite{gao2021simcse}.
In \citet{gao2021simcse}, the contrastive loss is used as the single objective to fine-tune off-the-shelf PLMs in an unsupervised manner, while in this work we use it jointly with supervised pre-training to fine-tune PLMs for few-shot learning.

\label{section-method-cor-reg}
\textbf{Correlation-matrix-based Regularizer.} The above regularizer enforces isotropization implicitly. Here, we propose a new regularizer that explicitly enforces isotropization.
The perfect isotropy is characterized by zero covariance and uniform variance~\cite{su2021whitening, Zhou2021IsoBNFB}, i.e., a covariance matrix with uniform diagonal elements and zero non-diagonal elements. Isotropization can be achieved by endowing the feature space with such statistical property. However, as will be shown in Section~\ref{section: analysis, compared to covariance}, it is difficult to determine the  appropriate scale of variance. Therefore, we base the regularizer on \emph{correlation matrix}
:
\begin{equation}
    \mathcal{L}_{\text{reg}}=\lVert\mathbf{\Sigma} - \mathbf{I}\rVert,
    \label{equation: regularizer correlation}
\end{equation}
where $\lVert\cdot\rVert$ denotes Frobenius norm, $\mathbf{I} \in \mathbb{R}^{d \times d}$ is the identity matrix, $\mathbf{\Sigma} \in \mathbb{R}^{d \times d}$ is the correlation matrix with $\mathbf{\Sigma}_{ij}$ being the Pearson correlation coefficient between the $i_\text{th}$ dimension and the $j_\text{th}$ dimension. As shown in Fig.~\ref{subFigure: figure: method_2reg_corReg}, $\mathbf{\Sigma}$ is estimated with utterances in the current batch. By pushing the correlation matrix towards the identity matrix during training, we can learn a more isotropic feature space.

Moreover, the proposed two regularizers can be used together as follows:
\begin{equation}
    \begin{split}
        \mathcal{L}=\mathcal{L}_{\text{ce}}(\mathcal{D}_{\text{source}};\theta)+\lambda_1 \mathcal{L}_{\text{cl}}(\mathcal{D}_{\text{source}};\theta) \\ +\lambda_2 \mathcal{L}_{\text{cor}}(\mathcal{D}_{\text{source}};\theta),
    \end{split}
    \label{equation: loss, general isotropy regularizer}
\end{equation}
where $\lambda_1$ and $\lambda_2$ are the weight parameters, and $\mathcal{L}_{\text{cl}}$ and $\mathcal{L}_{\text{cor}}$ denote CL-Reg and Cor-Reg, respectively. Our experiments show that better performance is often observed when they are used together.
\section{Pilot  Study}
\label{section: pilot experiments}
Before introducing our approach, we present pilot experiments to gain some insights into the interaction between isotropization and fine-tuning PLMs.

\subsection{Measuring isotropy}
Following \citet{DBLP:conf/iclr/MuV18, bis2021too}, we adopt the following measurement of isotropy:
\begin{equation}
    \begin{split}
        \text{I}(\mathbf{V}) = \frac{\min_{\mathbf{c}~\in~C}\text{Z}(\mathbf{c}, \mathbf{V})}{\max_{\mathbf{c}~\in~C}\text{Z}(\mathbf{c}, \mathbf{V})},
    \end{split}
    \label{equation: transfer learning task}
\end{equation}
where $\mathbf{V} \in \mathbb{R}^{N \times d} $ is the matrix of stacked embeddings of $N$ utterances (note that the embeddings have zero mean), $C$ is the set of unit eigenvectors of $\mathbf{V}^\top \mathbf{V}$, and $\text{Z}(\mathbf{c}, \mathbf{V})$ is the partition function~\cite{arora-etal-2016-latent} defined as:
\begin{equation}
    \begin{split}
        \text{Z}(\mathbf{c}, \mathbf{V}) = \sum_{i=1}^{N}{\exp{\left({\mathbf{c}^\top\mathbf{v}_i}\right)}},
    \end{split}
    \label{equation: transfer learning task}
\end{equation}
where $\mathbf{v}_i$ is the $i_{\text{th}}$ row of $\mathbf{V}$.
$\text{I}(\mathbf{V})\in\left[0,1\right]$, and 1 indicates perfect isotropy.

\input{tables/table_pilot_fineTune_isotropy}

\subsection{Fine-tuning Leads to Anisotropy}
\label{section: pilot experiments, fine-tuning -> anisotropy}
To observe the impact of fine-tuning on isotropy, we follow IntentBERT~\cite{zhang-etal-2021-effectiveness-pre} to fine-tune BERT~\cite{devlin2018bert} with standard supervised training on a small set of an intent detection benchmark OOS~\cite{larson2019evaluation} (details are given in Section~\ref{section_methodoloy_supervised_pre-training}). We then compare the isotropy of the original embedding space (BERT) and the embedding space after fine-tuning (IntentBERT) on target datasets. As shown in Table~\ref{table: pilot, fine-tune affects isotropy.}, after fine-tuning, the isotropy of the embedding space is notably decreased on all datasets.
Hence, it can be seen that \emph{fine-tuning may render the feature space more anisotropic}.

\input{figures/figure_pilot_contrastiveLearning_banking77}
\subsection{Isotropization after Fine-tuning May Have an Adverse Effect }
To examine the effect of isotropization on a fine-tuned model, we apply two strong isotropization techniques to IntentBERT: dropout-based contrastive learning~\cite{gao2021simcse} and whitening transformation~\cite{su2021whitening}. The former fine-tunes PLMs in a contrastive learning manner\footnote{We refer the reader to the original paper for details.}, while the latter transforms the semantic feature space into an isotropic space via matrix transformation.
These methods have been demonstrated highly effective~\cite{gao2021simcse, su2021whitening}
when applied to off-the-shelf PLMs, but things are different when they are applied to fine-tuned models.
As shown in Fig.~\ref{figure: twoStage_contrastiveLearning_hwu64}, contrastive learning improves isotropy, but it significantly lowers the performance on two benchmarks.
As for whitening transformation, it has inconsistent effects on the two datasets,
as shown in Fig.~\ref{figure: twoStage_whitening_hwu64_banking77}. It hurts the performance on HWU64~(Fig.~\ref{figure: twoStage_whitening_hwu64_banking7: HWU64}) but yields better results on BANKING77~(Fig.~\ref{figure: twoStage_whitening_hwu64_banking7: bank77}), while producing nearly perfect isotropy on both.
The above observations indicate that \emph{isotropization may hurt fine-tuned models}, which echoes the recent finding of~\citeauthor{rajaee2021does}.

\input{figures/figure_pilot_whitening_HWU64_banking77}
\section{Related Works}

\subsection{Few-shot Intent Detection}
With a surge of interest in few-shot learning~\cite{finn2017model, vinyals2016matching, snell2017prototypical}, few-shot intent detection has started to receive attention. Earlier works mainly focus on model design, using capsule network~\cite{geng2019few}, variational autoencoder~\cite{xia2020composed}, or metric functions~\cite{NAACL18, nguyen2020dynamic}. Recently, PLMs-based methods have shown promising performance in a variety of NLP tasks and become the model of choice for few-shot intent detection.
\citet{zhang-etal-2020-discriminative} cast few-shot intent detection into a natural language inference~(NLI) problem and fine-tune PLMs on NLI datasets. \citet{zhang-etal-2021-shot} propose to fine-tune PLMs on unlabeled utterances by contrastive learning. \citet{zhang-etal-2021-effectiveness-pre} leverage a small set of public annotated intent detection benchmarks to fine-tune PLMs with standard supervised training and observe promising performance on cross-domain few-shot intent detection. Meanwhile, the study of few-shot intent detection has been extended to other settings including semi-supervised learning~\cite{dopierre2020few, dopierre-etal-2021-protaugment}, generalized setting~\cite{nguyen2020dynamic}, multi-label classification~\cite{hou2020few}, and incremental learning~\cite{xia-etal-2021-incremental}. In this work, we consider standard few-shot intent detection, following the setup of \citet{zhang-etal-2021-effectiveness-pre} and aiming to improve supervised pre-training with isotropization.

\subsection{Further  Pre-training PLMs with Dialogue Corpora}
Recent works have shown that further pre-training off-the-shelf PLMs using dialogue corpora~\cite{henderson-etal-2019-training,peng2020few,peng2020soloist} are beneficial for task-oriented downstream tasks such as intent detection.
Specifically, TOD-BERT \cite{wu2020tod} conducts self-supervised learning on diverse task-oriented dialogue corpora. ConvBERT \cite{mehri2020dialoglue} is pre-trained on a $700$ million open-domain dialogue corpus.
\citet{vulic2021convfit} propose a two-stage procedure: adaptive
conversational fine-tuning followed by task-tailored conversational fine-tuning.
In this work, we follow \citet{zhang-etal-2021-effectiveness-pre} to further pre-train PLMs using a small amount of labeled utterances from public intent detection benchmarks.

\subsection{Anisotropy of PLMs}
Isotropy is a key geometric property of the semantic space of PLMs. Recent studies identify the anisotropy problem of PLMs~\cite{cai2020isotropy,  ethayarajh2019contextual, DBLP:conf/iclr/MuV18, rajaee2021isotropy}, which is also known as the representation degeneration problem~\cite{gao2019representation}: word embeddings occupy a narrow cone, which suppresses the expressiveness of PLMs. To resolve the problem, various methods have been proposed, including spectrum control~\cite{wang2019improving}, flow-based mapping~\cite{li2020sentence}, whitening transformation~\cite{su2021whitening, huang-etal-2021-whiteningbert-easy}, contrastive learning~\cite{gao2021simcse}, and cluster-based methods~\cite{rajaee2021cluster}.
Despite their effectiveness, these methods are designed for off-the-shelf PLMs.
The interaction between isotropization and fine-tuning PLMs remains under-explored. A most recent work by \citeauthor{rajaee2021does} shows that there might be a conflict between the two operations for the semantic textual similarity~(STS) task.
On the other hand, \citet{Zhou2021IsoBNFB} propose to fine-tune PLMs with isotropic batch normalization on some supervised tasks, but it requires a large amount of training data. In this work, we study the interaction between isotropization and supervised pre-training (fine-tuning) PLMs on intent detection tasks.

\section{Appendix}
\label{sec:appendix}
\subsection{Additional Results of Pilot Study}
To show the impact of fine-tuning on isotropy, we visualize the covariance matrix of the feature space in Fig.~\ref{figure_fineTune_affect_covarianceMatrix}. It is found that after fine-tuning,  the covariance is strengthened in general. Pefect isotropy requires uniform variance and zero covariance, and thus the changes in the covariance matrix agrees with the isotropy measurement presented in Table~\ref{table: pilot, fine-tune affects isotropy.}.
\input{figures/figure_fine-tune_affect_covarianceMatrix}