You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Text Classification based on Word Subspace with Term-Frequency

Text classification has become indispensable due to the rapid increase of text in digital form. Over the past three decades, efforts have been made to approach this task using various learning algorithms and statistical models based on bag-of-words (BOW) features. Despite its simple implementation, BOW features lack semantic meaning representation. To solve this problem, neural networks started to be employed to learn word vectors, such as the word2vec. Word2vec embeds word semantic structure into vectors, where the angle between vectors indicates the meaningful similarity between words. To measure the similarity between texts, we propose the novel concept of word subspace, which can represent the intrinsic variability of features in a set of word vectors. Through this concept, it is possible to model text from word vectors while holding semantic information. To incorporate the word frequency directly in the subspace model, we further extend the word subspace to the term-frequency (TF) weighted word subspace. Based on these new concepts, text classification can be performed under the mutual subspace method (MSM) framework. The validity of our modeling is shown through experiments on the Reuters text database, comparing the results to various state-of-art algorithms.

Text Representation with bag-of-words
The bag-of-words representation comes from the hypothesis that frequencies of words in a document can indicate the relevance of the document to a query BIBREF0 , that is, if documents and a query have similar frequencies for the same words, they might have a similar meaning. This representation is based on the vector space model (VSM), that was developed for the SMART information retrieval system BIBREF10 . In the VSM, the main idea is that documents in a collection can be represented as a vector in a vector space, where vectors close to each other represent semantically similar documents.
More formally, a document $d$ can be represented by a vector in $\mathbb {R}^{n}$ , where each dimension represents a different term. A term can be a single word, constituting the conventional bag-of-words, or combinations of $N$ words, constituting the bag-of-N-grams. If a term occurs in the document, its position in the vector will have a non-zero value, also known as term weight. Two documents in the VSM can be compared to each other by taking the cosine distance between them BIBREF1 .
There are several ways to compute the term weights. Among them, we can highlight some: Binary weights, term-frequency (TF) weights, and term-frequency inverse document-frequency (TF-IDF) weights.
Consider a corpus with documents $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ and a vocabulary with all terms in the corpus $V = \lbrace w_i\rbrace _{i=1}^{|V|}$ . The term weights can be defined as:
Binary weight: If a term occurs in the document, its weight is 1. Otherwise, it is zero.
Term-frequency weight (TF): The weight of a term $w$ is defined by the number of times it occurs in the document $d$ . 
$$TF(w,d) = n_d^w$$   (Eq. 8) 
Inverse document-frequency: The weight of a term $w$ , given the corpus $D$ , is defined as the total number of documents $|D|$ divided by the number of documents that have the term $w$ , $|D^w|$ . 
$$IDF(w | D) = \frac{|D|}{|D^w|}$$   (Eq. 10) 
Term-frequency inverse document-frequency (TF-IDF): The weight of a term $w$ is defined by the multiplication of its term-frequency and its inverse document-frequency. When considering only the TF weights, all terms have the same importance among the corpus. By using the IDF weight, words that are more common across all documents in $D$ receive a smaller weight, giving more importance to rare terms in the corpus. 
$$TFIDF(w,d | D)=TF \times IDF$$   (Eq. 12) 
In very large corpus, it is common to consider the logarithm of the IDF in order to dampen its effect. 
$$TFIDF(w,d | D)=TF \times log_{10}(IDF)$$   (Eq. 13) 

Text classification based on word subspace
We formulate our problem as a single label classification problem. Given a set of training documents, which we will refer as corpus, $D = \lbrace d_i\rbrace _{i=1}^{|D|}$ , with known classes $C = \lbrace c_j\rbrace _{j=1}^{|C|}$ , we wish to classify a query document $d_q$ into one of the classes in $C$ .
Text classification based on word subspace can be performed under the framework of mutual subspace method (MSM). This task involves two different stages: A learning stage, where the word subspace for each class is modeled, and a classification stage, where the word subspace for a query is modeled and compared to the word subspaces of the classes.
In the learning stage, it is assumed that all documents of the same class belong to the same context, resulting in a set of words $W_c = \lbrace w_c^k\rbrace _{k=1}^{N_c}$ . This set assumes that each word appears only once in each class. Each set $\lbrace W_c\rbrace _{c=1}^{|C|}$ is then modeled into a word subspace ${Y}_c$ , as explained in Section "Word subspace" . As the number of words in each class may vary largely, the dimension $m_c$ of each class word subspace is not set to the same value.
In the classification stage, for a query document $d_q$ , it is also assumed that each word occurs only once, generating a subspace ${Y}_q$ .
To measure the similarity between a class word subspace ${Y}_c$ and a query word subspace ${Y}_q$ , the canonical angles between the two word subspaces are used BIBREF21 . There are several methods for calculating canonical angles BIBREF22 , BIBREF23 , and BIBREF24 , among which the simplest and most practical is the singular value decomposition (SVD). Consider, for example, two subspaces, one from the training data and another from the query, represented as matrices of bases, ${Y}_{c} = [{\Phi }_{1} \ldots {\Phi }_{m_c}] \in \mathbb {R}^{p \times m_c}$ and ${Y}_{q} = [{\Psi }_{1} \ldots {\Psi }_{m_q}] \in \mathbb {R}^{p \times m_q}$ , where ${\Phi }_{i}$ are the bases for ${Y}_c$ and ${\Psi }_{i}$ are the bases for ${Y}_q$ . Let the SVD of ${Y}_c^{\top }{Y}_q \in \mathbb {R}^{m_c \times m_q}$ be ${Y}_c^{\top }{Y}_q = {U \Sigma V}^{\top }$ , where ${Y}_q$0 , ${Y}_q$1 represents the set of singular values. The canonical angles ${Y}_q$2 can be obtained as ${Y}_q$3 ${Y}_q$4 . The similarity between the two subspaces is measured by ${Y}_q$5 angles as follows: 
$$S_{({Y}_c,{Y}_q)}[t] = \frac{1}{t}\sum _{i = 1}^{t} \cos ^{2} \theta _{i},\; 1 \le t \le m_q, \; m_q \le m_c.$$   (Eq. 30) 
Fig. 1 shows the modeling and comparison of sets of words by MSM. This method can compare sets of different sizes, and naturally encodes proximity between sets with related words.
Finally, the class with the highest similarity with $d_q$ is assigned as the class of $d_q$ : 
$$prediction(d_q) = argmax_c(S_{({Y}_c,{Y}_q)}).$$   (Eq. 32) 

Experimental Evaluation
In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension. We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. Words in the texts were considered as they appeared, without performing stemming or typo correction. This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1 .
To obtain the vector representation of words, we used a freely available word2vec model, trained by BIBREF8 , on approximately 100 billion words, which encodes the vector representation in $\mathbb {R}^{300}$ of over 3 million words from several different languages. Since we decided to focus on English words only, we filtered these vectors to about 800 thousand words, excluding all words with non-roman characters.
To show the validity of our word subspace representation for text classification and the proposed extension, we divided our experiment section into two parts: The first one aims to verify if sets of word vectors are suitable for subspace representation, and the second one puts our methods in practice in a text classification test, comparing our results with the conventional methods described in Section "Related Work" .

Text classification experiment
In this experiment, we performed text classification among the classes in the Reuters-8 database. We compared the classification using the word subspace, and its weighted extension, based on MSM (to which we will refer as MSM and TF-MSM, respectively) with the baselines presented in Section "Related Work" : MVB, MNB, LSA, and SVM. Since none of the baseline methods work with vector set classification, we also compared to a simple baseline for comparing sets of vectors, defined as the average of similarities between all vector pair combinations of two given sets. For two matrices ${A}$ and ${B}$ , containing the sets of vectors $\lbrace  {x}^{i}_a \rbrace _{i = 1}^{N_A}$ and $\lbrace  {x}^{i}_b \rbrace _{i = 1}^{N_B}$ , respectively, where $N_A$ and $N_B$ are the number of main words in each set, the similarity is defined as: 
$$Sim_{(A,B)} = \frac{1}{N_A N_B}\sum _{i}^{N_A}\sum _{j}^{N_B}{{x}_a^i}^{\top }{x}_b^j.$$   (Eq. 41) 
We refer to this baseline as similarity average (SA). For this method, we only considered one occurrence of each word in each set.
Different features were used, depending on the method. Classification with SA, MSM, and TF-MSM was performed using word2vec features, to which we refer as w2v. For MVB, due to its nature, only bag-of-words features with binary weights were used (binBOW). For the same reason, we only used bag-of-words features with term-frequency weights (tfBOW) with MNB. Classification with LSA and SVM is usually performed using bag-of-words features and, therefore, we tested with binBOW, tfBOW, and with the term-frequency inverse document-frequency weight, tfidfBOW. We also tested them using word2vec vectors. In this case, we considered each word vector from all documents in each class to be a single sample.
To determine the dimensions of the class subspaces and query subspace of MSM and TF-MSM, and the dimension of the approximation performed by LSA, we performed a 10-fold cross validation, wherein each fold, the data were randomly divided into train (60%), validation (20%) and test set (20%).
The results can be seen in Table 2 . The simplest baseline, SA with w2v, achieved an accuracy rate of 78.73%. This result is important because it shows the validity of the word2vec representation, performing better than more elaborate methods based on BOW, such as MVB with binBOW.
LSA with BOW features was almost 10% more accurate than SA, where the best results with binary weights were achieved with an approximation with 130 dimensions, with TF weights were achieved with 50 dimensions, and with TF-IDF weights were achieved with 30 dimensions. SVM with BOW features was about 3% more accurate than LSA, with binary weights leading to a higher accuracy rate.
It is interesting to note that despite the reasonably high accuracy rates achieved using LSA and SVM with BOW features, they poorly performed when using w2v features.
Among the baselines, the best method was MNB with tfBOW features, with an accuracy of 91.47%, being the only conventional method to outperform MSM. MSM with w2v had an accuracy rate of 90.62%, with the best results achieved with word subspace dimensions for the training classes ranging from 150 to 181, and for the query ranging from 3 to 217. Incorporating the frequency information in the subspace modeling resulted in higher accuracy, with TF-MSM achieving 92.01%, with dimensions of word subspaces for training classes ranging from 150 to 172, and for the query, ranging from 2 to 109. To confirm that TF-MSM is significantly more accurate than MNB, we performed a t-test to compare their results. It resulted in a p-value of 0.031, which shows that at a 95% significance level, TF-MSM has produced better results.

Acknowledgment
This work is supported by JSPS KAKENHI Grant Number JP16H02842 and the Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) scholarship.

TABLE II RESULTS FOR THE TEXT CLASSIFICATION EXPERIMENT ON REUTERS-8 DATABASE

Question: Which dataset has been used in this work?
{"answer": ["Reuters-8 dataset without stop words"], "evidence": ["In this section we describe the experiments performed to demonstrate the validity of our proposed method and its extension. We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. Words in the texts were considered as they appeared, without performing stemming or typo correction. This database has eight different classes with the number of samples varying from 51 to over 3000 documents, as can be seen in Table 1 ."]}