You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Imparting Interpretability to Word Embeddings while Preserving Semantic Structure

As an ubiquitous method in natural language processing, word embeddings are extensively employed to map semantic properties of words into a dense vector representation. They capture semantic and syntactic relations among words but the vector corresponding to the words are only meaningful relative to each other. Neither the vector nor its dimensions have any absolute, interpretable meaning. We introduce an additive modification to the objective function of the embedding learning algorithm that encourages the embedding vectors of words that are semantically related to a predefined concept to take larger values along a specified dimension, while leaving the original semantic learning mechanism mostly unaffected. In other words, we align words that are already determined to be related, along predefined concepts. Therefore, we impart interpretability to the word embedding by assigning meaning to its vector dimensions. The predefined concepts are derived from an external lexical resource, which in this paper is chosen as Roget's Thesaurus. We observe that alignment along the chosen concepts is not limited to words in the Thesaurus and extends to other related words as well. We quantify the extent of interpretability and assignment of meaning from our experimental results. We also demonstrate the preservation of semantic coherence of the resulting vector space by using word-analogy and word-similarity tests. These tests show that the interpretability-imparted word embeddings that are obtained by the proposed framework do not sacrifice performances in common benchmark tests.

Related Work
Methodologically, our work is related to prior studies that aim to obtain “improved” word embeddings using external lexical resources, under some performance metric. Previous work in this area can be divided into two main categories: works that i) modify the word embedding learning algorithm to incorporate lexical information, ii) operate on pre-trained embeddings with a post-processing step.
Among works that follow the first approach, BIBREF28 extend the Skip-Gram model by incorporating the word similarity relations extracted from the Paraphrase Database (PPDB) and WordNet BIBREF29 , into the Skip-Gram predictive model as an additional cost term. In BIBREF30 , the authors extend the CBOW model by considering two types of semantic information, termed relational and categorical, to be incorporated into the embeddings during training. For the former type of semantic information, the authors propose the learning of explicit vectors for the different relations extracted from a semantic lexicon such that the word pairs that satisfy the same relation are distributed more homogeneously. For the latter, the authors modify the learning objective such that some weighted average distance is minimized for words under the same semantic category. In BIBREF31 , the authors represent the synonymy and hypernymy-hyponymy relations in terms of inequality constraints, where the pairwise similarity rankings over word triplets are forced to follow an order extracted from a lexical resource. Following their extraction from WordNet, the authors impose these constraints in the form of an additive cost term to the Skip-Gram formulation. Finally, BIBREF32 builds on top of the GloVe algorithm by introducing a regularization term to the objective function that encourages the vector representations of similar words as dictated by WordNet to be similar as well.
Turning our attention to the post-processing approach for enriching word embeddings with external lexical knowledge, BIBREF33 has introduced the retrofitting algorithm that acts on pre-trained embeddings such as Skip-Gram or GloVe. The authors propose an objective function that aims to balance out the semantic information captured in the pre-trained embeddings with the constraints derived from lexical resources such as WordNet, PPDB and FrameNet. One of the models proposed in BIBREF34 extends the retrofitting approach to incorporate the word sense information from WordNet. Similarly, BIBREF35 creates multi-sense embeddings by gathering the word sense information from a lexical resource and learning to decompose the pre-trained embeddings into a convex combination of sense embeddings. In BIBREF36 , the authors focus on improving word embeddings for capturing word similarity, as opposed to mere relatedness. To this end, they introduce the counter-fitting technique which acts on the input word vectors such that synonymous words are attracted to one another whereas antonymous words are repelled, where the synonymy-antonymy relations are extracted from a lexical resource. More recently, the ATTRACT-REPEL algorithm proposed by BIBREF37 improves on counter-fitting by a formulation which imparts the word vectors with external lexical information in mini-batches.
Most of the studies discussed above ( BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF36 , BIBREF37 ) report performance improvements in benchmark tests such as word similarity or word analogy, while BIBREF29 uses a different analysis method (mean reciprocal rank). In sum, the literature is rich with studies aiming to obtain word embeddings that perform better under specific performance metrics. However, less attention has been directed to the issue of interpretability of the word embeddings. In the literature, the problem of interpretability has been tackled using different approaches. BIBREF19 proposed non-negative matrix factorization (NMF) for learning sparse, interpretable word vectors from co-occurrence variant matrices where the resulting vector space is called non-negative sparse embeddigns (NNSE). However, since NMF methods require maintaining a global matrix for learning, they suffer from memory and scale issue. This problem has been addressed in BIBREF20 where an online method of learning interpretable word embeddings from corpora using a modified version of skip-gram model BIBREF0 is proposed. As a different approach, BIBREF21 combined text-based similarity information among words with brain activity based similarity information to improve interpretability using joint non-negative sparse embedding (JNNSE).
A common alternative approach for learning interpretable embeddings is to learn transformations that map pre-trained state-of-the-art embeddings to new interpretable semantic spaces. To obtain sparse, higher dimensional and more interpretable vector spaces, BIBREF22 and BIBREF23 use sparse coding on conventional dense word embeddings. However, these methods learn the projection vectors that are used for the transformation from the word embeddings without supervision. For this reason, labels describing the corresponding semantic categories cannot be provided. An alternative approach was proposed in BIBREF25 , where orthogonal transformations were utilized to increase interpretability while preserving the performance of the underlying embedding. However, BIBREF25 has also shown that total interpretability of an embedding is kept constant under any orthogonal transformation and it can only be redistributed across the dimensions. Rotation algorithms based on exploratory factor analysis (EFA) to preserve the performance of the original word embeddings while improving their interpretability was proposed in BIBREF26 . BIBREF24 proposed to deploy a sparse auto-encoder using pre-trained dense word embeddings to improve interpretability. More detailed investigation of semantic structure and interpretability of word embeddings can be found in BIBREF27 , where a metric was proposed to quantitatively measure the degree of interpretability already present in the embedding vector spaces.
Previous works on interpretability mentioned above, except BIBREF21 , BIBREF27 and our proposed method, do not need external resources, utilization of which has both advantages and disadvantages. Methods that do not use external resources require fewer resources but they also lack the aid of information extracted from these resources.

Quantitative Evaluation for Interpretability
One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon. A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT. Interpretability scores are calculated using Interpretability Score (IS) as given below:
 DISPLAYFORM0 
In ( EQREF17 ), INLINEFORM0 and INLINEFORM1 represents the interpretability scores in the positive and negative directions of the INLINEFORM2 dimension ( INLINEFORM3 , INLINEFORM4 number of dimensions in the embedding space) of word embedding space for the INLINEFORM5 category ( INLINEFORM6 , INLINEFORM7 is number of categories in SEMCAT, INLINEFORM8 ) in SEMCAT respectively. INLINEFORM9 is the set of words in the INLINEFORM10 category in SEMCAT and INLINEFORM11 is the number of words in INLINEFORM12 . INLINEFORM13 corresponds to the minimum number of words required to construct a semantic category (i.e. represent a concept). INLINEFORM14 represents the set of INLINEFORM15 words that have the highest ( INLINEFORM16 ) and lowest ( INLINEFORM17 ) values in INLINEFORM18 dimension of the embedding space. INLINEFORM19 is the intersection operator and INLINEFORM20 is the cardinality operator (number of elements) for the intersecting set. In ( EQREF17 ), INLINEFORM21 gives the interpretability score for the INLINEFORM22 dimension and INLINEFORM23 gives the average interpretability score of the embedding space.
Fig. FIGREF18 presents the measured average interpretability scores across dimensions for original GloVe embeddings, for the proposed method and for the other four methods we compare, along with a randomly generated embedding. Results are calculated for the parameters INLINEFORM0 and INLINEFORM1 . Our proposed method significantly improves the interpretability for all INLINEFORM2 compared to the original GloVe approach. Our proposed method is second to only SPINE in increasing interpretability. However, as we will experimentally demonstrate in the next subsection, in doing this, SPINE almost entirely destroys the underlying semantic structure of the word embeddings, which is the primary function of a word embedding.
The proposed method and interpretability measurements are both based on utilizing concepts represented by word-groups. Therefore it is expected that there will be higher interpretability scores for some of the dimensions for which the imparted concepts are also contained in SEMCAT. However, by design, word groups that they use are formed by using different sources and are independent. Interpretability measurements use SEMCAT while our proposed method utilizes Roget's Thesaurus.

TABLE I SAMPLE CONCEPTS AND THEIR ASSOCIATED WORD-GROUPS FROM ROGET’S THESAURUS

Fig. 2. Most frequent 1000 words sorted according to their values in the 32nd dimension of the original GloVe embedding are shown with blue markers. Red and green markers show the values of the same words for the 32nd dimension of the embedding obtained with the proposed method where the dimension is aligned with the concept JUDGMENT. Words with green markers are contained in the concept JUDGMENT while words with red markers are not contained.

TABLE V CORRELATIONS FOR WORD SIMILARITY TESTS

TABLE VII PRECISION SCORES FOR THE SEMANTIC ANALOGY TEST

Question: What experiments do they use to quantify the extent of interpretability?
{"answer": ["semantic category-based approach"], "evidence": ["One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon. A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT. Interpretability scores are calculated using Interpretability Score (IS) as given below:"]}