You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Imparting Interpretability to Word Embeddings while Preserving Semantic Structure

As an ubiquitous method in natural language processing, word embeddings are extensively employed to map semantic properties of words into a dense vector representation. They capture semantic and syntactic relations among words but the vector corresponding to the words are only meaningful relative to each other. Neither the vector nor its dimensions have any absolute, interpretable meaning. We introduce an additive modification to the objective function of the embedding learning algorithm that encourages the embedding vectors of words that are semantically related to a predefined concept to take larger values along a specified dimension, while leaving the original semantic learning mechanism mostly unaffected. In other words, we align words that are already determined to be related, along predefined concepts. Therefore, we impart interpretability to the word embedding by assigning meaning to its vector dimensions. The predefined concepts are derived from an external lexical resource, which in this paper is chosen as Roget's Thesaurus. We observe that alignment along the chosen concepts is not limited to words in the Thesaurus and extends to other related words as well. We quantify the extent of interpretability and assignment of meaning from our experimental results. We also demonstrate the preservation of semantic coherence of the resulting vector space by using word-analogy and word-similarity tests. These tests show that the interpretability-imparted word embeddings that are obtained by the proposed framework do not sacrifice performances in common benchmark tests.

Introduction
Distributed word representations, commonly referred to as word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , serve as elementary building blocks in the course of algorithm design for an expanding range of applications in natural language processing (NLP), including named entity recognition BIBREF4 , BIBREF5 , parsing BIBREF6 , sentiment analysis BIBREF7 , BIBREF8 , and word-sense disambiguation BIBREF9 . Although the empirical utility of word embeddings as an unsupervised method for capturing the semantic or syntactic features of a certain word as it is used in a given lexical resource is well-established BIBREF10 , BIBREF11 , BIBREF12 , an understanding of what these features mean remains an open problem BIBREF13 , BIBREF14 and as such word embeddings mostly remain a black box. It is desirable to be able to develop insight into this black box and be able to interpret what it means, while retaining the utility of word embeddings as semantically-rich intermediate representations. Other than the intrinsic value of this insight, this would not only allow us to explain and understand how algorithms work BIBREF15 , but also set a ground that would facilitate the design of new algorithms in a more deliberate way.
Recent approaches to generating word embeddings (e.g. BIBREF0 , BIBREF2 ) are rooted linguistically in the field of distributed semantics BIBREF16 , where words are taken to assume meaning mainly by their degree of interaction (or lack thereof) with other words in the lexicon BIBREF17 , BIBREF18 . Under this paradigm, dense, continuous vector representations are learned in an unsupervised manner from a large corpus, using the word cooccurrence statistics directly or indirectly, and such an approach is shown to result in vector representations that mathematically capture various semantic and syntactic relations between words BIBREF0 , BIBREF2 , BIBREF3 . However, the dense nature of the learned embeddings obfuscate the distinct concepts encoded in the different dimensions, which renders the resulting vectors virtually uninterpretable. The learned embeddings make sense only in relation to each other and their specific dimensions do not carry explicit information that can be interpreted. However, being able to interpret a word embedding would illuminate the semantic concepts implicitly represented along the various dimensions of the embedding, and reveal its hidden semantic structures.
In the literature, researchers tackled interpretability problem of the word embeddings using different approaches. Several researchers BIBREF19 , BIBREF20 , BIBREF21 proposed algorithms based on non-negative matrix factorization (NMF) applied to cooccurrence variant matrices. Other researchers suggested to obtain interpretable word vectors from existing uninterpretable word vectors by applying sparse coding BIBREF22 , BIBREF23 , by training a sparse auto-encoder to transform the embedding space BIBREF24 , by rotating the original embeddings BIBREF25 , BIBREF26 or by applying transformations based on external semantic datasets BIBREF27 .
Although the above-mentioned approaches provide better interpretability that is measured using a particular method such as word intrusion test, usually the improved interpretability comes with a cost of performance in the benchmark tests such as word similarity or word analogy. One possible explanation for this performance decrease is that the proposed transformations from the original embedding space distort the underlying semantic structure constructed by the original embedding algorithm. Therefore, it can be claimed that a method that learns dense and interpretable word embeddings without inflicting any damage to the underlying semantic learning mechanism is the key to achieve both high performing and interpretable word embeddings.
Especially after the introduction of the word2vec algorithm by Mikolov BIBREF0 , BIBREF1 , there has been a growing interest in algorithms that generate improved word representations under some performance metric. Significant effort is spent on appropriately modifying the objective functions of the algorithms in order to incorporate knowledge from external resources, with the purpose of increasing the performance of the resulting word representations BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 . Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings. By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts. This alignment is achieved by introducing a modification to the embedding learning process. In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function. Each embedding vector dimension is first associated with a concept. For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word's embedding vector dimension corresponding to the concept that the particular word belongs to. For words that do not belong to any one of the word-groups, the cost term is left untouched. Specifically, Roget's Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method. We quantitatively demonstrate the increase in interpretability by using the measure given in BIBREF27 , BIBREF40 as well as demonstrating qualitative results. We also show that the semantic structure of the original embedding has not been harmed in the process since there is no performance loss with standard word-similarity or word-analogy tests.
The paper is organized as follows. In Section SECREF2 , we discuss previous studies related to our work under two main categories: interpretability of word embeddings and joint-learning frameworks where the objective function is modified. In Section SECREF3 , we present the problem framework and provide the formulation within the GloVe BIBREF2 algorithm setting. In Section SECREF4 where our approach is proposed, we motivate and develop a modification to the original objective function with the aim of increasing representation interpretability. In Section SECREF5 , experimental results are provided and the proposed method is quantitatively and qualitatively evaluated. Additionally, in Section SECREF5 , results demonstrating the extent to which the original semantic structure of the embedding space is affected are presented by using word-analogy and word-similarity tests. We conclude the paper in Section SECREF6 .

Intrinsic Evaluation of the Embeddings
It is necessary to show that the semantic structure of the original embedding has not been damaged or distorted as a result of aligning the dimensions with given concepts, and that there is no substantial sacrifice involved from the performance that can be obtained with the original GloVe. To check this, we evaluate performances of the proposed embeddings on word similarity BIBREF42 and word analogy BIBREF0 tests. We compare the results with the original embeddings and the three alternatives excluding Parsimax BIBREF26 since orthogonal transformations will not affect the performance of the original embeddings on these tests.
Word similarity test measures the correlation between word similarity scores obtained from human evaluation (i.e. true similarities) and from word embeddings (usually using cosine similarity). In other words, this test quantifies how well the embedding space reflects human judgements in terms of similarities between different words. The correlation scores for 13 different similarity test sets are reported in Table TABREF20 . We observe that, let alone a reduction in performance, the obtained scores indicate an almost uniform improvement in the correlation values for the proposed algorithm, outperforming all the alternatives in almost all test sets. Categories from Roget's thesaurus are groupings of words that are similar in some sense which the original embedding algorithm may fail to capture. These test results signify that the semantic information injected into the algorithm by the additional cost term is significant enough to result in a measurable improvement. It should also be noted that scores obtained by SPINE is unacceptably low on almost all tests indicating that it has achieved its interpretability performance at the cost of losing its semantic functions.
Word analogy test is introduced in BIBREF1 and looks for the answers of the questions that are in the form of "X is to Y, what Z is to ?" by applying simple arithmetic operations to vectors of words X, Y and Z. We present precision scores for the word analogy tests in Table TABREF21 . It can be seen that the alternative approaches that aim to improve interpretability, have poor performance on the word analogy tests. However, our proposed method has comparable performance with the original GloVe embeddings. Our method outperforms GloVe in semantic analogy test set and in overall results, while GloVe performs slightly better in syntactic test set. This comparable performance is mainly due to the cost function of our proposed method that includes the original objective of the GloVe.
To investigate the effect of the additional cost term on the performance improvement in the semantic analogy test, we present Table TABREF22 . In particular, we present results for the cases where i) all questions in the dataset are considered, ii) only the questions that contains at least one concept word are considered, iii) only the questions that consist entirely of concept words are considered. We note specifically that for the last case, only a subset of the questions under the semantic category family.txt ended up being included. We observe that for all three scenarios, our proposed algorithm results in an improvement in the precision scores. However, the greatest performance increase is seen for the last scenario, which underscores the extent to which the semantic features captured by embeddings can be improved with a reasonable selection of the lexical resource from which the concept word-groups were derived.

TABLE II GLOVE PARAMETERS
----------------------------------------

Question: What is the additive modification to the objective function?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 453 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: What is the additive modification to the objective function?

Answer JSON object:
{"answer": "The cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function. . Each embedding vector dimension is first associated with a concept. For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word\u0027s embedding vector dimension corresponding to the concept that the particular word belongs to,", "evidence": ["Especially after the introduction of the word2vec algorithm by Mikolov BIBREF0 , BIBREF1 , there has been a growing interest in algorithms that generate improved word representations under some performance metric. Significant effort is spent on appropriately modifying the objective functions of the algorithms in order to incorporate knowledge from external resources, with the purpose of increasing the performance of the resulting word representations BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 . Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings. By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts. This alignment is achieved by introducing a modification to the embedding learning process. In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function. Each embedding vector dimension is first associated with a concept. For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word\u0027s embedding vector dimension corresponding to the concept that the particular word belongs to. For words that do not belong to any one of the word-groups, the cost term is left untouched. Specifically, Roget\u0027s Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method. We quantitatively demonstrate the increase in interpretability by using the measure given in BIBREF27 , BIBREF40 as well as demonstrating qualitative results. We also show that the semantic structure of the original embedding has not been harmed in the process since there is no performance loss with standard word-similarity or word-analogy tests."]}