You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Knowledge Graph Representation with Jointly Structural and Textual Encoding

The objective of knowledge graph embedding is to encode both entities and relations of knowledge graphs into continuous low-dimensional vector spaces. Previously, most works focused on symbolic representation of knowledge graph with structure information, which can not handle new entities or entities with few facts well. In this paper, we propose a novel deep architecture to utilize both structural and textual information of entities. Specifically, we introduce three neural models to encode the valuable information from text description of entity, among which an attentive model can select related information as needed. Then, a gating mechanism is applied to integrate representations of structure and text into a unified architecture. Experiments show that our models outperform baseline by margin on link prediction and triplet classification tasks. Source codes of this paper will be available on Github.

Knowledge Graph Embedding
In this section, we briefly introduce the background knowledge about the knowledge graph embedding.
Knowledge graph embedding aims to model multi-relational data (entities and relations) into a continuous low-dimensional vector space. Given a pair of entities $(h,t)$ and their relation $r$ , we can represent them with a triple $(h,r,t)$ . A score function $f(h,r, t)$ is defined to model the correctness of the triple $(h,r,t)$ , thus to distinguish whether two entities $h$ and $t$ are in a certain relationship $r$ . $f(h,r, t)$ should be larger for a golden triplet $(h, r, t)$ that corresponds to a true fact in real world, otherwise $r$0 should be lower for an negative triplet.
The difference among the existing methods varies between linear BIBREF2 , BIBREF8 and nonlinear BIBREF3 score functions in the low-dimensional vector space.
Among these methods, TransE BIBREF2 is a simple and effective approach, which learns the vector embeddings for both entities and relationships. Its basic idea is that the relationship between two entities is supposed to correspond to a translation between the embeddings of entities, that is, $\textbf {h}+ \mathbf {r}\approx \mathbf {t}$ when $(h,r,t)$ holds.
TransE's score function is defined as: 
$$f(h,r,t)) &= -\Vert \textbf {h}+\mathbf {r}-\mathbf {t}\Vert _{2}^2$$   (Eq. 5) 
 where $\textbf {h},\mathbf {t},\mathbf {r}\in \mathbb {R}^d$ are embeddings of $h,t,r$ respectively, and satisfy $\Vert \textbf {h}\Vert ^2_2=\Vert \mathbf {t}\Vert ^2_2=1$ . The $\textbf {h}, \mathbf {r}, \mathbf {t}$ are indexed by a lookup table respectively.

Bag-of-Words Encoder
A simple and intuitive method is the neural bag-of-words (NBOW) model, in which the representation of text can be generated by summing up its constituent word representations.
We denote the text description as word sequence $x_{1:n} = x_1,\cdots ,x_n$ , where $x_i$ is the word at position $i$ . The NBOW encoder is 
$$\mathrm {enc_1}(x_{1:n}) = \sum _{i=1}^{n} \mathbf {x}_i,$$   (Eq. 7) 
 where $\mathbf {x}_i \in \mathbb {R}^d$ is the word embedding of $x_i$ .

LSTM Encoder
To address some of the modelling issues with NBOW, we consider using a bidirectional long short-term memory network (LSTM) BIBREF14 , BIBREF15 to model the text description.
LSTM was proposed by BIBREF16 to specifically address this issue of learning long-term dependencies BIBREF17 , BIBREF18 , BIBREF16 in RNN. The LSTM maintains a separate memory cell inside it that updates and exposes its content only when deemed necessary.
Bidirectional LSTM (BLSTM) can be regarded as two separate LSTMs with different directions. One LSTM models the text description from left to right, and another LSTM models text description from right to left respectively. We define the outputs of two LSTM at time step $i$ are $\overrightarrow{\mathbf {z}}_i$ and $\overleftarrow{\mathbf {z}}_i$ respectively.
The combined output of BLSTM at position $i$ is ${\mathbf {z}_i} = \overrightarrow{\mathbf {z}}_i \oplus \overleftarrow{\mathbf {z}}_i$ , where $\oplus $ denotes the concatenation operation.
The LSTM encoder combines all the outputs $\mathbf {z}_i \in \mathbb {R}^d$ of BLSTM at different position. 
$$\mathrm {enc_2}(x_{1:n}) = \sum _{i=1}^{n} {\mathbf {z}_i}.$$   (Eq. 9) 

Attentive LSTM Encoder
While the LSTM encoder has richer capacity than NBOW, it produces the same representation for the entire text description regardless of its contexts. However, the text description may present an entity from various aspects, and various relations only focus on fractional aspects of the description. This phenomenon also occurs in structure embedding for an entity BIBREF8 , BIBREF19 .
Given a relation for an entity, not all of words/phrases in its text description are useful to model a specific fact. Some of them may be important for the given relation, but may be useless for other relations. Therefore, we introduce an attention mechanism BIBREF20 to utilize an attention-based encoder that constructs contextual text encodings according to different relations.
For each position $i$ of the text description, the attention for a given relation $r$ is defined as $\alpha _i(r)$ , which is 
$$e_i(r) &= \mathbf {v}_a^T \tanh (\mathbf {W}_a {\mathbf {z}}_i + \mathbf {U}_a \mathbf {r}), \\
\alpha _i(r)&=\operatorname{\mathbf {softmax}}(e_i(r))\nonumber \\
&=\frac{\exp (e_i(r))}{\sum ^{n}_{j=1} \exp (e_j(r))},$$   (Eq. 12) 
 where $\mathbf {r}\in \mathbb {R}^d$ is the relation embedding; ${\mathbf {z}}_i \in \mathbb {R}^d$ is the output of BLSTM at position $i$ ; $\mathbf {W}_a,\mathbf {U}_a \in \mathbb {R}^{d\times d}$ are parameters matrices; $\mathbf {v}_a \in \mathbb {R}^{d}$ is a parameter vector.
The attention $\alpha _i(r)$ is interpreted as the degree to which the network attends to partial representation $\mathbf {z}_{i}$ for given relation $r$ .
The contextual encoding of text description can be formed by a weighted sum of the encoding $\mathbf {z}_{i}$ with attention. 
$$\mathbf {enc_3}(x_{1:n};r) &= \sum _{i=1}^{n} \alpha _i(r) * \mathbf {z}_i.$$   (Eq. 13) 

Joint Structure and Text Encoder
Since both the structure and text description provide valuable information for an entity , we wish to integrate all these information into a joint representation.
We propose a united model to learn a joint representation of both structure and text information. The whole model can be end-to-end trained.
For an entity $e$ , we denote $\mathbf {e}_s$ to be its embedding of structure information, $\mathbf {e}_d$ to be encoding of its text descriptions. The main concern is how to combine $\mathbf {e}_s$ and $\mathbf {e}_d$ .
To integrate two kinds of representations of entities, we use gating mechanism to decide how much the joint representation depends on structure or text.
The joint representation $\mathbf {e}$ is a linear interpolation between the $\mathbf {e}_s$ and $\mathbf {e}_d$ . 
$$\mathbf {e}= \textbf {g}_e \odot \mathbf {e}_s + (1-\textbf {g}_e)\odot \mathbf {e}_d,$$   (Eq. 14) 
 where $\textbf {g}_e$ is a gate to balance two sources information and its elements are in $[0,1]$ , and $\odot $ is an element-wise multiplication. Intuitively, when the gate is close to 0, the joint representation is forced to ignore the structure information and is the text representation only.

Datasets
We use two popular knowledge bases: WordNet BIBREF0 and Freebase BIBREF1 in this paper. Specifically, we use WN18 (a subset of WordNet) BIBREF24 and FB15K (a subset of Freebase) BIBREF2 since their text descriptions are easily publicly available. Table 1 lists statistics of the two datasets.

Link Prediction
Link prediction is a subtask of knowledge graph completion to complete a triplet $(h, r, t)$ with $h$ or $t$ missing, i.e., predict $t$ given $(h, r)$ or predict $h$ given $(r, t)$ . Rather than requiring one best answer, this task emphasizes more on ranking a set of candidate entities from the knowledge graph.
Similar to BIBREF2 , we use two measures as our evaluation metrics. (1) Mean Rank: the averaged rank of correct entities or relations; (2) Hits@p: the proportion of valid entities or relations ranked in top $p$ predictions. Here, we set $p=10$ for entities and $p=1$ for relations. A lower Mean Rank and a higher Hits@p should be achieved by a good embedding model. We call this evaluation setting “Raw”. Since a false predicted triplet may also exist in knowledge graphs, it should be regard as a valid triplet. Hence, we should remove the false predicted triplets included in training, validation and test sets before ranking (except the test triplet of interest). We call this evaluation setting “Filter”. The evaluation results are reported under these two settings.
We select the margin $\gamma $ among $\lbrace 1, 2\rbrace $ , the embedding dimension $d$ among $\lbrace 20, 50, 100\rbrace $ , the regularization $\eta $ among $\lbrace 0, 1E{-5}, 1E{-6}\rbrace $ , two learning rates $\lambda _s$ and $\lambda _t$ among $\lbrace 0.001, 0.01, 0.05\rbrace $ to learn the parameters of structure and text encoding. The dissimilarity measure is set to either $L_1$ or $\lbrace 1, 2\rbrace $0 distance.
In order to speed up the convergence and avoid overfitting, we initiate the structure embeddings of entity and relation with the results of TransE. The embedding of a word is initialized by averaging the linked entity embeddings whose description include this word. The rest parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ .
The final optimal configurations are: $\gamma = 2$ , $d=20$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $\lambda _t = 0.1$ , and $L_1$ distance on WN18; $\gamma =2$ , $d=100$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $d=20$0 , and $d=20$1 distance on FB15K.
Experimental results on both WN18 and FB15k are shown in Table 2 , where we use “Jointly(CBOW)”, “Jointly(LSTM)” and “Jointly(A-LSTM)” to represent our jointly encoding models with CBOW, LSTM and attentive LSTM text encoders. Our baseline is TransE since that the score function of our models is based on TransE.
From the results, we observe that proposed models surpass the baseline, TransE, on all metrics, which indicates that knowledge representation can benefit greatly from text description.
On WN18, the reason why “Jointly(A-LSTM)” is slightly worse than “Jointly(LSTM)” is probably because the number of relations is limited. Therefore, the attention mechanism does not have obvious advantage. On FB15K, “Jointly(A-LSTM)” achieves the best performance and is significantly higher than baseline methods on mean rank.
Although the Hits@10 of our models are worse than the best state-of-the-art method, TransD, it is worth noticing that the score function of our models is based on TransE, not TransD. Our models are compatible with other state-of-the-art knowledge embedding models. We believe that our model can be further improved by adopting the score functions of other state-of-the-art methods, such as TransD.
Besides, textual information largely alleviates the issue of sparsity and our model achieves substantial improvement on Mean Rank comparing with TransD. However, textual information may slightly degrade the representation of frequent entities which have been well-trained. This may be another reason why our Hits@10 is worse than TransD which only utilizes structural information.
For the comparison of Hits@10 of different kinds of relations, we categorized the relationships according to the cardinalities of their head and tail arguments into four classes: 1-to-1, 1-to-many, many-to-1, many-to-many. Mapping properties of relations follows the same rules in BIBREF2 .
Table 3 shows the detailed results by mapping properties of relations on FB15k. We can see that our models outperform baseline TransE in all types of relations (1-to-1, 1-to-N, N-to-1 and N-to-N), especially when (1) predicting “1-to-1” relations and (2) predicting the 1 side for “1-to-N” and “N-to-1” relations.
To get more insights into how the joint representation is influenced by the structure and text information. We observe the activations of gates, which control the balance between two sources of information, to understand the behavior of neurons. We sort the entities by their frequencies and divide them into 50 equal-size groups of different frequencies, and average the values of all gates in each group.
Figure 3 gives the average of gates in ten groups from high- to low-frequency. We observe that the text information play more important role for the low-frequency entities.

Conclusion
We propose a united representation for knowledge graph, utilizing both structure and text description information of the entities. Experiments show that our proposed jointly representation learning with gating mechanism is effective, which benefits to modeling the meaning of an entity.
In the future, we will consider the following research directions to improve our model:

Table 1: Statistics of datasets used in experiments.

Table 2: Results on link prediction.

Table 3: Detailed results by category of relationship on FB15K.

Figure 4: Visualization of gates with different entity frequencies on FB15K.

Question: What neural models are used to encode the text?
{"answer": ["neural bag-of-words (NBOW) model", "bidirectional long short-term memory network (LSTM)", "attention-based encoder"], "evidence": ["A simple and intuitive method is the neural bag-of-words (NBOW) model, in which the representation of text can be generated by summing up its constituent word representations.", "To address some of the modelling issues with NBOW, we consider using a bidirectional long short-term memory network (LSTM) BIBREF14 , BIBREF15 to model the text description.", "Given a relation for an entity, not all of words/phrases in its text description are useful to model a specific fact. Some of them may be important for the given relation, but may be useless for other relations. Therefore, we introduce an attention mechanism BIBREF20 to utilize an attention-based encoder that constructs contextual text encodings according to different relations."]}