You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation

Automated analysis of clinical notes is attracting increasing attention. However, there has not been much work on medical term abbreviation disambiguation. Such abbreviations are abundant, and highly ambiguous, in clinical documents. One of the main obstacles is the lack of large scale, balance labeled data sets. To address the issue, we propose a few-shot learning approach to take advantage of limited labeled data. Specifically, a neural topic-attention model is applied to learn improved contextualized sentence representations for medical term abbreviation disambiguation. Another vital issue is that the existing scarce annotations are noisy and missing. We re-examine and correct an existing dataset for training and collect a test set to evaluate the models fairly especially for rare senses. We train our model on the training set which contains 30 abbreviation terms as categories (on average, 479 samples and 3.24 classes in each term) selected from a public abbreviation disambiguation dataset, and then test on a manually-created balanced dataset (each class in each term has 15 samples). We show that enhancing the sentence representation with topic information improves the performance on small-scale unbalanced training datasets by a large margin, compared to a number of baseline models.

Introduction
Medical text mining is an exciting area and is becoming attractive to natural language processing (NLP) researchers. Clinical notes are an example of text in the medical area that recent work has focused on BIBREF0, BIBREF1, BIBREF2. This work studies abbreviation disambiguation on clinical notes BIBREF3, BIBREF4, specifically those used commonly by physicians and nurses. Such clinical abbreviations can have a large number of meanings, depending on the specialty BIBREF5, BIBREF6. For example, the term MR can mean magnetic resonance, mitral regurgitation, mental retardation, medical record and the general English Mister (Mr.). Table TABREF1 illustrates such an example. Abbreviation disambiguation is an important task in medical text understanding task BIBREF7. Successful recognition of the abbreviations in the notes can contribute to downstream tasks such as classification, named entity recognition, and relation extraction BIBREF7.
Recent work focuses on formulating the abbreviation disambiguation task as a classification problem, where the possible senses of a given abbreviation term are pre-defined with the help of domain experts BIBREF6, BIBREF5. Traditional features such as part-of-speech (POS) and Term Frequency-Inverse Document Frequency (TF-IDF) are widely investigated for clinical notes classification. Classifiers like support vector machines (SVMs) and random forests (RFs) are used to make predictions BIBREF1. Such methods depend heavily on feature engineering. Recently, deep features have been investigated in the medical domain. Word embeddings BIBREF8 and Convolutional Neural Networks (CNNs) BIBREF9, BIBREF10 provide very competitive performance on text classification for clinical notes and abbreviation disambiguation task BIBREF0, BIBREF11, BIBREF12, BIBREF6. Beyond vanilla embeddings, BIBREF13 utilized contextual features to do abbreviation expansion. Another challenge is the difficulty in obtaining training data: clinical notes have many restrictions due to privacy issues and require domain experts to develop high-quality annotations, thus leading to limited annotated training and testing data. Another difficulty is that in the real world (and in the existing public datasets), some abbreviation term-sense pairs (for example, AB as abortion) have a very high frequency of occurrence BIBREF5, while others are rarely found. This long tail issue creates the challenge of training under unbalanced sample distributions. We tackle these problems in the setting of few-shot learning BIBREF14, BIBREF15 where only a few or a low number of samples can be found in the training dataset to make use of limited resources. We propose a model that combines deep contextual features based on ELMo BIBREF16 and topic information to solve the abbreviation disambiguation task.
Our contributions can be summarized as: 1) we re-examined and corrected an existing dataset for training and we collected a test set for evaluation with focus especially for rare senses; 2) we proposed a few-shot learning approach which combines topic information and contextualized word embeddings to solve clinical abbreviation disambiguation task. The implementation is available online; 3) as limited research are conducted on this particular task, we evaluated and compared a number of baseline methods including classical models and deep models comprehensively.

Datasets
Training Dataset UM Inventory BIBREF5 is a public dataset created by researchers from the University of Minnesota, containing about 37,500 training samples with 75 abbreviation terms. Existing work reports abbreviation disambiguation results on 50 abbreviation terms BIBREF6, BIBREF5, BIBREF17. However, after carefully reviewing this dataset, we found that it contains many samples where medical professionals disagree: wrong samples and uncategorized samples. Due to these mistakes and flaws of this dataset, we removed the erroneous samples and eventually selected 30 abbreviation terms as our training dataset that can be made public. Among all the abbreviation terms, we have 11,466 samples, and 93 term-sense pairs in total (on average 123.3 samples/term-sense pair and 382.2 samples/term). Some term-sense pairs are very popular with a larger number of training samples but some are not (typically less than 5), we call them rare-sense cases . More details can be found in Appendix SECREF7.
Testing Dataset Our testing dataset takes MIMIC-III BIBREF18 and PubMed as data sources. Here we are referring to all the notes data from MIMIC-III (NOTEEVENTS table ) and Case Reports articles from PubMed, as this type of contents are close to medical notes. We provide detailed information in Appendix SECREF8, including the steps to create the testing dataset. Eventually, we have a balanced testing dataset, where each term-sense pair has at least 11 and up to 15 samples for training (on average, each pair has 14.56 samples and the median sample number is 15).

Topic-attention Model
ELMo ELMo is a new type of word representation introduced by BIBREF16 which considers contextual information captured by pre-trained BiLSTM language models. Similar works like BERT BIBREF23 and BioBERT BIBREF24 also consider context but with deeper structures. Compared with those models, ELMo has less parameters and not easy to be overfitting. We trained our ELMo model on the MIMIC-III corpus. Since some sentences also appear in the test set, one may raise the concern of performance inflation in testing. However, we pre-trained ELMo using the whole corpus of MIMIC in an unsupervised way, so the classification labels were not visible during training. To train the ELMo model, we adapted code from AllenNLP, we set the word embedding dimension to 64 and applied two BiLSTM layers. For all of our experiments that involved with ELMo, we initialized the word embeddings from the last layer of ELMo.
Topic-attention We propose a neural topic-attention model for text classification. Our assumption is that the topic information plays an important role in doing the disambiguation given a sentence. For example, the abbreviation of the medical term FISH has two potential senses: general English fish as seafood and the sense of fluorescent in situ hybridization BIBREF17. The former case always goes with the topic of food, allergies while the other one appears in the topic of some examination test reports. In our model, a topic-attention module is applied to add topic information into the final sentence representation. This module calculates the distribution of topic-attention weights on a list of topic vectors. As shown in Figure SECREF5, we took the content vector $c_i$ (from soft-attention BIBREF22) and a topic matrix $T_{topic}=[t_1,t_2,..,t_j]$ (where each $t_i$ is a column vector in the figure and we illustrate four topics) as the inputs to the topic-attention module, and then calculated a weighted summation of the topic vectors. The final sentence representation $r_i$ for the sentence $i$ was calculated as the following:
where $W_{topic}$ and $b _ { topic }$ are trainable parameters, $\beta _{it}$ represents the topic-attention weights. The final sentence representation is denoted as $r_i$, and $[\cdot ,\cdot ]$ means concatenation. Here $s_i$ is the representation of the sentence which includes the topic information. The final sentence representation $r_i$ is the concatenation of $c_i$ and $s_i$, now we have both context-aware sentence representation and topic-related representation. Then we added a fully-connected layer, followed by a Softmax to do classification with cross-entropy loss function.
Topic Matrix To generate the topic matrix $T_{topic}$ as in Equation DISPLAY_FORM9, we propose a convolution-based method to generate topic vectors. We first pre-trained a topic model using the Latent Dirichlet Allocation (LDA) BIBREF25 model on MIMIC-III notes as used by the ELMo model. We set the number of topics to be 50 and ran for 30 iterations. Then we were able to get a list of top $k$ words (we set $k=100$) for each topic. To get the topic vector $t$ for a specific topic:
where $e_j$ (column vector) is the pre-trained Doc2vec word embedding for the top word $j$ from the current topic, and $Conv(\cdot )$ indicates a convolutional layer; $maxpool(\cdot )$ is a max pooling layer. We finally reshaped the output $t$ into a 1-dimension vector. Eventually we collected all topic vectors as the topic matrix $T_{topic}$ in Figure SECREF5.

Evaluation
We did three groups of experiments including two baseline groups and our proposed model. The first group was the TF-IDF features in Section SECREF3 for traditional models. The Naïve Bayesian classifier (NB) has the highest scores among all traditional methods. The second group of experiments used neural models described in Section SECREF3, where LSTM with self attention model (LSTM-self) has competitive results among this group, we choose this model as our base model. Notably, this is the first study that compares and evaluates LSTM-based models on the medical term abbreviation disambiguation task.
tableExperimental Results: we report macro F1 in all the experiments. figureTopic-attention Model
The last group contains the results of our proposed model with different settings. We used Topic Only setting on top of the base model, where we only added the topic-attention layer, and all the word embeddings were from our pre-trained Doc2vec model and were fine-tuned during back propagation. We could observe that compared with the base model, we have an improvement of 7.36% on accuracy and 9.69% on F1 score. Another model (ELMo Only) is to initialize the word embeddings of the base model using the pre-trained ELMo model, and here no topic information was added. Here we have higher scores than Topic Only, and the accuracy and F1 score increased by 9.87% and 12.26% respectively compared with the base model. We then conducted a combination of both(ELMo+Topic), where the word embeddings from the sentences were computed from the pre-trained ELMo model, and the topic representations were from the pre-trained Doc2vec model. We have a remarkable increase of 12.27% on the accuracy and 14.86% on F1 score.
To further compare our proposed topic-attention model and the base model, we report an average of area under the curve(AUC) score among all 30 terms: the base model has an average AUC of 0.7189, and our topic-attention model (ELMo+Topic) achieves an average AUC of 0.8196. We provide a case study in Appendix SECREF9. The results show that the model can benefit from ELMo, which considers contextual features, and the topic-attention module, which brings in topic information. We can conclude that under the few-shot learning setting, our proposed model can better capture the sentence features when only limited training samples are explored in a small-scale dataset.

Conclusion
In this paper, we propose a neural topic-attention model with few-shot learning for medical abbreviation disambiguation task. We also manually cleaned and collected training and testing data for this task, which we release to promote related research in NLP with clinical notes. In addition, we evaluated and compared a comprehensive set of baseline models, some of which had never been applied to the medical term abbreviation disambiguation task. Future work would be to adapt other models like BioBERT or BERT to our proposed topic-attention model. We will also extend the method into other clinical notes tasks such as drug name recognition and ICD-9 code auto-assigning BIBREF26. In addition, other LDA-based approach can be investigated.

Table 1: Examples showing the abbreviation term MR: MR from Note 1 is the abbreviation of ‘mental retardation’. One could easily misinterpret it as ‘mitral regurgitation’which may lead to further imaging studies with subsequent delaying or avoidance of non-urgent interventions due to higher risk of complications. In Note 2, MR means ‘mitral regurgitation’[6].

Figure 2: Comparison of sample numbers. X-axis: all the 93 term-sense pairs (due to space limits, we did not list the pairs in the plot). Y-axis: how many samples in each pair.

Figure 3: The term-sense pair count distribution histogram among 93 pairs.

Table 3: Training and Testing sample distribution among classes in category AC and IM.

Question: What existing dataset is re-examined and corrected for training?
{"answer": [" UM Inventory "], "evidence": ["Training Dataset UM Inventory BIBREF5 is a public dataset created by researchers from the University of Minnesota, containing about 37,500 training samples with 75 abbreviation terms. Existing work reports abbreviation disambiguation results on 50 abbreviation terms BIBREF6, BIBREF5, BIBREF17. However, after carefully reviewing this dataset, we found that it contains many samples where medical professionals disagree: wrong samples and uncategorized samples. Due to these mistakes and flaws of this dataset, we removed the erroneous samples and eventually selected 30 abbreviation terms as our training dataset that can be made public. Among all the abbreviation terms, we have 11,466 samples, and 93 term-sense pairs in total (on average 123.3 samples/term-sense pair and 382.2 samples/term). Some term-sense pairs are very popular with a larger number of training samples but some are not (typically less than 5), we call them rare-sense cases . More details can be found in Appendix SECREF7."]}