You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Extracting clinical concepts from user queries

Clinical concept extraction often begins with clinical Named Entity Recognition (NER). Often trained on annotated clinical notes, clinical NER models tend to struggle with tagging clinical entities in user queries because of the structural differences between clinical notes and user queries. User queries, unlike clinical notes, are often ungrammatical and incoherent. In many cases, user queries are compounded of multiple clinical entities, without comma or conjunction words separating them. By using as dataset a mixture of annotated clinical notes and synthesized user queries, we adapt a clinical NER model based on the BiLSTM-CRF architecture for tagging clinical entities in user queries. Our contribution are the following: 1) We found that when trained on a mixture of synthesized user queries and clinical notes, the NER model performs better on both user queries and clinical notes. 2) We provide an end-to-end and easy-to-implement framework for clinical concept extraction from user queries.

Framework ::: Entity Tagging
We use the same BiLSTM-CRF model proposed by Huang et al. BIBREF2. An illustration of the architecture is shown in Figure FIGREF6 . Given a sequence (or sentence) of n tokens, $r = (w_1, w_2,..., w_n)$, we use a fine-tuned ELMo model to generate contextual word embeddings for all the tokens in the sentence, where a token refers to a word or punctuation. We denote the ELMo embedding, $x$, for a token $w$ in the sentence $r$ by $x = ELMo(w|r)$. The notation and the procedure described here can be adopted for Flair embeddings or other embeddings. Now, given a sequence of tokens in ELMo embeddings, $X =(x_1, x_2, ..., x_n)$, the BiLSTM layer generates a matrix of scores, $P(\theta )$ of size $n \times k$, where $k$ is the number of tag types, and $\theta $ is the parameters of the BiLSTM. To simplify notation, we will omit the $\theta $ and write $P$. Then, $P_{i,j}$ denotes the score of the token, $x_i$, being assigned to the $j$th tag. Since certain constraints may exist in the transition between tags, an "O" tag should not be followed by an "I" tag, for example, a transition matrix, $A$, of dimension $(k+2)\times (k+2)$, is initialized to model the constraints. The learnable parameters, $A_{i,j}$, represent the probability of the $j$th tag follows the $i$th tag in a sequence. For example, if we index the tags by: 1:“B”, 2:“I”, and 3:“O”, then $A_{1,3}$ would be the probability that an “O” tag follows a “B” tag. A beginning transition and an end transition are inserted in $A$ and hence $A$ is of dimension $(k+2)\times (k+2)$.
Given a sequence of tags, $Y=(y_1,y_2,...,y_n)$, where each $y_i$, $1\le i \le n$, corresponds to an index of the tags, the score of the sequence is then given by
The probability of the sequence of tags is then calculated by a softmax,
where $\lbrace Y_x\rbrace $ denotes the set of all possible tag sequences. During training, the objective function is to maximize $\log (P(Y|X))$ by adjusting $A$ and $P$.

Experiments ::: Data
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. The synthesized user queries are generated using the aforementioned dermatology glossary. Tagged sentences are extracted from the clinical notes. Sentences with no clinical entity present are ignored. 22,489 tagged sentences are extracted from the clinical notes. We will refer to these tagged sentences interchangeably as the i2b2 data. The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1. The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order. When combining the clinical terms, we attach the BIO tags to their constituent words. The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio. Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively. This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same.
The reason for combining the two data is their drastic structural difference (See figure FIGREF16 for an example). Previously, when trained on the i2b2 data only, the BiLSTM-CRF model was not able to segment clinical entities at the correct boundary. It would fail to recognize the user query in Figure FIGREF16(a) as four separate entities. On the other hand, if the model was trained solely on the synthesized user queries, we could imagine that it would fail miserably on any queries that resemble the sentence in Figure FIGREF16(b) because the model would have never seen an “O” tag in the training data. Therefore, it is necessary to use the hybrid training data containing both the i2b2 data and the synthesized user queries.
To make the hybrid training data, we need to unify the tags. Recall that in Section SECREF1 we point out that the tags are different for the different tasks and datasets. Since we use custom tags for dermatology glossary in our problem, we would need to convert the tags used in 2010 i2b2/VA. But this would be an infeasible job as we need experts to manually do that. An alternative is to avoid distinguishing the tag types and label all tags under the generic BIO tags.

Experiments ::: Results
From the hyperparameter tuning we found that the best combination was
embeddings: “ELMo on pubmed”,
hidden_size: 256,
learning_rate: 0.05,
mini_batch_size: 32.
With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0.995$ on synthesized queries and $0.948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0.441$ on synthesized queries and $0.927$ on clinical notes (See Table TABREF23).
Since there was no ground truth available for the retrieved terms, we randomly picked a few samples to assess its performance. Some example outputs of our complete framework on real user queries are shown in Figure FIGREF24. For example, from the figure we see that the query "child fever double vision dizzy" was correctly tagged with four entities: "child", "fever", "double vision", and "dizzy". A list of terms from our glossary was matched to each entity. In real world application, the lists of terms will be presented to the user as the retrieval results to their queries.

Discussion
In most real user queries we sampled, the entities were tagged at the correct boundary and the tagging was complete (such as the ones shown in Figure FIGREF24). Only on a few user queries the tagging was controversial. For example, the query “Erythematous blanching round, oval patches on torso, extremities” was tagged as “Erythematous blanching” and “oval patches on torso”. The entity “extremities” was missing. The segmentation was also not correct. A more appropriate tagging would be “Erythematous blanching round, oval patches”, “torso”, and “extremities”. The tagging could be further improved by synthesizing more realistic user queries. Recall that the synthesized user queries were created by randomly combining terminologies from the dermatology glossary, which, while providing data that helped the model learn entity segmentation, did not reflect the co-occurrence information in real user queries. For example, there could be two clinical entities that often co-occur or never co-occur in a user query. But since the synthesized user queries we used combined terms randomly, the co-occurrence information was thus missing.
The final retrieval results of our framework were not evaluated quantitatively in terms of recall and precision, due the the lack of ground truth. When ground truth becomes available, we will be able to evaluate our framework more thoroughly. Recently, a fine-tuned BERT model in the medical domain called BioBERT BIBREF17 has attracted some attention in the medical NLP domain. We could experiment with BioBERT embeddings in the future. We could also include query expansion technique for term matching. When finding candidate terms for an entity, our first step was still based on string matching. Given that there might be multiple entities that could be matched to the same term, it could be hard to include all these entities in the glossary and hard to match terms to these entities.

Conclusion
In this project, we tackle the problem of extracting clinical concepts from user queries on medical search engines. By training a BiLSTM-CRF model on a hybrid data consisting of synthesized user queries and sentences from clinical note, we adopt a CE framework for clinical user queries with minimal effort spent on annotating user queries. We find that the hybrid data enables the NER model perform better on both tagging the user queries and the clinical note sentences. Furthermore, our framework is built on an easy-to-use deep learning NLP Python library, which lends it more prospective value to various online medical applications that employ medical search engines.

Acknowledgment
This paper results from a technical report of a project the authors have worked on with visualDx, a healthcare informatics company that provides web-based clinical decision support system. The authors would like to thank visualDx for providing them the opportunity to work on such an exciting project. In particular, the authors would like to thank Roy Robinson, the Vice President of Technology and Medical Informatics at visualDx, for providing the synthesized user queries, as well as preliminary feedback on the performance of our framework.

Figure 1: The BiLSTM-CRF model for tagging entities in user queries. A user query, "boy double vision dizziness", enters at the bottom level. The word embeddings, shown as arrays of red dots which each represents a real number, are generated by the fine-tuned ELMo model and then fed into a left-to-right LSTM and a right-to-left LSTM. The scores output by the two LSTMs are concatenated to form the word features for the CRF layer. The CRF layer finds the most likely BIO tags for each word based on the word features and tag sequence constraints between tags. The output is the clinical entities in the query, "double vision" and "dizziness". Figure after Jurafsky & Martin [16].

Figure 2: First step in the computation for the similarity score between E and C. A matrix of similarity scores between each word in C and each word in E is created. Then, for each word in C, get the maximum similarity score, max(~si), 1 ≤ i ≤ m, where m is the number of words in C.

Figure 3: (a) and (b) are equivalent in terms of what they describe. However, (a) is what a typical user query looks like and (b) is what a sentence in a clinical note looks like. Clinical entities, regardless of their type, are colored red. The tags for each token (word or punctuation) are labeled below the token. Notice that (a) is incoherent as a sentence and is several entities compounded together without any conjunction words or punctuation. (b) on the other hand, is cohesive. It contains a proportion of “O” tags.

Table 1: Micro F1 scores of the two NER models two test sets.

Question: where did they obtain the annotated clinical notes from?
{"answer": ["clinical notes from the CE task in 2010 i2b2/VA"], "evidence": ["Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. The synthesized user queries are generated using the aforementioned dermatology glossary. Tagged sentences are extracted from the clinical notes. Sentences with no clinical entity present are ignored. 22,489 tagged sentences are extracted from the clinical notes. We will refer to these tagged sentences interchangeably as the i2b2 data. The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1. The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order. When combining the clinical terms, we attach the BIO tags to their constituent words. The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio. Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively. This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same."]}