You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Extractive Summarization of EHR Discharge Notes

Patient summarization is essential for clinicians to provide coordinated care and practice effective communication. Automated summarization has the potential to save time, standardize notes, aid clinical decision making, and reduce medical errors. Here we provide an upper bound on extractive summarization of discharge notes and develop an LSTM model to sequentially label topics of history of present illness notes. We achieve an F1 score of 0.876, which indicates that this model can be employed to create a dataset for evaluation of extractive summarization methods.

Related Work
In the broader field of summarization, automization was meant to standardize output while also saving time and effort. Pioneering strategies in summarization started by extracting "significant" sentences in the whole corpus to build an abstract where "significant" sentences were defined by the number of frequently occurring words BIBREF6 . These initial methods did not consider word meaning or syntax at either the sentence or paragraph level, which made them crude at best. More advanced extractive heuristics like topic modeling BIBREF7 , cue word dictionary approaches BIBREF8 , and title methods BIBREF9 for scoring content in a sentence followed soon after. For example, topic modeling extends initial frequency methods by assigning topics scores by frequency of topic signatures, clustering sentences with similar topics, and finally extracting the centroid sentence, which is considered the most representative sentence BIBREF10 . Recently, abstractive summarization approaches using sequence-to-sequence methods have been developed to generate new text that synthesizes original text BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ; however, the field of abstractive summarization is quite young.
Existing approaches within the field of electronic health record summarization have largely been extractive and indicative, meaning that summaries point to important pieces in the original text rather than replacing the original text altogether. Few approaches have been deployed in practice, and even fewer have demonstrated impact on quality of care and outcomes BIBREF15 . Summarization strategies have ranged from extraction of “relevant” sentences from the original text to form the summary BIBREF16 , topic modeling of EHR notes using Latent Dirichlet allocation (LDA) or bayesian networks BIBREF15 , and knowledge based heuristic systems BIBREF17 . To our knowledge, there is no literature to date on extractive or abstractive EHR summarization using neural networks.

Data
MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012. The database contains all of the notes associated with each patient's time spent in the ICU as well as 55,177 discharge reports and 4,475 discharge addendums for 41,127 distinct patients. Only the original discharge reports were included in our analyses. Each discharge summary was divided into sections (Date of Birth, Sex, Chief Complaint, Major Surgical or Invasive Procedure, History of Present Illness, etc.) using a regular expression.

Upper Bound on Summarization
Extractive summarization of discharge summaries relies on the assumption that the information in the discharge summary is documented elsewhere in the rest of the patient's notes. However, sometimes clinicians will document information in the discharge summary that may have been discussed throughout the hospital visit, but was never documented in the EHR. Thus, our first aim was to determine the upper bound of extractive summarization.
For each patient, we compared the text of the discharge summary to the remaining notes for the patient's current admission as well as their entire medical record. Concept Unique Identifiers (CUIs) from the Unified Medical Language System (UMLS) were compared in order to assess whether clinically relevant concepts in the discharge summary could be located in the remaining notes BIBREF18 . CUIs were extracted using Apache cTAKES BIBREF19 and filtered by removing the CUIs that are already subsumed by a longer spanning CUI. For example, CUIs for "head" and "ache" were removed if a CUI existed for "head ache" in order to extract the most clinically relevant CUIs.
In order to understand which sections of the discharge summaries would be the easiest or most difficult to summarize, we performed the same CUI overlap comparison for the chief complaint, major surgical or invasive procedure, discharge medication, and history of present illness sections of the discharge note separately. We calculated which fraction of the CUIs in each section were located in the rest of the patient's note for a specific hospital stay. We also calculated what percent of the genders recorded in the discharge summary were also recorded in the structured data for the patient.
For each of the 55,177 discharge summary reports in the MIMIC database, we calculated what fraction of the CUIs in the discharge summary could be found in the remaining notes for the patient's current admission ( INLINEFORM0 ) and in their entire longitudinal medical record ( INLINEFORM1 ). Table TABREF13 shows the CUI recall averaged across all discharge summaries by both subject_id and hadm_id. The low recall suggests that clinicians may incorporate information in the discharge note that had not been previously documented in the EHR. Figure FIGREF11 plots the relationship between the number of non-discharge notes for each patient and the CUI recall (top) and the number of total CUIs in non-discharge notes and the CUI recall (middle). The number of CUIs is a proxy for the length of the notes, and as expected, the CUI recall tends to be higher in patients with more and longer notes. The bottom panel in Figure FIGREF11 demonstrates that recall is not correlated with the patient's length of stay outside the ICU, which indicates that our upper bound calculation is not severely impacted by only having access to the patient's notes from their stay in the ICU.
Finally, Table TABREF14 shows the recall for the sex, chief complaint, procedure, discharge medication, and HPI discharge summary sections averaged across all the discharge summaries. The procedure section has the highest recall of 0.807, which is understandable because procedures undergone during an inpatient stay are most likely to be documented in an EHR. The recall for each of these five sections is much higher than the overall recall in Table TABREF13 , suggesting that extractive summarization may be easier for some sections of the discharge note.
Overall, this upper bound analysis suggests that we may not be able to recreate a discharge summary with extractive summarization alone. While CUI comparison allows for comparing medically relevant concepts, cTAKES's CUI labelling process is not perfect, and further work, perhaps through sophisticated regular expressions, is needed to define the limits of extractive summarization.

Conclusion
In this paper we developed a CUI-based upper bound on extractive summarization of discharge summaries and presented a NN architecture that jointly classifies words in history of present illness notes. We demonstrate that our model can achieve excellent performance on a small dataset with known heterogeneity among annotators. This model can be applied to the 55,000 discharge summaries in MIMIC to create a dataset for evaluation of extractive summarization methods.

Acknowledgments
We would like to thank our annotators, Andrew Goldberg, Laurie Alsentzer, Elaine Goldberg, Andy Alsentzer, Grace Lo, and Josh Donis. We would also like to acknowledge Pete Szolovits for his guidance and for providing the pretrained word embeddings and Tristan Naumann for providing the MIMIC CUIs.

Table 1. HPI Categories and Annotation Instructions

Figure 1. LSTM Model for sequential HPI word classification. Xi: token i, Z1 to Zl: characters of Xi, t: word embeddings, c: character embeddings, e1:k: hybrid token embeddings, h1:j : contextual representation embedings, s1:m

Figure 2. Relationship between the number of non-discharge notes (top), number of non-discharge CUIs (middle), or the number of hours the patient spent outside the ICU (bottom) and the recall for each discharge summary.

Table 3. Average Recall for five sections of the discharge summary. Recall for each patient’s sex was calculated by examining the structured data for the patient’s current admission, and recall for the remaining sections was calculated by comparing CUI overlap between the section and the remaining notes for the current admission.

Table 2. Discharge CUI recall for each patient’s current hospital admission (by hadm id) and their entire EHR (by subject id), averaged across all discharge summaries

Figure 3. Confusion Matrix for the Learned Word Embedding model on the test set. True labels are on the x-axis, and predicted labels are on the y-axis.

Table 4. F1-Scores on the development set under various model architectures. The first model in each section is the same model described in the methods section. Top performing models in each section are in bold.

Question: what datasets were used?
{"answer": ["MIMIC-III"], "evidence": ["MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012. The database contains all of the notes associated with each patient\u0027s time spent in the ICU as well as 55,177 discharge reports and 4,475 discharge addendums for 41,127 distinct patients. Only the original discharge reports were included in our analyses. Each discharge summary was divided into sections (Date of Birth, Sex, Chief Complaint, Major Surgical or Invasive Procedure, History of Present Illness, etc.) using a regular expression."]}