You will be presented with the abstract, introduction, and discussion section from a biomedical research article. Your task is to create a summary that a layperson can understand, capturing the essence of the research article. Keep essential scientific terms, but ensure that the language remains clear and the concepts are explained in an uncomplicated manner.

Title: Scaling up data curation using deep learning: An application to literature triage in genomic variation resources}
Article:
Abstract:
Manually curating biomedical knowledge from publications is necessary to build a knowledge based service that provides highly precise and organized information to users. The process of retrieving relevant publications for curation, which is also known as document triage, is usually carried out by querying and reading articles in PubMed. However, this query-based method often obtains unsatisfactory precision and recall on the retrieved results, and it is difficult to manually generate optimal queries. To address this, we propose a machine-learning assisted triage method. We collect previously curated publications from two databases UniProtKB/Swiss-Prot and the NHGRI-EBI GWAS Catalog, and used them as a gold-standard dataset for training deep learning models based on convolutional neural networks. We then use the trained models to classify and rank new publications for curation. For evaluation, we apply our method to the real-world manual curation process of UniProtKB/Swiss-Prot and the GWAS Catalog. We demonstrate that our machine-assisted triage method outperforms the current query-based triage methods, improves efficiency, and enriches curated content. Our method achieves a precision 1. 81 and 2. 99 times higher than that obtained by the current query-based triage methods of UniProtKB/Swiss-Prot and the GWAS Catalog, respectively, without compromising recall. In fact, our method retrieves many additional relevant publications that the query-based method of UniProtKB/Swiss-Prot could not find. As these results show, our machine learning-based method can make the triage process more efficient and is being implemented in production so that human curators can focus on more challenging tasks to improve the quality of knowledge bases.
Introduction:
The question of how genetic variation in a population influences phenotypic variation is of major importance in biology. Naturally occurring genetic variants, both rare and common, can provide insight into disease mechanism and protein function. This understanding, coupled with the recent explosion in next-generation sequencing, has meant a dramatic increase in publications on the subject. This also means that it is now impossible for individual researchers to collect and collate all the variant information that may be relevant to them. To assist researchers, variant information in publications is selected, summarized, organized, and stored in a searchable form in knowledge bases such as UniProtKB/Swiss-Prot[1, 2] and the NHGRI-EBI GWAS Catalog[3] for easier access and greater usability. However, such knowledge bases require domain experts to collect and manually curate high quality information from the literature[4], a highly time consuming and therefore costly process. In addition, these knowledge bases have their own focus and collect information from publications according to their own specific guidelines. As shown in the study of Baumgartner et al., the manual curation of information for genomic databases is, for the most part, not scalable due to the fast-growing number of publications[5]. To overcome this scalability issue, automated or semi-automated methods can be used[6]. For this purpose, text mining and machine learning tools that support information extraction and annotation have been developed[7–14]. Poux et al.[15] demonstrated that manual curation can be more efficient and scalable with the proper text-mining services and techniques. As a typical first step in the manual curation process, the document triage process involves identifying publications of interest[16, 17]. On average, three thousand biomedical papers are indexed in PubMed every day.( In 2016, more than 1. 1 million publications were indexed in PubMed). Triage is then required to select a subset of relevant publications for curation. As Hirschman et al.[16] explained, triage is usually carried out using pre-defined queries in the PubMed database. The queries are generated using general terms related to topics such as “GWAS” and “Drug screening” or a list of entities( e. g., onco-gene list[18]). Publication date or publication type( e. g., review, book) are also included for a query as needed. However, a triage process using pre-defined queries has several major limitations. Using general topic terms for a query may be futile because publications on a certain topic may not contain the specific terms. For example, if a user wishes to find publications on cancer-related genes, the query should contain specific terms such as “HER2, ” “Carcinoma, ” “Tumor, ” and so on, rather than “cancer-related gene” because many publications on cancer-related genes do not contain the general term “cancer-related gene. ” Furthermore, the topic words in a publication may provide only background information and may not be on the main topic of the publication. Also, longstanding databases have their own complex and detailed guidelines for manual curation and it can be difficult to create a PubMed query that fulfills their conditions[19]. For these reasons, query-based triage is limited in retrieving highly precise and complete set of relevant publications for further use, not to mention the process itself is also very labor intensive and time consuming. To overcome the limitations of the query-based triage and manual curation processes, machine learning-assisted curation research studies have been conducted. Poux et al.[15] used PubTator, a web-based curation support system that assists users in annotating publications in PubMed. Curators can read publications that are highlighted and pre-annotated by automated named-entity recognition tools. Curators can also easily upload and generate their curation collection and save their results with a simple mouse click. In their study, Poux et al. selected only thirteen journals from which to collect protein function information, and ranked the publications of the journals by the number of proteins mentioned in the text. However, other than the thirteen selected journals, there are many journals that have published papers on protein function. Also, prioritizing the publications by the number of proteins may not be the best method because some papers include valuable information on a small number of proteins. Almeida et al.[20] built a method called mycoSORT using support vector machine( SVM), Naïve Bayes and Logistic Model Trees on the triage task for the mycoCLAP database[21]; however, this required several text preprocessing steps and an extensive feature extraction process which are data/domain dependent. Because of these dependencies, their method is not directly applicable to other types of databases. Their feature extraction process is time consuming, labor intensive, and requires domain knowledge from human experts[6]. In recent years, newly proposed deep learning-based text mining methods have started outperforming traditional machine learning-based methods in various tasks[22–28]. In addition, these deep neural network models do not require intensive feature engineering by domain experts; hence, they can also be easily generalized to other tasks with datasets in different domains. In this paper, we propose to employ convolutional neural network( CNN), a class of deep, feed-forward artificial neural networks, for the identification of publications relevant for variant curation. By comparing the results of our method with those of mycoSORT, the method proposed by Almeida et al.[20], we demonstrate how our deep learning-based classifier performs better than traditional machine learning classifiers, even without feature engineering. For assessing the utility of our approach, we applied our method to two external knowledge bases( UniProtKB/Swiss-Prot and the GWAS Catalog) in real-world circumstances. We compare the performance of our proposed method with that of each of the knowledge base-specific query-based methods to demonstrate that our method can greatly improve the efficiency of the document triage step in these two databases. While the usage of CNNs to classify text documents is not new, the application of deep learning to speed up the real-world triage process for biomedical literature is, to the best of our knowledge.
Discussion:
Our results demonstrate that our machine learning triage method works well on different datasets in different settings. Both the UniProtKB/Swiss-Prot and the GWAS Catalog manual curation teams confirmed that our method achieved higher precision than the previous query-based triage methods without significantly compromising recall. In the UniProtKB/Swiss-Prot result, our method could find many relevant publications that the query-based method could not find. In the GWAS Catalog case, our method significantly improved the efficiency of the curation process by reducing the number of papers that required review. Both results show that our method can replace the traditional query-based triage methods of manually curated databases. However, even though our method works well and improves the efficiency of the manual curation process, it is difficult for our method to include new types of articles based on new interests. For example, the GWAS Catalog, which has collected publications related to only genome-wide array-based association studies so far, might find it difficult to find whole-exome sequencing publications since our classifier is trained on only the publications related to array-genotyped GWAS and there are no whole-exome sequencing papers in the training set. This is known as the “filter bubble” problem of personalized search, social media and news recommender systems[33]. For example, based on the behavior of a user, personalized news recommendation algorithms suggest news articles on certain topics as results. The user selects some of the results recommended by the algorithm, and the algorithm learns from those results again to recommend similar results. After this process is repeated a few times, the recommendation algorithm narrows down the recommendation results based on the interests of the user, which makes it more difficult for users to view new information other than the results recommended by the algorithm. To solve this problem, the curators need to manually find new topic publications as a separate activity. However, the curators and our team both agreed that our classification and ranking methods would save curators considerable time in standard triage. Thus they have extra time to manually find new subjects, using new queries in PubMed or using their intuition, which can solve the filter bubble problem. The other limitation of our method comes from the “black box” property of the deep neural network models[34, 35]. In the utility assessment of the GWAS Catalog section, our method did not retrieve a publication( PMID 29313844) while the query-based method did. The confidence score of the publication obtained by our method was 0. 451 which is slightly lower than the default threshold of the classifiers( 0. 5). We investigated why the publication was scored low even though it contains GWAS relevant terms, such as “p-value” and “GWAS, ” however we could not clearly understand why this publication was not scored highly enough because deep neural networks are “black boxes. ” With other traditional machine learning models such as decision tree, k-nearest neighbor and logistic regression, it is possible to see how the classification of each case is made by looking at the decision rules, neighbors, or weight of features, respectively; however, it is not possible in deep neural network models. This example shows the limitation of deep learning methods. The hardest part of designing our method was that there is no negative gold-standard dataset available for training the classifiers. We could not collect any of the negative data for training / testing our method as we were able to for mycoSet[20]. We constructed negative datasets using randomly collected variant containing publications using tmVar after filtering out the positive publications; however, there are chances that some positive examples are included in the negative dataset. Using tmVar for generating negative dataset may also cause some bias in the classification results, because all the negative publications are tmVar positive. Our method still over-performed compared to the query-based triage methods. If we can collect enough gold-standard negative publications, our method may perform better. Since PubTator is now used as a manual literature triage framework for several knowledge bases such as UniProtKB/Swiss-Prot and the GWAS Catalog, we can easily collect negative publications that curators mark as irrelevant and we expect to use them as negative gold-standard data for training the classifiers in the future. In this research, we focus on the triage of variant-related knowledge bases; however, our method can be applied to any knowledge base that relies on manual curation and to a triage process if it has a sufficient number of documents for training the classifier. We also believe it would be interesting to use our method for knowledge bases of biomedical relations such as protein-protein interactions and drug-drug interactions. Although we did not use any preprocessing in our method, named-entity tagging might be helpful in these tasks.
As the volume of literature on genomic variants continues to grow at an increasing rate , it is becoming more difficult for a curator of a variant knowledge base to keep up with and curate all the published papers . Here , we suggest a deep learning-based literature triage method for genomic variation resources . Our method achieves state-of-the-art performance on the triage task . Moreover , our model does not require any laborious preprocessing or feature engineering steps , which are required for traditional machine learning triage methods . We applied our method to the literature triage process of UniProtKB/Swiss-Prot and the NHGRI-EBI GWAS Catalog for genomic variation by collaborating with the database curators . Both the manual curation teams confirmed that our method achieved higher precision than their previous query-based triage methods without compromising recall . Both results show that our method is more efficient and can replace the traditional query-based triage methods of manually curated databases . Our method can give human curators more time to focus on more challenging tasks such as actual curation as well as the discovery of novel papers/experimental techniques to consider for inclusion .