You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations

A sentence is typically treated as the minimal syntactic unit used for extracting valuable information from a longer piece of text. However, in written Thai, there are no explicit sentence markers. We proposed a deep learning model for the task of sentence segmentation that includes three main contributions. First, we integrate n-gram embedding as a local representation to capture word groups near sentence boundaries. Second, to focus on the keywords of dependent clauses, we combine the model with a distant representation obtained from self-attention modules. Finally, due to the scarcity of labeled data, for which annotation is difficult and time-consuming, we also investigate and adapt Cross-View Training (CVT) as a semi-supervised learning technique, allowing us to utilize unlabeled data to improve the model representations. In the Thai sentence segmentation experiments, our model reduced the relative error by 7.4% and 10.5% compared with the baseline models on the Orchid and UGWC datasets, respectively. We also applied our model to the task of pronunciation recovery on the IWSLT English dataset. Our model outperformed the prior sequence tagging models, achieving a relative error reduction of 2.5%. Ablation studies revealed that utilizing n-gram presentations was the main contributing factor for Thai, while the semi-supervised training helped the most for English.

Introduction
Automatic summarization, machine translation, question answering, and semantic parsing operations are useful for processing, analyzing, and extracting meaningful information from text. However, when applied to long texts, these tasks usually require some minimal syntactic structure to be identified, such as sentences BIBREF0 , BIBREF1 , BIBREF2 , which always end with a period (“.”) in English BIBREF3 .
However, written Thai does not use an explicit end-of-sentence marker to identify sentence boundaries BIBREF4 . Prior works have adapted traditional machine learning models to predict the beginning position of a sentence. The authors of BIBREF5 , BIBREF6 , BIBREF7 proposed traditional models to determine whether a considered space is a sentence boundary based on the words and their part of speech (POS) near the space. Meanwhile, Zhou N. et al. BIBREF8 considered Thai sentence segmentation as a sequence tagging problem and proposed a CRF-based model with n-gram embedding to predict which word is the sentence boundary. This method achieves the state-of-the-art result for Thai sentence segmentation and achieves greater accuracy than other models by approximately 10% on an Orchid dataset BIBREF9 .
Several deep learning approaches have been applied in various tasks of natural language processing (NLP), including the long short-term memory BIBREF10 , self-attention BIBREF11 , and other models. Huang Z. et al. BIBREF12 proposed a deep learning sequence tagging model called Bi-LSTM-CRF, which integrates a conditional random field (CRF) module to gain the benefit of both deep learning and traditional machine learning approaches. In their experiments, the Bi-LSTM-CRF model achieved an improved level of accuracy in many NLP sequence tagging tasks, such as named entity recognition, POS tagging and chunking.
The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline. This paper makes the following three contributions to improve Bi-LSTM-CRF for sentence segmentation.
First, we propose adding n-gram embedding to Bi-LSTM-CRF due to its success in BIBREF8 and BIBREF12 . By including n-gram embedding, the model can capitalize on both approaches. First, the model gains the ability to extract past and future input features and sentence level tag information from Bi-LSTM-CRF; moreover, with the n-gram addition, it can also extract a local representation from n-gram embedding, which helps in capturing word groups that exist near sentence boundary. Although Jacovi A. et al. BIBREF13 reported that a convolutional neural network (CNN) can be used as an n-gram detector to capture local features, we chose n-gram embedding over a CNN due to its better accuracy, as will be shown in Section SECREF8 .
Second, we propose adding incorporative distant representation into the model via a self-attention mechanism, which can focus on the keywords of dependent clauses that are far from the considered word. Self-attention has been used in many recent state-of-the-art models, most notably the transformer BIBREF11 and BERT BIBREF14 . BERT has outperformed Bi-LSTM on numerous tasks, including question answering and language inference. Therefore, we choose to use self-attention modules to extract distant representations along with local representations to improve model accuracy.
Third, we also apply semi-supervised learning BIBREF15 , allowing us to employ unlimited amounts of unlabeled data, which is particularly important for low-resource languages such as Thai, for which annotation is costly and time-consuming. Many semi-supervised learning approaches have been proposed in the computer vision BIBREF16 , BIBREF17 and natural language processing BIBREF18 , BIBREF19 , BIBREF20 fields. Our choice for semi-supervised learning to enhance model representation is Cross-View Training (CVT) BIBREF20 . Clark K. et al. BIBREF20 claims that CVT can improve the representation layers of the model, which is our goal. However, CVT was not designed to be integrated with self-attention and CRF modules; consequently, we provide a modified version of CVT in this work.
Based on the above three contributions, we pursue two main experiments. The first experiment was conducted on two Thai datasets, Orchid and UGWC BIBREF21 , to evaluate our Thai sentence segmentation model. In this case, our model achieves F1 scores of 92.5% and 88.9% on Orchid and UGWC, respectively, and it outperforms all the baseline models. The second experiment was executed on the IWSLT dataset BIBREF22 and involves an English-language punctuation restoration task. This experiment demonstrates that our model is generalizable to different languages. Our model, which does not require pretrained word vectors, improved the overall F1 score by 0.9% compared to the baselines, including a model that uses pretrained word vectors.
There are five sections in the remainder of this paper. Section SECREF2 reviews the related works on Thai sentence segmentation, English punctuation restoration and introduces the original CVT. Section SECREF3 describes the proposed model architecture and the integration of cross-view training. The datasets, implementation process and evaluation metrics are explained in Section SECREF4 . The results of the experiments are discussed in Section SECREF5 . Finally, Section SECREF6 concludes the paper.

English punctuation restoration
Most languages use a symbol that functions as a sentence boundary; however, a few do not use sentence markers including Thai, Lao and Myanmar. Thus, few studies have investigated sentence segmentation in raw text. However, studies on sentence segmentation, which is sometimes called sentence boundary detection, are still found in the speech recognition field BIBREF25 . The typical input to speech recognition model is simply a stream of words. If two sentences are spoken back to back, by default, a recognition engine will treat it as one sentence. Thus, sentence boundary detection is also considered a punctuation restoration task in speech recognition because when the model attempts to restores the period in the text, the sentence boundary position will also be defined.
Punctuation restoration not only provides a minimal syntactic structure for natural language processing, similar to sentence boundary detection but also dramatically improves the readability of transcripts. Therefore, punctuation restoration has been extensively studied. Many approaches have been proposed for punctuation restoration that use different features, such as audio and textual features. Moreover, punctuation restoration is also considered to be a different machine learning problem, such as word sequence tagging and machine translation.
A combination of audio and textual features were utilized in BIBREF26 , BIBREF27 , BIBREF28 to predict and restore punctuation, including pitch, intensity and pause duration, between words. We ignore these features in our experiment because our main task—Thai sentence segmentation— does not include audio features.
Focusing only on textual features, there are two main approaches, namely, word sequence tagging and machine translation. For the machine translation approach, punctuation is treated as just another type of token that needs to be recovered and included in the output. The methods in BIBREF29 , BIBREF30 , BIBREF31 restore punctuation by translating from unpunctuated text to punctuated text. However, our main task, sentence segmentation, is an upstream task in text processing, unlike punctuation restoration, which is considered a downstream task. Therefore, the task needs to operate rapidly; consequently, we focus only on the sequence tagging model, which is less complex than the machine translation model.
In addition to those machine translation tasks, both traditional approaches and deep learning approaches must solve a word sequence tagging problem. Of the traditional approaches, contextual features around the considered word were used to predict following punctuation in the n-gram BIBREF32 and CRF model approaches BIBREF33 , BIBREF34 . Meanwhile, in the deep learning approaches, a deep convolutional neural network BIBREF35 , T-LSTM (Textual-LSTM) BIBREF26 and a bidirectional LSTM model with an attention mechanism, called T-BRNN BIBREF36 , have been adopted to predict a punctuation sequence from the word sequence. T-BRNN BIBREF36 was proposed to solve the task as a word-sequence tagging problem, and it is currently the best model that uses the word sequence tagging approach. Tilk O. et al. BIBREF36 also proposed a variant named T-BRNN-pre, which integrates pretrained word vectors to improve the accuracy.
To demonstrate that our model is generalizable to other languages, we compare it with other punctuation restoration models, including T-LSTM, T-BRNN, and T-BRNN-pre. These models adopt a word sequence tagging approach and do not utilize any prosodic or audio features.

Effect of distant representation
The effect of this contribution can be found by comparing the model that integrates the distant representation and the model that does not. The model with distant features integrated is represented as INLINEFORM0 (row (g)) in both tables. In this case, the distant representation is composed of the self-attention modules in both the low- and high-level modules, as shown in Fig. .
From the combination of local and distant representation, the results in tab:thairesult,tab:engresult show that the distant feature improves the accuracy of the model on all datasets compared to the model with no distant representation. The F1 scores of the sentence segmentation models improved slightly, from 92.4% and 88.7% (row (f)) to 92.5% and 88.8% (row (g)) on the Orchid and UGWC datasets, respectively. For the IWSLT dataset, the distant feature can recover the overall F1 score of punctuation restoration, which is degraded by the n-gram embedding; it improves from 63.6% (row (f)) to 64.5% (row (g)). The reason is that the self-attention modules focus selectively on certain parts of the passage. Thus, the model focuses on the initial words of the dependent clauses, which helps in classifying which type of punctuation should be restored. An example is shown in Fig. : the model with distant representation classifies the punctuation after ”her” as a ”COMMA” because ”Before” is the word that indicates the dependent clause. Meanwhile, the model without distant representation predicts the punctuation as a ”PERIOD” because there is no self-attention module; therefore, it does not focus on the word ”Before”. Overall, the model that includes both local and distant representation can generally be used for both sentence segmentation and punctuation restoration, and it outperforms both baseline models.
result/cherrypickdistant

Comparison with baseline models
For the Thai sentence segmentation task, our model is superior to all the baselines on both Thai sentence segmentation datasets, as shown in Table TABREF45 . On the Orchid dataset, the supervised model that includes both local and distant representation was adopted for comparison to the baseline model. Our model improves the F1 score achieved by CRF-ngram, which is the state-of-the-art model for Thai sentence segmentation in Orchid, from 91.9% (row (d)) to 92.5% (row (g)). Meanwhile, in the UGWC dataset, our CVT model (row (h)) achieves an F1 score of 88.9%, which is higher than the F1 score of both the baselines (CRF-ngram and Bi-LSTM-CRF (rows d and e, respectively)). Thus, our model is now the state-of-the-art model for Thai sentence segmentation on both the Orchid and UGWC datasets.
Our model outperforms all the sequence tagging models. T-BRNN-pre (row (c)) is the current state-of-the-art model, as shown in Table TABREF47 . The CVT model improves the overall F1 score from the 64.4% of T-BRNN-pre to 65.3% (row (h)), despite the fact that T-BRNN-pre integrates a pretrained word vector. Moreover, our model also achieves a 2-class F1 score 1.3% higher than that of Bi-LSTM-CRF (row (e)).

Comparison of CNN and n-gram models for local representation
Jacovi A. et al. BIBREF13 proposed that a CNN can be used as an n-gram detector to capture local text features. Therefore, we also performed an experiment to compare a CNN and n-gram embedded as local structures. The results in Table TABREF56 show that the model using the embedded n-gram yields greater improvement than the one using an embedded CNN on the Orchid and UGWC datasets.

Statistical Tests for Thai sentence segmentation
To prove the significance of the model improvements, we compared the cross-validation results using paired t-tests to obtain the p-values, which are shown in Table TABREF57 for the Orchid dataset and Table TABREF58 for the UGWC dataset.

Figure 5: An example of sentence boundary prediction by a normal Bi-LSTM-CRF and by the model with local representation (+local). Here, sb indicates that the word is predicted as the sentence boundary.

Question: Which deep learning architecture do they use for sentence segmentation?
{"answer": ["Bi-LSTM-CRF"], "evidence": ["The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline. This paper makes the following three contributions to improve Bi-LSTM-CRF for sentence segmentation."]}