You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Deep LSTM for Large Vocabulary Continuous Speech Recognition

Recurrent neural networks (RNNs), especially long short-term memory (LSTM) RNNs, are effective network for sequential task like speech recognition. Deeper LSTM models perform well on large vocabulary continuous speech recognition, because of their impressive learning ability. However, it is more difficult to train a deeper network. We introduce a training framework with layer-wise training and exponential moving average methods for deeper LSTM models. It is a competitive framework that LSTM models of more than 7 layers are successfully trained on Shenma voice search data in Mandarin and they outperform the deep LSTM models trained by conventional approach. Moreover, in order for online streaming speech recognition applications, the shallow model with low real time factor is distilled from the very deep model. The recognition accuracy have little loss in the distillation process. Therefore, the model trained with the proposed training framework reduces relative 14\% character error rate, compared to original model which has the similar real-time capability. Furthermore, the novel transfer learning strategy with segmental Minimum Bayes-Risk is also introduced in the framework. The strategy makes it possible that training with only a small part of dataset could outperform full dataset training from the beginning.

Introduction
Recently, deep neural network has been widely employed in various recognition tasks. Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0 . Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 .
Training neural network becomes more challenge when it goes deep. A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5 . The discriminating features of linear classifier is the hidden units of a intermediate layer. For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers. Therefore, the tool shows the difficulty of deep neural model training visually.
Layer-wise pre-training is a successful method to train very deep neural networks BIBREF6 . The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8 . But the deeper network which is initialized with a shallower trained network could converge well.
The size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably. Therefore, parallel training with multi-GPUs is more suitable for LVCSR system. Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure. Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10 . It can many times speed up the training time without decreasing the accuracy. Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11 . Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement.
In order to further improve the performance of deep neural network with parallel training, several methods are proposed. Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training. Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging. It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14 .
Moving averaged (MA) approaches are also proposed for parallel training. It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15 . One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16 . Exponential moving average (EMA) is proposed as a non-interference method BIBREF17 . EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process. EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER). It is also easy to implement in existing parallel training systems.
Frame stacking can also speed up the training time BIBREF18 . The super frame is stacked by several regular frames, and it contains the information of them. Thus, the network can see multiple frames at a time, as the super frame is new input. Frame stacking can also lead to faster decoding.
For streaming voice search service, it needs to display intermediate recognition results while users are still speaking. As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one. High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably. The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network. Deep model outperforms because it contains more knowledge, but it is also cumbersome. As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19 . It provided a effective way to employ the deep model to the real-time system.
In this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application. The deep learning systems benefit highly from a large quantity of labeled training data. Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset. It is a generic dataset sampled from diverse aspects of search queries. The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task. The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time. Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model. Transfer learning expends less data and less training time than full training. In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR). As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data.
Our deep LSTM training framework for LVCSR is presented in Section 2. Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task. The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5.

Parallel Training
It is demonstrated that training with larger dataset can improve recognition accuracy. However, larger dataset means more training samples and more model parameters. Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9 . The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs. Each GPU trains with one split of training dataset locally. All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13 .
Model average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training. Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well. It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14 . In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs. On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2 
Where INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update. There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3 . Then, the global model is updated as INLINEFORM4 
Consequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method.
Averaged SGD is proposed to further accelerate the convergence speed of SGD. Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2 
Where INLINEFORM0 is computed by model averaging or BMUF. It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training. It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization. However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training.
Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF. But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0 
The weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order. But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2 . As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero. After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3 
Here INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate. EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted. Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly.

Deployment
There is a high real time requirement in real world application, especially in online voice search system. Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking. Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition.

Training Data
A large quantity of labeled data is needed for training a more accurate acoustic model. We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China. The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones. This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on.
In the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search. To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search. 7300 hours labeled data is collected in the similar way of Shenma voice search data collection.
Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10 . The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously. The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test.

Layer-wise Training
In layer-wise training, the deeper model learns both parameters and knowledge from the shallower model. The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one. Two targets have the same weights in our framework. The teacher model is trained with CE. For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement. In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training. There is no obvious performance enhancement when the size of sMBR training dataset increases.
For very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers. Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model. Consequently, deeper LSTM is also trained in the same way. It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model. As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep. Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model. A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12.6% decreasing of CER. It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0.73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising.

Transfer Learning
2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model. It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search. Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training. If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough. Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR. As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario. 2-layers Shenma model further trained with sMBR achieves about 8.1% relative reduction, compared with 2-layers regular-trained Amap model. Both training sMBR datasets contain the same 1000 hours data. As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data. Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time.

Fig. 1. Mesh AllReduce of model averaging and BMUF.

Table 3. The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM.

Table 4. The CER of different 2-layers models, which are Shenma distilled model, Amap model further trained with Amap dataset, and Shenma model trained with sMBR on Amap dataset.
----------------------------------------

Question: how small of a dataset did they train on?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 19 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: how small of a dataset did they train on?

Answer JSON object:
{"answer": "23085 hours of data", "evidence": ["Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10 . The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously. The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test."]}