You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Attention-based Wav2Text with Feature Transfer Learning

Conventional automatic speech recognition (ASR) typically performs multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units. But, it is widely known that information loss in the earlier stage can propagate through the later stages. After the resurgence of deep learning, interest has emerged in the possibility of developing a purely end-to-end ASR system from the raw waveform to the transcription without any predefined alignments and hand-engineered models. However, the successful attempts in end-to-end architecture still used spectral-based features, while the successful attempts in using raw waveform were still based on the hybrid deep neural network - Hidden Markov model (DNN-HMM) framework. In this paper, we construct the first end-to-end attention-based encoder-decoder model to process directly from raw speech waveform to the text transcription. We called the model as"Attention-based Wav2Text". To assist the training process of the end-to-end model, we propose to utilize a feature transfer learning. Experimental results also reveal that the proposed Attention-based Wav2Text model directly with raw waveform could achieve a better result in comparison with the attentional encoder-decoder model trained on standard front-end filterbank features.

Introduction
Conventional large-vocabulary continuous speech recognition (LVCSR) systems typically perform multi-level pattern recognition tasks that map the acoustic speech waveform into a hierarchy of speech units such as sub-words (phonemes), words, and strings of words (sentences). Such systems basically consist of several sub-components (feature extractor, acoustic model, pronunciation lexicon, language model) that are trained and tuned separately BIBREF0 . First, the speech signal is processed into a set of observation features based on a carefully hand-crafted feature extractor, such as Mel frequency cepstral coefficients (MFCC) or Mel-scale spectrogram. Then the acoustic model classifies the observation features into sub-unit or phoneme classes. Finally, the search algorithm finds the most probable word sequence based on the evidence of the acoustic model, the lexicon, and the language model. But, it is widely known that information loss in the earlier stage can propagate through the later stages.
Deep learning algorithms have produced many state-of-the-art performances in various tasks that have revitalized the use of neural networks for ASR. One of the important factors behind the popularity of deep learning is the possibility of simplifying many complicated hand-engineered models by letting DNNs find their way to map from input to output spaces. Interest has emerged recently in the possibility of learning DNN-based acoustic models directly from the raw speech waveform without any predefined alignments and hand-engineered models. In this way, the feature extractor and acoustic model can be integrated into a single architecture. Palaz et al. BIBREF1 , BIBREF2 proposed a convolutional neural network (CNN) to directly train an acoustic model from the raw speech waveform. Sainath et al. BIBREF3 used time-convolutional layers over raw speech and trained them jointly with the long short-term memory deep neural network (CLDNN) acoustic model. The results showed that raw waveform CLDNNs matched the performance of log-mel CLDNNs on a voice search task. Ghahremani et al. BIBREF4 recently proposed a CNN time-delay neural network (CNN-TDNN) with network-in-network (NIN) architecture, and also showed that their model outperformed MFCC-based TDNN on the Wall Street Journal (WSJ) BIBREF5 task. But despite significant progress that has been made, the successful models were mostly demonstrated only within the hybrid DNN-HMM speech recognition frameworks.
On the other hand, some existing works constructed end-to-end neural network models for ASR and replaced the acoustic model, the lexicon model, and the language model with a single integrated model, thus simplifying the pipeline. Graves et al. BIBREF6 , BIBREF7 successfully built an end-to-end ASR based on the connectionist temporal classification (CTC) framework. Amodei et al. BIBREF8 also constructed an end-to-end CTC-based ASR that directly produced character strings instead of phoneme sequences. But the CTC-based architecture still predicts the target outputs for every frame without any implicit knowledge about the language model. Another approach uses a sequence-to-sequence attention-based encoder-decoder that explicitly uses the history of previous outputs. Chorowski et al. BIBREF9 and Chan et al. BIBREF10 has successfully demonstrated encoder-decoder based ASR frameworks. Unfortunately, most of these works still used the standard spectral features (i.e., Mel-scale spectrogram, MFCC) as the input. The only attempt on end-to-end speech recognition for a raw waveform was recently proposed by BIBREF11 . Their system used a deep CNN and was trained with the automatic segmentation criterion (ASG) as an alternative to CTC. However, similar with CTC, the model did not explicitly use the history of the previous outputs assuming they were conditionally independent of each other. Furthermore, its performance was only reported using a very large data set (about 1000h of audio files).
To the best of our knowledge, few studies have explored a single end-to-end ASR architecture trained on raw speech waveforms to directly output text transcription, and none of those models were built based on an encoder-decoder architecture. In this paper, we take a step forward to construct an end-to-end ASR using an attentional-based encoder-decoder model for processing raw speech waveform, naming it as “Attention-based Wav2Text". We investigate the performance of our proposed models on standard ASR datasets. In practice, optimizing an encoder-decoder framework is more difficult than a standard neural network architecture BIBREF10 . Therefore, we propose a feature transfer learning method to assist the training process for our end-to-end attention-based ASR model.

Feature Transfer Learning
Deep learning is well known for its ability to learn directly from low-level feature representation such as raw speech BIBREF1 , BIBREF3 . However, in most cases such models are already conditioned on a fixed input size and a single target output (i.e., predicting one phoneme class for each input frame). In the attention-based encoder-decoder model, the training process is not as easy as in a standard neural network model BIBREF10 because the attention-based model needs to jointly optimize three different modules simultaneously: (1) an encoder module for producing representative information from a source sequence; (2) an attention module for calculating the correct alignment; and (3) a decoder module for generating correct transcriptions. If one of these modules has difficulty fulfilling its own tasks, then the model will fail to produce good results.
To ease the burden on training the whole encoder-decoder architecture directly to predict the text transcription given the raw speech waveform, we utilize a transfer learning method on the encoder part. Specifically, we only train the encoder's lower layers consisting of the convolutional and NIN layers to predict the spectral features given the corresponding raw waveform. In this work, we utilize two widely used spectral features: MFCC and log Mel-scale spectrogram as the transfer learning target. Figure FIGREF5 shows our feature transfer learning architecture. First, given segmented raw speech waveform INLINEFORM0 , we extract corresponding INLINEFORM1 -dimensional spectral features INLINEFORM2 . Then we process raw speech INLINEFORM3 with several convolutions, followed by NIN layers in the encoder part. In the last NIN-layer, we set a fixed number of channels as INLINEFORM4 channels and apply mean-pooling across time. Finally, we get predictions for corresponding spectral features INLINEFORM5 and optimize all of the parameters by minimizing the mean squared error between predicted spectral features INLINEFORM6 and target spectral features INLINEFORM7 : DISPLAYFORM0 
In this paper, we also explore multi target feature transfer using a similar structure as in Figure FIGREF5 but with two parallel NIN layers, followed by mean-polling at the end. One of the output layers is used to predicts log Mel-scale spectrogram and another predicts MFCC features. We modify the single target loss function from Eq. EQREF6 into the following: DISPLAYFORM0 
 where INLINEFORM0 are the predicted Mel-scale spectrogram and the MFCC values, and INLINEFORM1 are the real Mel-scale spectrogram and MFCC features for frame INLINEFORM2 . After optimizing all the convolutional and NIN layer parameters, we transfer the trained layers and parameters and integrate them with the Bi-LSTM encoder. Finally, we jointly optimize the whole structure together.

Speech Data
In this study, we investigate the performance of our proposed models on WSJ BIBREF5 . We used the same definitions of the training, development and test set as the Kaldi s5 recipe BIBREF18 . The raw speech waveforms were segmented into multiple frames with a 25ms window size and a 10ms step size. We normalized the raw speech waveform into the range -1 to 1. For spectral based features such as MFCC and log Mel-spectrogram, we normalized the features for each dimension into zero mean and unit variance. For WSJ, we separated into two experiments by using WSJ-SI84 only and WSJ-SI284 data. We used dev_93 for our validation set and eval_92 for our test set. We used the character sequence as our decoder target and followed the preprocessing step proposed by BIBREF19 . The text from all the utterances was mapped into a 32-character set: 26 (a-z) alphabet, apostrophe, period, dash, space, noise, and “eos".

Result
An example of our transfer learning results is shown in Figure FIGREF8 , and Table TABREF14 shows the speech recognition performance in CER for both the WSJ-SI84 and WSJ-SI284 datasets. We compared our method with several published models like CTC, Attention Encoder-Decoder and Joint CTC-Attention model that utilize CTC for training the encoder part. Besides, we also train our own baseline Attention Encoder-Decoder with Mel-scale spectrogram. The difference between our Attention Encoder-Decoder (“Att Enc-Dec (ours)", “Att Enc-Dec Wav2Text") with Attention Encoder-Decoder from BIBREF24 (“Att Enc-Dec Content", “Att Enc-Dec Location") is we used the current hidden states to generate the attention vector instead of the previous hidden states. Another addition is we utilized “input feedback" method BIBREF13 by concatenating the previous context vector into the current input along with the character embedding vector. By using those modifications, we are able to improve the baseline performance.
Our proposed Wav2Text models without any transfer learning failed to converge. In contrast, with transfer learning, they significantly surpassed the performance of the CTC and encoder-decoder from Mel-scale spectrogram features. This suggests that by using transfer learning for initializing the lower part of the encoder parameters, our model also performed better then their original features.

Related Work
Transfer learning is the ability of a learning algorithm to convey knowledge across different tasks. The initial idea is to reuse previously obtained knowledge to enhance the learning for new things. The standard procedure are : first, train the model on a base dataset and task, then the learned features and/or parameters are reused for learning a second target dataset and task. Bengio et al. BIBREF25 provided deep reviews about multi-task and transfer learning on deep learning models. Jason et al. BIBREF26 showed that a model with transferred parameter consistently outperformed a randomly initialized one.
In speech recognition research, transfer learning has been studied for many years, including successful cases of speaker adaptation and cross-lingual acoustic modeling BIBREF27 . One popular scheme for utilizing DNNs for transfer learning within ASR frameworks is a tandem approach BIBREF28 . This idea first trains a DNN with a narrow hidden bottleneck layer to perform phoneme classification at the frame level and then reuses the activations from the narrow hidden bottleneck layer as discriminative features in conventional GMM-HMM or hybrid DNN-HMM models BIBREF29 . Another study introduced a convolutional bottleneck network as an alternative tandem bottleneck feature architecture BIBREF30 . However, although such a feature transfer learning framework provides many advantages in ASR, the usage in an end-to-end attention-based ASR framework has not been explored.
This study performs feature transfer learning on the encoder part of the end-to-end attention-based ASR architecture. We train the convolutional encoder to predict the spectral features given the corresponding raw speech waveform. After that, we transfer the trained layers and parameters, integrate them with the LSTM encoder-decoder, and eventually optimize the whole structure to predict the correct output text transcription given the raw speech waveform.

Conclusion
This paper described the first attempt to build an end-to-end attention-based encoder-decoder speech recognition that directly predicts the text transcription given raw speech input. We also proposed feature transfer learning to assist the encoder-decoder model training process and presented a novel architecture that combined convolutional, NIN and Bi-LSTM layers into a single encoder part for raw speech recognition. Our results suggest that transfer learning is a very helpful method for constructing an end-to-end system from such low-level features as raw speech signals. With transferred parameters, our proposed attention-based Wav2Text models converged and matched the performance with the attention-based encoder-decoder model trained on standard spectral-based features. The best performance was achieved by Wav2Text models with transfer learning from multi target scheme.

Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP 17K00237. 

Fig. 1. Attention-based Wav2Text architecture.

Fig. 2. Feature transfer learning: train lower layers of the encoder (convolutional and NIN layers) to predict spectral features given corresponding raw waveform; then transfer the trained layers and parameters (marked by orange square) into attention-based encoder decoder model (see Figure 1).

Fig. 3. Example of our transfer learning model output: top is the original Mel-spectrogram, and bottom is the predicted Melspectrogram.

Table 1. Layer setting details for convolutional and NIN layers. Sorted from the input layer to the output layer.

Table 2. Character error rate (CER) result from baseline and proposed models on WSJ0 and WSJ1 dataset. All of these results are produced without using language model or lexicon dictionary. Word error rate (WER) for Att Wav2Text + transfer multi-target is 17.04%, compared to Joint CTC+Att (MTL)[25] 18.2% and standard Enc-Dec Att [23] 18.6%.

Question: Which dataset do they use?
{"answer": ["WSJ"], "evidence": ["In this study, we investigate the performance of our proposed models on WSJ BIBREF5 . We used the same definitions of the training, development and test set as the Kaldi s5 recipe BIBREF18 . The raw speech waveforms were segmented into multiple frames with a 25ms window size and a 10ms step size. We normalized the raw speech waveform into the range -1 to 1. For spectral based features such as MFCC and log Mel-spectrogram, we normalized the features for each dimension into zero mean and unit variance. For WSJ, we separated into two experiments by using WSJ-SI84 only and WSJ-SI284 data. We used dev_93 for our validation set and eval_92 for our test set. We used the character sequence as our decoder target and followed the preprocessing step proposed by BIBREF19 . The text from all the utterances was mapped into a 32-character set: 26 (a-z) alphabet, apostrophe, period, dash, space, noise, and \u201ceos\"."]}