You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Contextual Joint Factor Acoustic Embeddings

Embedding acoustic information into fixed length representations is of interest for a whole range of applications in speech and audio technology. We propose two novel unsupervised approaches to generate acoustic embeddings by modelling of acoustic context. The first approach is a contextual joint factor synthesis encoder, where the encoder in an encoder/decoder framework is trained to extract joint factors from surrounding audio frames to best generate the target output. The second approach is a contextual joint factor analysis encoder, where the encoder is trained to analyse joint factors from the source signal that correlates best with the neighbouring audio. To evaluate the effectiveness of our approaches compared to prior work, we chose two tasks - phone classification and speaker recognition - and test on different TIMIT data sets. Experimental results show that one of our proposed approaches outperforms phone classification baselines, yielding a classification accuracy of 74.1%. When using additional out-of-domain data for training, an additional 2-3% improvements can be obtained, for both for phone classification and speaker recognition tasks.

Introduction
In recent years, word embeddings have been successfully used in natural language processing (NLP), the most commonly known models are Word2Vec BIBREF0 and GloveBIBREF1. The reasons for such success are manifold. One key attribute of embedding methods is that word embedding models take into account context information of words, thereby allowing a more compact and manageable representation for wordsBIBREF2, BIBREF3. The embeddings are widely applied in many downstream NLP tasks such as neural machine translation, dialogue system or text summarisation BIBREF4, BIBREF5, BIBREF6, as well as in language modelling for speech recognitionBIBREF7.
Embeddings of acoustic (and speech) signals are of more recent interest. The objective is to represent audio sequence information in compact form, replacing the raw audio data with one that contains latent factors BIBREF8, BIBREF9. The projection into such (latent) spaces should take into account different attributes, such as phonemes, speaker properties, speaking styles, the acoustic background or the recording environment. Acoustic embeddings have been explored for a variety of speech tasks such as speech recognitionBIBREF10, speaker verificationBIBREF11 or voice conversionBIBREF12. However, learning acoustic embeddings is challenging: attributes mentioned above, e.g. speaker properties and phonemes, operate at different levels of abstraction and are often strongly interdependent, and therefore are difficult to extract and represent in a meaningful formBIBREF8.
For speech processing BIBREF13, BIBREF14, BIBREF15 also use context information to derive acoustic embeddings. However, BIBREF13, BIBREF14 focus on learning word semantic representations from raw audio instead of signal properties such as phonemes and speaker properties. BIBREF15 focus on learning speaker representations by modelling of context information with a Siamese networks that discriminate whether a speech segment is the neighbourhood of a target segment or not.
In this paper, two unsupervised approaches to generate acoustic embeddings using context modelling are proposed. Both methods make use of the variational auto-encoder framework as proposed inBIBREF16 and both approaches aim to find joint latent variables between the target acoustic segments and its surrounding frames. In the first instance a representation is derived from surrounding audio frames that allows to predict current frame, thereby generating target audio from common factors. The encoder element of the associated auto-encoder is further referred to as Contextual Joint Factor Synthesis (CJFS) encoder. In the second instance an audio frame is used to predict surrounding audio, which is further referred to as Contextual Joint Factor Analysis (CJFA) encoding. As shown in previous work variational auto-encoders can be used to derive latent variables such as speaker information and phonemesBIBREF8 more robustly. In this work it is shown that including temporal information can further improve performance and robustness, for both phoneme classification and speaker identification tasks. Furthermore the use of additional unlabelled out-of-domain data can improved modelling for the proposed approaches. As outlined above, prior work has made use of surrounding audio in different forms. To the best of our knowledge this work is the first to show that predicting surrounding audio allows for efficient extraction of latent factors in speech signals.
The rest of paper is organised as follows: In §SECREF2 related work is described, Methods for deriving acoustic embeddings, and context modelling methods in NLP, computer vision and speech are discussed. This is followed by the description of the two approaches for modelling context as used in this work, in §SECREF3. The experimental framework is described in §SECREF4, including the data organisation, baseline design and task definitions; in §SECREF5 and §SECREF6 experiments results are shown and discussed. This is followed by the conclusions and future work in §SECREF7.

Related Works ::: Acoustic Embeddings
Most interest in acoustic embeddings can be observed on acoustic word embeddings, i.e. projections that map word acoustics into a fixed size vector space. Objective functions are chosen to project different word realisations to close proximity in the embedding space. Different approaches were used in the literature - for both supervised and unsupervised learning. For the supervised case, BIBREF9 introduced a convolutional neural network (CNN) based acoustic word embedding system for speech recognition, where words that sound alike are nearby in Euclidean distance. In their work, a CNN is used to predict a word from the corresponding acoustic signal, the output of the bottleneck layer before the final softmax layer is taken to be the embedding for the corresponding word. Further work used different network architectures to obtain acoustic word embeddings: BIBREF10 introduces a recurrent neural network (RNN) based approach instead.
For the case that the word boundary information is available but the word label itself is unknown, BIBREF12 proposed word similarity Siamese CNNs. These are used to minimise a distance function between representations of two instances of the same word type whilst at the same time maximising the distance between two instances of different words.
Unsupervised approaches also exist. BIBREF17 proposed a convolutional variational auto-encoder based approach to derive acoustic embedding, in unsupervised fashion. The authors chose phoneme and speaker classification tasks on TIMIT data to assess the quality of their embeddings - an approach replicated in the work presented in this paper. BIBREF8, BIBREF18 proposed an approach called factorised hierarchical variational auto-encoder. The work introduces the concepts of global and local latent factors, i.e. latent variables that are shared on the complete utterance, or latent variables that change within the sequence, respectively. Results are again obtained using the same data and tasks as above.

Experimental Framework ::: Data and Use
Taking the VAE experiments as baseline, the TIMIT data is used for this workBIBREF25. TIMIT contains studio recordings from a large number of speakers with detailed phoneme segment information. Work in this paper makes use of the official training and test sets, covering in total 630 speakers with 8 utterances each. There is no speaker overlap between training and test set, which comprise of 462 and 168 speakers, respectively. All work presented here use of 80 dimensional Mel-scale filter bank coefficients.

Experimental Framework ::: Baselines
Work on VAE in BIBREF17 to learn acoustic embeddings conducted experiments using the TIMIT data set. In particular the tasks of phone classification and speaker recognition where chosen. As work here is an extension of such work we we follow the experimentation, however with significant extensions (see Section SECREF13). With guidance from the authors of the original workBIBREF17 our own implementation of VAE was created and compared with the published performance - yielding near identical results. This implementation then was also used as the basis for CJFS and CJFA, as introduced in § SECREF6.
For the assessment of embedded vector quality our work also follows the same task types, namely phone classification and speaker recognition (details in §SECREF13), with identical task implementations as in the reference paper. It is important to note that phone classification differs from the widely reported phone recognition experiments on TIMIT. Classification uses phone boundaries which are assumed to be known. However, no contextual information is available, which is typically used in the recognition setups, by means of triphone models, or bigram language models. Therefore the task is often more difficult than recognition. The baseline performance for VAE based phone classification experiments in BIBREF17 report an accuracy of 72.2%. The re-implementation forming the basis for our work gave an accuracy of 72.0%, a result that was considered to provide a credible basis for further work.
For the purpose of speaker recognition it is important to take into account the overlap between training and testing. Thus three different task configurations are considered, different to the setting in BIBREF17. Their baseline will be further referred as VAE baseline.

Results and Discussion
Table TABREF17 shows phone classification and speaker recognition results for the three model configurations: the VAE baseline, the CJFS encoder and the CJFA encoder. In our experiments the window size was set to 30 frames, namely 10 frames for the target and 10 frames for left and right neighbours, and an embedding dimension of 150. This was used for both CJFS and CJFA models alike. Results show that the CJFA encoder obtains significantly better phone classification accuracy than the VAE baseline and also than the CJFS encoder. These results are replicated for speaker recognition tasks. The CJFA encoder performs better on all tasks than the VAE baseline by a significant margin. It is noteworthy that performance on Task b is generally significantly lower than for Task a, for reasons of training overlap but also smaller training set sizes.
To further explore properties of the embedding systems a change of window size ($N$) and embedding dimension ($K$) is explored. One might argue that modelling context effectively widens the input data access. Hence these experiments should explore if there is benefit in the structure beyond data size. Graphs in Fig. FIGREF14 illustrate phone classification accuracy and speaker recognition performance for all three models under variation of latent size and window sizes. It is important to note that the target window size remains the same (10 frames) with an increase of $N$. Therefore e.g. $N=70$ describes the target window size is 10 frames, and the other two neighbour windows have 30 frames at either side (30,10,30 left to right). Better speaker recognition results are consistently obtained with the CJFA encoder for any configuration with competitive performance, compared with the VAE baseline and also CJFS settings - and CJFS settings mostly outperform the baseline. However the situation for phone classification is different. It is not surprising to see CJFS perform poorly on phone classification as the target frame in not present in the input, therefore the embedding just does not have the phone segment information. However, as per speaker recognition results, speaker information is retained.
A variation of the window sizes to larger windows seems detrimental in almost all cases, aside from the more difficult Task b. This may be in part the effect of the amount of training data available, however it confirms that contextual models outperform the baseline VAE model configuration, generally, and in particular also with the same amount of input data for speaker recognition. It is also noticeable that the decline or variation as a function of window size is less pronounced for the CJFA case, implying increased stability. For phone classification the trade-off benefit for window size is less clear.
For phone classification, increasing the embedding $K$ is helpful, but performance remains stable at $K=150$. Hence in all of the rest of our experiments, the embedding dimension is set to 150 for all of the rest configurations. For speaker recognition the observed variations are small.
A further set of experiments investigated the use of out of domain data for improving classification in a completely unsupervised setting. The RM corpus BIBREF29 was used in this case to augment the TIMIT data for training the embeddings only. All other configurations an training settings are unchanged. Table TABREF18 shows improvement after using additional out-of-domain data for training, except for in the case of CJFS and for phone classification. The improvement on all tasks with the simple addition of unlabelled audio data is remarkable. This is also true for the baseline, but the benefit of the proposed methods seems unaffected. The CJFA encoder performs better in comparison of the other two approaches and a absolute accuracy improvement of 7.9% for speaker recognition Task b is observed. The classification tasks benefits from the additional data even though the labelled data remains the same.

Analysis
To further evaluate the embeddings produced by the 3 models, visualisation using the t-SNE algorithmBIBREF30 is a common approach, although interpretation is sometimes difficult. Fig. FIGREF19 visualises the embeddings of phonemes in two-dimensional space, each phoneme symbol represents the mean vector of all of the embeddings belonging to the same phone classBIBREF31. One can observe that the CJFA encoder appears to generate more meaningful embeddings than the other two approaches - as phonemes belonging to the same sound classesBIBREF32 are grouped together in closer regions. The VAE baseline also has this behaviour but for example plosives are split and nasal separation seems less clear. Instead CJFS shows more confusion - as expected and explained above.

Fig. 2. Data split of the TIMIT corpus for definition of data sets for speaker recognition. Training and test sets are split int 4 parts of 2 utterances each. Different combination of sets for training and test are used of different tasks.

Table 1. Definition of training configurations a, b, and c.

Table 2. % Phone classification and speaker recognition accuracy with three different model types. Embedding dimension is 150 and target window size is 10 frames, neighbour window sizes are 10 frames each.

Fig. 4. The t-SNE visualisation of phones in the test set for three models: (a): VAE baseline (b): CJFS encoder and (c): CJFA encoder

Question: What classification baselines are used for comparison?
{"answer": ["VAE"], "evidence": ["Work on VAE in BIBREF17 to learn acoustic embeddings conducted experiments using the TIMIT data set. In particular the tasks of phone classification and speaker recognition where chosen. As work here is an extension of such work we we follow the experimentation, however with significant extensions (see Section SECREF13). With guidance from the authors of the original workBIBREF17 our own implementation of VAE was created and compared with the published performance - yielding near identical results. This implementation then was also used as the basis for CJFS and CJFA, as introduced in \u00a7 SECREF6.", "For the assessment of embedded vector quality our work also follows the same task types, namely phone classification and speaker recognition (details in \u00a7SECREF13), with identical task implementations as in the reference paper. It is important to note that phone classification differs from the widely reported phone recognition experiments on TIMIT. Classification uses phone boundaries which are assumed to be known. However, no contextual information is available, which is typically used in the recognition setups, by means of triphone models, or bigram language models. Therefore the task is often more difficult than recognition. The baseline performance for VAE based phone classification experiments in BIBREF17 report an accuracy of 72.2%. The re-implementation forming the basis for our work gave an accuracy of 72.0%, a result that was considered to provide a credible basis for further work."]}