You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Emotion Detection in Text: Focusing on Latent Representation

In recent years, emotion detection in text has become more popular due to its vast potential applications in marketing, political science, psychology, human-computer interaction, artificial intelligence, etc. In this work, we argue that current methods which are based on conventional machine learning models cannot grasp the intricacy of emotional language by ignoring the sequential nature of the text, and the context. These methods, therefore, are not sufficient to create an applicable and generalizable emotion detection methodology. Understanding these limitations, we present a new network based on a bidirectional GRU model to show that capturing more meaningful information from text can significantly improve the performance of these models. The results show significant improvement with an average of 26.8 point increase in F-measure on our test data and 38.6 increase on the totally new dataset.

Introduction
There have been many advances in machine learning methods which help machines understand human behavior better than ever. One of the most important aspects of human behavior is emotion. If machines could detect human emotional expressions, it could be used to improve on verity of applications such as marketing BIBREF0 , human-computer interactions BIBREF1 , political science BIBREF2 etc.
Emotion in humans is complex and hard to distinguish. There have been many emotional models in psychology which tried to classify and point out basic human emotions such as Ekman's 6 basic emotions BIBREF3 , Plutchik's wheel of emotions BIBREF4 , or Parrott's three-level categorization of emotions BIBREF5 . These varieties show that emotions are hard to define, distinguish, and categorize even for human experts.
By adding the complexity of language and the fact that emotion expressions are very complex and context dependant BIBREF6 , BIBREF7 , BIBREF8 , we can see why detecting emotions in textual data is a challenging task. This difficulty can be seen when human annotators try to assign emotional labels to the text, but using various techniques the annotation task can be accomplished with desirable agreement among the annotators BIBREF9 .

Related Work
A lot of work has been done on detecting emotion in speech or visual data BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 . But detecting emotions in textual data is a relatively new area that demands more research. There have been many attempts to detect emotions in text using conventional machine learning techniques and handcrafted features in which given the dataset, the authors try to find the best feature set that represents the most and the best information about the text, then passing the converted text as feature vectors to the classifier for training BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 . During the process of creating the feature set, in these methods, some of the most important information in the text such as the sequential nature of the data, and the context will be lost.
Considering the complexity of the task, and the fact that these models lose a lot of information by using simpler models such as the bag of words model (BOW) or lexicon features, these attempts lead to methods which are not reusable and generalizable. Further improvement in classification algorithms, and trying out new paths is necessary in order to improve the performance of emotion detection methods. Some suggestions that were less present in the literature, are to develop methods that go above lexical representations and consider the flow of the language.
Due to this sequential nature, recurrent and convolutional neural networks have been used in many NLP tasks and were able to improve the performance in a variety of classification tasks BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 . There have been very few works in using deep neural network for emotion detection in text BIBREF31 , BIBREF32 . These models can capture the complexity an context of the language better not only by keeping the sequential information but also by creating hidden representation for the text as a whole and by learning the important features without any additional (and often incomplete) human-designed features.
In this work, we argue that creating a model that can better capture the context and sequential nature of text , can significantly improve the performance in the hard task of emotion detection. We show this by using a recurrent neural network-based classifier that can learn to create a more informative latent representation of the target text as a whole, and we show that this can improve the final performance significantly. Based on that, we suggest focusing on methodologies that increase the quality of these latent representations both contextually and emotionally, can improve the performance of these models. Based on this assumption we propose a deep recurrent neural network architecture to detect discrete emotions in a tweet dataset. The code can be accessed at GitHub [https://github.com/armintabari/Emotion-Detection-RNN].

Baseline Approaches
We compare our approach to two other, the first one uses almost the same tweet data as we use for training, and the second one is the CrowdFlower dataset annotated for emotions.
In the first one Wang et al. BIBREF21 downloaded over 5M tweets which included one of 131 emotional hashtags based on Parrott's three-level categorization of emotions in seven categories: joy, sadness, anger, love, fear, thankfulness, surprise. To assess the quality of using hashtags as labels, the sampled 400 tweets randomly and after comparing human annotations by hashtag labels they came up with simple heuristics to increase the quality of labeling by ignoring tweets with quotations and URLs and only keeping tweets with 5 terms or more that have the emotional hashtags at the end of the tweets. Using these rules they extracted around 2.5M tweets. After sampling another 400 random tweets and comparing it to human annotation the saw that hashtags can classify the tweets with 95% precision. They did some pre-processing by making all words lower-case, replaced user mentions with @user, replaced letters/punctuation that is repeated more than twice with the same two letters/punctuation (e.g., ooooh INLINEFORM0 ooh, !!!!! INLINEFORM1 !!); normalized some frequently used informal expressions (e.g., ll → will, dnt INLINEFORM2 do not); and stripped hash symbols. They used a sub-sample of their dataset to figure out the best approaches for classification, and after trying two different classifiers (multinomial Naive Bayes and LIBLINEAR) and 12 different feature sets, they got their best results using logistic regression branch for LIBLINEAR classifier and a feature set consist of n-gram(n=1,2), LIWC and MPQA lexicons, WordNet-Affect and POS tags.
In the second one, the reported results are from a paper by BIBREF33 in which they used maximum entropy classifier with bag of words model to classify various emotional datasets. Here we only report part of their result for CrowdFlower dataset that can be mapped to one of our seven labels.

Data and preparation
There are not many free datasets available for emotion classification. Most datasets are subject-specific (i.e. news headlines, fairy tails, etc.) and not big enough to train deep neural networks. Here we use the tweet dataset created by Wang et al. As mentioned in the previous section, they have collected over 2 million tweets by using hashtags for labeling their data. They created a list of words associated with 7 emotions (six emotions from BIBREF34 love, joy, surprise, anger, sadness fear plus thankfulness (See Table TABREF3 ), and used the list as their guide to label the sampled tweets with acceptable quality.
After pre-processing, they have used 250k tweets as the test set, around 250k as development test and the rest of the data (around 2M) as training data. their best results using LIBLINEAR classifier and a feature set containing n-gram(n=1,2), LIWC and MPQA lexicons, WordNet-Affect and POS tags can be seen in Table TABREF4 . It can be seen that their best results were for high count emotions like joy and sadness as high as 72.1 in F-measure and worst result was for a low count emotion surprise with F-measure of 13.9.
As Twitter is against polishing this many tweets, Wang et al. provided the tweet ids along with their label. For our experiment, we retrieved the tweets in Wang et al.'s dataset by tweet IDs. As the dataset is from 7 years ago We could only download over 1.3 million tweets from around 2.5M tweet IDs in the dataset. The distribution of the data can be seen in Table TABREF5 .
In our experiment, we used simpler pre-processing steps which will be explained later on in the "Experiment" section.

Experiment
Minimal pre-processing was done by converting text to lower case, removing the hashtags at the end of tweets and separating each punctuation from the connected token (e.g., awesome!! INLINEFORM0 awesome !!) and replacing comma and new-line characters with white space. The text, then, was tokenized using TensorFlow-Keras tokenizer. Top N terms were selected and added to our dictionary where N=100k for higher count emotions joy, sadness, anger, love and N=50k for thankfulness and fear and N=25k for surprise. Seven binary classifiers were trained for the seven emotions with a batch size of 250 and for 20 epochs with binary cross-entropy as the objective function and Adam optimizer. The architecture of the model can be seen in Figure FIGREF6 . For training each classifier, a balanced dataset was created with selecting all tweets from the target set as class 1 and a random sample of the same size from other classes as class 0. For each classifier, 80% of the data was randomly selected as the training set, and 10% for the validation set, and 10% as the test set. As mentioned before we used the two embedding models, ConceptNet Numberbatch and fastText as the two more modern pre-trained word vector spaces to see how changing the embedding layer can affect the performance. The result of comparison among different embeddings can be seen in Table TABREF10 . It can be seen that the best performance was divided between the two embedding models with minor performance variations.
The comparison of our result with Wang et al. can be seen in Table TABREF9 . as shown, the results from our model shows significant improvement from 10% increase in F-measure for a high count emotion joy up to 61.7 point increase in F-measure for a low count emotion surprise. on average we showed 26.8 point increase in F-measure for all categories and more interestingly our result shows very little variance between different emotions compare to results reported by Wang et al.

Model Performances on New Dataset
To asses the performance of these models on a totally unseen data, we tried to classify the CrowdFlower emotional tweets dataset. The CrowdFlower dataset consists of 40k tweets annotated via crowd-sourcing each with a single emotional label. This dataset is considered a hard dataset to classify with a lot of noise. The distribution of the dataset can be seen in Table TABREF18 . The labeling on this dataset is non-standard, so we used the following mapping for labels:
sadness INLINEFORM0 sadness
worry INLINEFORM0 fear
happiness INLINEFORM0 joy
love INLINEFORM0 love
surprise INLINEFORM0 surprise
anger INLINEFORM0 anger
We then classified emotions using the pre-trained models and emotionally fitted fastText embedding. The result can be seen in Table TABREF19 . The baseline results are from BIBREF33 done using BOW model and maximum entropy classifier. We saw a huge improvement from 26 point increase in F-measure for the emotion joy (happiness) up to 57 point increase for surprise with total average increase of 38.6 points. Bostan and Klinger did not report classification results for the emotion love so we did not include it in the average. These results show that our trained models perform exceptionally on a totally new dataset with a different method of annotation.

Conclusion and Future Work
In this paper, we have shown that using the designed RNN based network we could increase the performance of classification dramatically. We showed that keeping the sequential nature of the data can be hugely beneficial when working with textual data especially faced with the hard task of detecting more complex phenomena like emotions. We accomplished that by using a recurrent network in the process of generating our hidden representation. We have also used a max-pooling layer to capture the most relevant features and an average pooling layer to capture the text as a whole proving that we can achieve better performance by focusing on creating a more informative hidden representation. In future we can focus on improving these representations for example by using attention networks BIBREF37 , BIBREF38 to capture a more contextual representation or using language model based methods like BERT BIBREF39 that has been shown very successful in various NLP tasks.

Table 1: Statistics in the original dataset from Wang et al.

Table 2: Results of final classification in Wang et al.

Figure 1: Bidirectional GRU architecture used in our experiment.

Table 3: Statistics in the downloaded dataset from Wang et al (2012). This is the main dataset used for training the model.

Table 5: Results of classification using two embedding models and bidirectional GRU. No meaningful differences was seen between the two models. Reported numbers are F1measures.

Table 4: Results of classification using bidirectional GRU. Reported numbers are F1-measures.

Table 6: Distribution of labels in CrowdFlower dataset.

Table 7: Results from classifying CrowdFlower data using pre-trained model. Reported numbers are F1-measure.
----------------------------------------

Question: What are the hyperparameters of the bi-GRU?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 152 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: What are the hyperparameters of the bi-GRU?

Answer JSON object:
{"answer": "They use the embedding layer with a size 35 and embedding dimension of 300. They use a dense layer with 70 units and a dropout layer with a rate of 50%.", "evidence": ["Minimal pre-processing was done by converting text to lower case, removing the hashtags at the end of tweets and separating each punctuation from the connected token (e.g., awesome!! INLINEFORM0 awesome !!) and replacing comma and new-line characters with white space. The text, then, was tokenized using TensorFlow-Keras tokenizer. Top N terms were selected and added to our dictionary where N=100k for higher count emotions joy, sadness, anger, love and N=50k for thankfulness and fear and N=25k for surprise. Seven binary classifiers were trained for the seven emotions with a batch size of 250 and for 20 epochs with binary cross-entropy as the objective function and Adam optimizer. The architecture of the model can be seen in Figure FIGREF6 . For training each classifier, a balanced dataset was created with selecting all tweets from the target set as class 1 and a random sample of the same size from other classes as class 0. For each classifier, 80% of the data was randomly selected as the training set, and 10% for the validation set, and 10% as the test set. As mentioned before we used the two embedding models, ConceptNet Numberbatch and fastText as the two more modern pre-trained word vector spaces to see how changing the embedding layer can affect the performance. The result of comparison among different embeddings can be seen in Table TABREF10 . It can be seen that the best performance was divided between the two embedding models with minor performance variations."]}