You will be shown sections from a scientific research paper, together with a question about the paper. Paragraphs in the paper are separated by newlines. Your task is to answer the question based on the contents of the paper.

Paper:
----------------------------------------
Crowdsourcing for Beyond Polarity Sentiment Analysis A Pure Emotion Lexicon

Sentiment analysis aims to uncover emotions conveyed through information. In its simplest form, it is performed on a polarity basis, where the goal is to classify information with positive or negative emotion. Recent research has explored more nuanced ways to capture emotions that go beyond polarity. For these methods to work, they require a critical resource: a lexicon that is appropriate for the task at hand, in terms of the range of emotions it captures diversity. In the past, sentiment analysis lexicons have been created by experts, such as linguists and behavioural scientists, with strict rules. Lexicon evaluation was also performed by experts or gold standards. In our paper, we propose a crowdsourcing method for lexicon acquisition, which is scalable, cost-effective, and doesn't require experts or gold standards. We also compare crowd and expert evaluations of the lexicon, to assess the overall lexicon quality, and the evaluation capabilities of the crowd.

Introduction
Sentiment analysis aims to uncover the emotion conveyed through information. In online social networks, sentiment analysis is mainly performed for political and marketing purposes, product acceptance and feedback systems. This involves the analysis of various social media information types, such as text BIBREF0 , emoticons and hashtags, or multimedia BIBREF1 . However, to perform sentiment analysis, information has to be labelled with a sentiment. This relationship is defined in a lexicon.
Lexicon acquisition is a requirement for sentiment classification. During the acquisition process, individual or grouped information elements are labelled based on a class, usually an emotion. Sentiment classification is the task that uses the acquired lexicon and a classification method to classify a sentence, phrase, or social media submission as a whole, based on the aggregation of its labels. Thus, lexicon quality directly affects sentiment classification accuracy.
Both tasks can either be performed automatically BIBREF2 or manually BIBREF3 where the labelling by linguists or researchers themselves BIBREF4 . Apart from experts, manual labbeling can also be performed with the help of a wide network of people, known as crowdsourcing BIBREF5 . Crowdsourcing is widely used for polarity lexicons, but rarely for beyond polarity and never for the discovery of linguistic elements.
Sentiment analysis is commonly performed in polarity basis, i.e. the distinction between positive and negative emotion . These poles correspond to agreement and disagreement, or acceptance and disapproval, for candidates and products repsectively BIBREF6 .
Beyond polarity (also known as pure emotion) sentiment analysis aims to uncover an exact emotion, based on emotional theories BIBREF7 , BIBREF8 . Applications such as sentiment tracking, marketing, text correction, and text to speech systems can be improved with the use of distinct emotion lexicons.
However, beyond polarity studies acquire lexicons based on a set of strict rules, and the evaluation of experts. These lexicons use only a single emotion per term BIBREF9 . The problems of these approaches is the lack of uniformity and contribution freedom when relying on gold standards, and high costs with low scalability when employing experts. Natural Language Processing (NLP) applications that only rely on experts are less comprehensive, restricted, and not scalable, compared to crowdsourced NLP applications BIBREF10 .
This paper presents our approach for the acquisition of a multiclass and scalable crowdsourced pure emotion lexicon (PEL), based on Plutchik's eight basic emotions. Furthermore, the crowd is also responsible for identifying linguistic elements, namely intensifiers, negators, and stop words. Usually these elements are pooled from existing lists BIBREF11 created by experts. We also introduce a worker filtering method to identify and exclude dishonest or spamming contributors, that doesn't require gold standards. Our goal is to maintain an end to end automated work-flow for a crowdsourced (annotation and evaluation wise) lexicon acquisition process. Therefore, to highlight crowd's performance on evaluation, we compare evaluations from linguistic experts and the crowd itself.

Reliability
Single annotation reliability agreement is the degree of agreement between annotators, for term groups that have annotation majority in exactly one sub class. In our lexicon, single annotation reliability agreement was low, mainly due to the low number of annotators for each term group in relation to the high number of possible categories.
Based on Fleiss Kappa BIBREF30 (simply referred as k), and as seen in Table TABREF35 , term groups with 2 annotations had the lowest reliability agreement, while term groups with 6 annotations the highest reliability agreement. As the number of annotators rises, the number of possible agreement permutations increases but the number of major annotated subclasses decreases. More annotators have a positive effect in both k and certainty of classification.
As we restrict our lexicon to emotions, reliability increases for any number of annotators except two. This is explained by the decrease in the number of possible categories. When we restrict our analysis on emotion related annotation the probability for agreement in annotations increases, resulting in a high emotional k. The best way to increase k is to provide additional annotations that will eventually converge to a majority class or a limited group of classes.

Crowd and experts comparison
We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations). The experts are two Ph.D. linguists, while the crowd is made up of random high quality contributors that choose to participate in the task. As a reference, the cost of hiring two experts is equal to the cost of employing nineteen contributors in Crowdflower.
Evaluators were given a summary of the annotations received for the term group in the form of:The term group "inequality inequity" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered.
The summary of the evaluation for both experts and crowd can be seen in Figure FIGREF36 . The first graph presents the validity over the number of annotations in the main class of the term group. Although this information is hidden from the evaluators, high annotational agreement results in high evaluation scores. Both experts and the crowd follow that positive trend. Crowd contributors are more strict in their evaluations, but after four annotations we observe a significant validity increase on both crowd and experts.
Likewise, the annotation percentage for the majority class has a positive influence to the evaluation score, with the exception of 100% agreement, second graph Figure FIGREF36 . The weighing factor for term groups with 100% annotation agreement is the reduced number of total annotations, as the mean number of total annotations drops abruptly on the 100%, and total agreement is more frequent in term groups with low number of total annotations. It's worth noting that certain percentages can only occur on specific number of total annotations, e.g. 17% and 83% can only occur when the number of total annotations is six.
In emotion annotations, as seen on the third graph of Figure FIGREF36 crowd and experts follow a similar evaluation pattern. Anticipation and joy had the exact same evaluation, while every other emotion and stop words were evaluated lower from the crowd. The only subclasses evaluated higher from the crowd were intensifiers and negators, with a significant difference in the evaluations for the latter. Section 6.3 provides a more detailed evaluation for term groups that received at least one annotation as intensifiers or negators.
The final graph in Figure FIGREF36 presents a clear negative correlation of subclass agreement and evaluation scores. The highest number of subclasses that do not affect evaluation scores is three, above that there is a steady decline of the evaluation scores, for both the crowd and the experts.
The evaluation results provide some key insights in the importance of the number of annotations. The evaluation scores start to improve after four annotations. Annotational agreement and majority voting are less important. Subclass agreement has a negative effect on three or more subclasses. Most importantly and compared to experts, the crowd is a stricter evaluator with significantly lower costs, and higher scalability. Since strict evaluation leads to higher quality annotations, the evaluation can be performed by the crowd instead of experts. Crowd contributors can be found in high numbers and multiple platforms, compared to expert linguists.
Evaluation of intensifiers and negators, was also a batch of evaluation and annotation tasks, as mentioned in Section 6.2. However, the difference was that now evaluators had to answer if a term group included at least one valid intensifier or negator. The evaluation was again performed by experts and the crowd, as described in Section 6.2.1. Based on the annotations received in PEL, we used 541 term groups that had at least one annotation in any of the intensifying subclasses. Although, the particular selection of term groups is statistically significant, we expect relatively low evaluation scores. That is because the number of intensifying annotations is low in most of the selected term groups.
In Figure FIGREF40 , we define varying levels of agreement on the validity of the intensifying class, based on the agreement of evaluations. For the experts group, low agreement refers to term groups that received at least one out of two evaluations as valid, while high agreement requires the evaluation agreement of both experts. Similarly for the crowd, low agreement refers to a minimum of two valid evaluations, mid agreement corresponds to at least three, and high agreement requires an absolute agreement of all four evaluators.
Experts are far more strict than the crowd in the evaluation of intensifiers and negators. When the validity agreement is low on both evaluation groups, the average valid term group difference is more than 40%, but the high validity agreement the difference is just 5.33%. When high agreement evaluation is applied, the crowd and expert evaluations are almost identical. The number of crowd evaluations is the factor that provides a degree of freedom in the evaluation strictness.

Limitations
Lexicon acquisition is a complex task that includes a mixture of objective and subjective tasks. While annotation of emotions is more subjective, annotation of linguistic elements (such as stop words, emotion shift terms, intensifiers etc.) is purely objective. We presented a novel work flow that provides quality results for both subjective and objective tasks.
Subcomponents of the lexicon acquisition could be improved on an individual basis. Spell check can include spelling recommendations, filtering could incorporate rewarding and penalties, evaluation process can include experts and so on.
Crowd diversity in the annotation and evaluation process is another limiting factor. Ideally we would prefer a fixed number of individuals, each to annotate and evaluate the whole corpus. However, the uniformity of expert judgement is replaced with the diversity and mass of contributors.
The corpus may be limiting the term groups in the lexicon to specific domain-specific subjects. Comparisons with existing lexicons, such as NRC BIBREF21 indicate a moderate overlap with 40% common terms. Additionally, the number of annotations for a number of term groups is relatively low. However, the batch task of evaluation and annotation provided almost ten thousand annotations, and increased the mean number of annotations from 2.3 to 3.2.

Figure 1: The circumplex of emotions

Figure 2: PEL creation work-flow

Figure 3: Subclasses based on class selection

Figure 4: Percentages of excluded workers over single annotation

Figure 5: Validity over: Majority subclass annotations, Majority subclass annotations percentage, Subclasses, Subclass Agreement

Figure 6: Intensifying class evaluations
----------------------------------------

Question: How do they compare lexicons?

To answer the question, format your response as a `json` object with two fields:

"answer": A string providing a succinct answer to the question, in your own words.
"evidence": An array of strings. Each entry should be a full paragraph from the paper. Together, the evidence should serve as a justification for the answer.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": "BERT and RoBERTa."
  "evidence": ["We compare our approach against two baselines. In Table 1, we compare against BERT. In Table 2, we compare against RoBERTa. Our findings indicate that our approach improves over both baeslines..."]
}

The "answer" field should be roughly 146 characters in length.

Do not include any text in your response other than the json. If the question is unanswerable given the provided excerpts, respond with the single word "null".

To repeat, the question is: How do they compare lexicons?

Answer JSON object:
{"answer": "Human evaluators were asked to evaluate on a scale from 1 to 5 the validity of the lexicon annotations made by the experts and crowd contributors.", "evidence": ["We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations). The experts are two Ph.D. linguists, while the crowd is made up of random high quality contributors that choose to participate in the task. As a reference, the cost of hiring two experts is equal to the cost of employing nineteen contributors in Crowdflower.", "Evaluators were given a summary of the annotations received for the term group in the form of:The term group \"inequality inequity\" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered."]}