You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: X-Stance: A Multilingual Multi-Target Dataset for Stance Detection

We extract a large-scale stance detection dataset from comments written by candidates of elections in Switzerland. The dataset consists of German, French and Italian text, allowing for a cross-lingual evaluation of stance detection. It contains 67 000 comments on more than 150 political issues (targets). Unlike stance detection models that have specific target issues, we use the dataset to train a single model on all the issues. To make learning across targets possible, we prepend to each instance a natural question that represents the target (e.g."Do you support X?"). Baseline results from multilingual BERT show that zero-shot cross-lingual and cross-target transfer of stance detection is moderately successful with this approach.

Related Work ::: Representation Learning for Stance Detection
In a target-specific setting, BIBREF10 perform a systematic evaluation of stance detection approaches. They also evaluate Bert BIBREF5 and find that it consistently outperforms previous approaches.
However, they only experimented with a single-segment encoding of the input, preventing cross-target transfer of the model. BIBREF11 propose a conditional encoding approach to encode both the target and the tweet as sequences. They use a bidirectional LSTM to condition the encoding of the tweets on the encoding of the target, and then apply a nonlinear projection on the conditionally encoded tweet. This allows them to train a model that can generalize to previously unseen targets.

The x-stance Dataset ::: Data Collection ::: Provenance
We downloaded questions and answers via the Smartvote API. The downloaded data cover 175 communal, cantonal and national elections between 2011 and 2020.
All candidates in an election who participate in Smartvote are asked the same set of questions, but depending on the locale they see translated versions of the questions. They can answer each question with either `yes', `rather yes', `rather no', or `no'. They can supplement each answer with a comment of at most 500 characters.
The questions asked on Smartvote have been edited by a team of political scientists. They are intended to cover a broad range of political issues relevant at the time of the election. A detailed documentation of the design of Smartvote and the editing process of the questions is provided by BIBREF12.

The x-stance Dataset ::: Data Collection ::: Preprocessing
We merged the two labels on each pole into a single label: `yes' and `rather yes' were combined into `favor'; `rather no', or `no' into `against`. This improves the consistency of the data and the comparability to previous stance detection datasets.
We did not further preprocess the text of the comments.

The x-stance Dataset ::: Data Collection ::: Language Identification
As the API does not provide the language of comments, we employed a language identifier to automatically annotate this information. We used the langdetect library BIBREF13. For each responder we classified all the comments jointly, assuming that responders did not switch code during the answering of the questionnaire.
We applied the identifier in a two-step approach. In the first run we allowed the identifier to output all 55 languages that it supports out of the box, plus Romansh, the fourth official language in Switzerland. We found that no Romansh comments were detected and that all unexpected outputs were misclassifications of German, French or Italian comments. We further concluded that little or no Swiss German comments are in the dataset: If they were, some of them would have manifested themselves in the form of misclassifications (e.g. as Dutch).
In the second run, drawing from these conclusions, we restricted the identifier's output to English, French, German and Italian.

The x-stance Dataset ::: Data Collection ::: Filtering
We pre-filtered the questions and answers to improve the quality of the dataset. To keep the domain of the data surveyable, we set a focus on national-level questions. Therefore, all questions and corresponding answers pertaining to national elections were included.
In the context of communal and cantonal elections, candidates have answered both local questions and a subset of the national questions. Of those elections, we only considered answers to the questions that also had been asked in a national election. Furthermore, they were only used to augment the training set while the validation and test sets were restricted to answers from national elections.
We discarded the less than 20 comments classified as English. Furthermore, instances that met any of the following conditions were filtered from the dataset:
Question is not a closed question or does not address a clearly defined political issue.
No comment was submitted by the candidate or the comment is shorter than 50 characters.
Comment starts with “but” or a similar indicator that the comment is not a self-contained statement.
Comment contains a URL.
In total, a fifth of the original comments were filtered out.

The x-stance Dataset ::: Data Collection ::: Topics
The questions have been organized by the Smartvote editors into categories (such as “Economy”). We further consolidated the pre-defined categories into 12 broad topics (Table TABREF7).

The x-stance Dataset ::: Data Split
We held out the topics “Healthcare” and “Political System” from the training data and created a separate cross-topic test set that contains the questions and answers related to those topics.
Furthermore, in order to test cross-question generalization performance within previously seen topics, we manually selected 16 held-out questions that are distributed over the remaining 10 topics. We selected the held-out questions manually because we wanted to make sure that they are truly unseen and that no paraphrases of the questions are found in the training set.
We designated Italian as a test-only language, since relatively few comments have been written in Italian. From the remaining German and French data we randomly selected a percentage of respondents as validation or as test respondents.
As a result we received one training set, one validation set and four test sets. The sizes of the sets are listed in Table TABREF8. We did not consider test sets that are cross-lingual and cross-target at the same time, as they would have been too small to yield significant results.

The x-stance Dataset ::: Analysis
Some observations regarding the composition of x-stance can be made.

The x-stance Dataset ::: Analysis ::: Linguistic Properties
Not every question is unique; some questions are paraphrases describing the same political issue. For example, in the 2015 election, the candidates were asked: “Should the consumption of cannabis as well as its possession for personal use be legalised?” Four years later they were asked: “Should cannabis use be legalized?” However, we do not see any need to consolidate those duplicates because they contribute to the diversity of the training data.
We further observe that while some questions in the dataset are quite short, some questions are rather convoluted. For example, a typical long question reads:
Some 1% of direct payments to Swiss agriculture currently go to organic farming operations. Should this proportion be increased at the expense of standard farming operations as part of Switzerland's 2014-2017 agricultural policy?
Such longer questions might be more challenging to process semantically.

Baseline Experiments ::: Multilingual BERT Baseline
Secondly, we fine-tune multilingual Bert (M-Bert) on the task BIBREF5 which has been pretrained jointly in 104 languages and has established itself as a state of the art for various multilingual tasks BIBREF18, BIBREF19. Within the field of stance detection, Bert can outperform both feature-based and other neural approaches in a monolingual English setting BIBREF10.

Baseline Experiments ::: Multilingual BERT Baseline ::: Architecture
In the context of Bert we interpret the x-stance task as sequence pair classification inspired by natural language inference tasks BIBREF20. We follow the procedure outlined by BIBREF5 for such tasks. We designate the question as segment A and the comment as segment B. The two segments are separated with the special token [SEP], and the special token [CLS] is prepended to the sequence. The final hidden state corresponding to [CLS] is then classified by a linear layer.
We fine-tune the full model with a cross-entropy loss, using the AllenNLP library BIBREF21 as a basis for our implementation.

Baseline Experiments ::: Multilingual BERT Baseline ::: Training
We upsampled the `favor' class so that the two classes are balanced when summing over all questions and topics. A maximum sequence length of 512 subwords and a batch size of 16 was chosen for all training runs. We then performed a grid search within the following range of hyperparameters based on the validation accuracy:
Learning rate: 5e-5, 3e-5, 2e-5
Number of epochs: 3, 4
The grid search was repeated independently for every variant that we tested. Furthermore, the standard recommendations for fine-tuning Bert were used: Adam with ${\beta }_1=0.9$ and ${\beta }_2=0.999$; an L2 weight decay of $0.01$; a learning rate warmup over the first 10% of the steps; and a linear decay of the learning rate. A dropout probability of 0.1 was set on all layers.

Baseline Experiments ::: How Important are Spelled-Out Targets?
Finally we test whether the target really needs to be represented by natural language (e.g. “Do you support X?”). Namely, an alternative is to represent the target with a trainable embedding instead of a question.
In order to fit target embeddings smoothly into our architecture, we represent each target type with a different reserved symbol from the M-Bert vocabulary. Segment A is then set to this symbol instead of a natural language question.
The results for this experiment are listed in the bottom row of Table TABREF39. An M-Bert model that learns target embeddings instead of encoding a question performs clearly worse in the supervised and cross-lingual settings. From this we conclude that spelled-out natural language questions provide important linguistic detail that can help in stance detection.

Discussion
The baseline experiments confirm that M-Bert can achieve a reasonable accuracy on x-stance.
To put the supervised score into context we list scores that variants of Bert have achieved on other stance detection datasets in Table TABREF46. It seems that the supervised part of x-stance has a similar difficulty as the SemEval-2016 BIBREF0 or MPCHI BIBREF22 datasets on which Bert has previously been evaluated.
On the other hand, in the cross-lingual and cross-target settings, the mean score drops by 6–8 percentage points compared to the supervised setting; while zero-shot transfer is possible to a degree, it can still be improved.
The additional experiments (Table TABREF39) validate the results and show that the sequence-pair classification approach to stance detection is justified.
It is interesting to see what errors the M-Bert model makes. Table TABREF47 presents instances where it predicts the wrong label with a high confidence. These examples indicate that many comments express their stance only on a very implicit level, and thus hint at a potential weakness of the dataset. Because on the voting advice platform the label is explicitly shown to readers in addition to the comments, the comments do not need to express the stance explicitly. Manual annotation could eliminate very implicit samples in a future version of the dataset. However, the sheer size and breadth of the dataset could not realistically be achieved with manual annotation, and, in our view, largely compensates for the implicitness of the texts.

Conclusion
We have presented a new dataset for political stance detection called x-stance. The dataset extends over a broad range of topics and issues regarding national Swiss politics. This diversity of topics opens up an opportunity to further study multi-target learning. Moreover, being partly Swiss Standard German, partly French and Italian, the dataset promotes a multilingual approach to stance detection.
By compiling formal commentary that politicians have written on political questions, we add a new text genre to the field of stance detection. We also propose a question–answer format that allows us to condition stance detection models on a target naturally.
Our baseline results with multilingual Bert show that the model has some capability to perform zero-shot transfer to unseen languages and to unseen targets (both within a topic and to unseen topics). However, there is some gap in performance that future work could address. We expect that the x-stance dataset could furthermore be a valuable resource for fields such as argument mining, argument search or topic classification.

Acknowledgments
This work was funded by the Swiss National Science Foundation (project MUTAMUR; no. 176727). We would like to thank Isabelle Augenstein for helpful feedback.

Data Statement ::: Curation rationale
In order to study the automatic detection of stances on political issues, questions and candidate responses on the voting advice application smartvote.ch were downloaded. Mainly data pertaining to national-level issues were included to reduce variability.

Data Statement ::: Language variety
The training set consists of questions and answers in Swiss Standard German and Swiss French (74.1% de-CH; 25.9% fr-CH). The test sets also contain questions and answers in Swiss Italian (67.1% de-CH; 24.7% fr-CH; 8.2% it-CH). The questions have also been translated into English.

Data Statement ::: Speaker demographic (answers)
Candidates for communal, cantonal or national elections in Switzerland who have filled out an online questionnaire.
Age: 18 or older – mixed.
Gender: Unknown – mixed.
Race/ethnicity: Unknown – mixed.
Native language: Unknown – mixed.
Socioeconomic status: Unknown – mixed.
Different speakers represented: 7581.
Presence of disordered speech: Unknown.

Data Statement ::: Text characteristics
Questions, answers, arguments and comments regarding political issues.

Figure 1: Example of a question and two answers in the X-stance dataset. The answers were submitted by electoral candidates on a voting advice website. The author of the comment in German declared to be in favor of the issue and added the comment as an explanation. The comment in French has been written by another candidate to explain why a negative stance has been taken towards the issue. While X-stance contains hundreds of answers to this question, we have picked the two comments as examples due to their brevity.

Table 1: Number of questions and answers per topic.

Table 4: Baseline scores in the cross-target setting. For each test set we separately report a German and a French score, as well as their harmonic mean.

Table 5: Results for additional experiments. The cross-lingual score is the F1-score on the Italian test set. For the supervised, cross-question and cross-topic settings we report the harmonic mean of the German and French scores.

Table 6: Performance of BERT-like models on different supervised stance detection benchmarks.

Question: What annotations are present in dataset?
{"answer": ["answer each question with either `yes\u0027, `rather yes\u0027, `rather no\u0027, or `no\u0027.", "can supplement each answer with a comment of at most 500 characters"], "evidence": ["All candidates in an election who participate in Smartvote are asked the same set of questions, but depending on the locale they see translated versions of the questions. They can answer each question with either `yes\u0027, `rather yes\u0027, `rather no\u0027, or `no\u0027. They can supplement each answer with a comment of at most 500 characters."]}