You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Multipurpose Intelligent Process Automation via Conversational Assistant

Intelligent Process Automation (IPA) is an emerging technology with a primary goal to assist the knowledge worker by taking care of repetitive, routine and low-cognitive tasks. Conversational agents that can interact with users in a natural language are potential application for IPA systems. Such intelligent agents can assist the user by answering specific questions and executing routine tasks that are ordinarily performed in a natural language (i.e., customer support). In this work, we tackle a challenge of implementing an IPA conversational assistant in a real-world industrial setting with a lack of structured training data. Our proposed system brings two significant benefits: First, it reduces repetitive and time-consuming activities and, therefore, allows workers to focus on more intelligent processes. Second, by interacting with users, it augments the resources with structured and to some extent labeled training data. We showcase the usage of the latter by re-implementing several components of our system with Transfer Learning (TL) methods.

Evaluation ::: Human Evaluation and Error Analysis
To evaluate our system on irregular examples, we conducted experiments with human tutors from the OMB+ platform. The tutors are experienced regarding rare or complex questions, ambiguous answers, misspellings and other infrequent but still very relevant problems, which occur during a dialogue. In the following, we investigate some common errors and make additional observations.
Misspellings and confusable spellings occur quite often in the user-generated text, and since we attempt to let the conversation remain very natural from the user side and thus, cannot require formal writing, we have to deal with various writing issues. One of the most frequent problems is misspellings. German words are generally long and can be complicated, and since users type quickly, this often leads to the wrong order of characters within words. To tackle this challenge, we used fuzzy match within ES. However, the maximum allowed edit distance in Elasticsearch is set to 2 characters. This means, that all the misspellings beyond this threshold could not be accurately recognized by ES (e.g., Differentialrechnung vs Differnezialrechnung). Another characteristic example would be the writing of the section or question number. The equivalent information can be written in several distinct ways, which has to be considered in our RegEx unit (e.g., Exercise 5 a, Exercise V a, Exercise 5 (a)). A similar problem occurs with confusable spelling (i.e.: Differentialrechnung vs Differentialgleichung). We analyzed the cases mentioned above and added some of the most common issues to the ES database or handled them with RegEx during the preprocessing step.
Elasticsearch Threshold: In some cases, the system failed to extract information, although the user provided it. In other cases, ES extracts information that was not mentioned in a user query at all. That occurs due to the relevancy scoring algorithm of Elasticsearch, where a document's score is a combination of textual similarity and other metadata based scores. Our analysis revealed that ES mostly fails to extract the information if the sentence (i.e., user message) is quite short (e.g., 5 words). To overcome this difficulty, we combined the current input $u_{t}$ with the dialog history. This step eliminated the problem and improved the retrieval quality. To solve the case where Elasticsearch extracts incorrect information (or information that was not mentioned in a query) is more challenging. We discovered that the problem comes from short words or sub-words (e.g., suffixes, prefixes), which ES considers to be credible enough. The Elasticsearch documentation suggests getting rid of stop words to eliminate this behavior. However, this did not improve the search in our case. Also, fine-tuning of ES parameters such as the relevance threshold, prefix length and minimum should match parameter did not bring significant improvements. To cope with this problem, we implemented a verification step, where a user is given a chance to correct the erroneously retrieved information.
The overall feedback from the tutors included reduced repetitive activities as well as reduced waiting times for students until their questions were processed. Also, tutors reported that the rate of cancelled sessions (switching to a human tutor) is rather low.

Structured Dialogue Acquisition
As we already mentioned, our system attempts to support the human tutor by assisting students, but it also collects structured and labeled training data in the background. In a trial run of the rule-based system, we were able to accumulate a toy-dataset with training dialogues. The assembled dialogues have the following format:
Plain dialogues with unique dialogue indexes;
Plain Information Dictionary information (e.g., extracted entities) collected for the whole dialogue;
Pairs of questions (i.e., user requests) and responses (i.e., bot responses) with the unique dialogue- and turn-indexes;
Triples in the form of (User Request, Next Action, Response). Information on the next system's action could be employed to train a Dialogue Manager unit with (deep-) machine learning algorithms;
For each state in the dialogue, we saved the entities that the system was able to extract from the provided user query, along with their position in the utterance. This information could be used to train a custom, domain specific Named Entity Recognition model.

Related Work
Individual components of a particular dialogue system could be implemented using a different kind of approach, starting with entirely rule- and template-based methods, and going towards hybrid approaches (using learnable components along with handcrafted units) and end-to-end trainable machine learning methods.
Rule-based Approaches: Though many of the latest research approaches handle NLU and NLG units by using statistical NLP models BIBREF5, BIBREF6, BIBREF7, most of the industrially deployed dialogue systems still use manual features or handcrafted rules for the state and action prediction, intent classification, and slot filling tasks BIBREF8, BIBREF9. The rule-based approach ensures robustness and stable performance that is crucial for industrial systems that interact with a large number of users simultaneously. However, it is highly expensive and time-consuming to deploy a real dialogue system built in this manner. The major disadvantage is that the usage of handcrafted systems is restricted to a specific domain, and possible domain adaptation requires extensive manual engineering.
End-to-End Learning Approaches: Due to the recent advance of end-to-end neural generative models BIBREF10, many efforts have been made to build an end-to-end trainable architecture for dialogue systems. Rather than using the traditional pipeline, an end-to-end model is conceived as a single module BIBREF8. Despite having better adaptability compared to any rule-based system and being easy to train, end-to-end approaches remain unattainable for commercial conversational agents operating on real-world data. A well and carefully constructed task-oriented dialogue system in a known domain using handcrafted rules and predefined responses, still outperforms the end-to-end systems due to its robustness BIBREF11, BIBREF12.
Hybrid Approaches: Though end-to-end learning is an attractive solution for dialogue systems, current techniques are data-intensive and require large amounts of dialogues to learn simple actions. To overcome this difficulty, BIBREF13 (BIBREF13) introduce Hybrid Code Networks (HCNs), which is an ensemble of retrieval and trainable units. The authors report, that compared to existing end-to-end methods, their approach considerably reduces the amount of data required for training BIBREF13. Hybrid models appear to replace the established rule- and template-based approaches which are currently utilized in an industrial setting.

Mutually Exclusive Rules
Assume a list of all theoretically possible dialogue states: $S$ = [topic, sub-topic, training, exercise, chapter level, section level, quiz, final examination, question number] and for each element $s_{n}$ in $S$ is true that:
This would give us all general (resp. possible) dialogue states without reference to the design of the OMB+ platform. However, to make the dialogue states fully suitable for the OMB+, from the general states, we take only those, which are valid. To define the validness of the state, we specify the following five Mutually Exclusive Rules (MER):
Rule ($R_{1}$) in Table TABREF54 denotes admissible configurations for topic and means that we are either given a topic ($T$) or not ($\lnot T$).
Rule ($R_{2}$) in Table TABREF55 denotes that either no information on the examination mode is given, or examination mode is Training ($TR$) or Exercise ($E$) or Quiz ($Q$) or Final Examination ($FE$), but not more than one mode at the same time.
Rule ($R_{3}$) in Table TABREF56 indicates that either no level information is provided, or the level corresponds to chapter level ($CL$) or to section level ($SL$), but not to both at the same time.
Rule ($R_{4}$) in Table TABREF57 means that Training ($TR$) and Exercise ($E$) examination modes can either belong to chapter level ($CL$) or to section level ($SL$), but not to both at the same time.
Rule ($R_{5}$) in Table TABREF58 symbolizes that we could be either given only a topic ($T$) or the combination of topic and sub-topic ($ST$) at the same time, or only sub-topic, or no information on this point at all.
We then define a valid dialogue state, as a dialogue state that meets all requirements of the abovementioned rules:
After we get the valid states for our dialogues, we want to make a mapping from each valid dialogue state to the next possible systems action. For that, we first define five transition rules :
means that no topic ($T$) is found in the ID (i.e., could not be extracted from user input).
indicates that no examination mode ($EM$) is found in the ID.
denotes that the extracted examination mode ($EM$) is either Training ($TR$) or Exercise ($E$).
means that no sub-topic ($ST$) is provided by a user, but ID either already contains the combination of topic ($T$), training ($TR$) and section level ($SL$), or the combination of topic, exercise ($E$) and section level, or the combination of topic and quiz ($Q$).
indicates that no question number ($QNR$) was provided by a student (or could not be successfully extracted).
Finally, we assumed the list of possible next actions for the system:
Following the transition rules, we mapped each valid dialogue state to the possible next action $a_{m}$ in $A$:
in the case where we do not have any topic provided, the next action is to ask for the topic ($T$).
if no examination mode is provided by a user (or it could not be successfully extracted from the user query), the next action is defined as ask for examination mode ($EM$).
in case where we know the examination mode $EM$ $\in $ [Training, Exercise], we have to ask about the level (i.e., training at chapter level or training at section level), thus the next action is ask for level ($L$).
if no sub-topic is provided, but the examination mode $EM$ $\in $ [Training, Exercise] at section level, the next action is defined as ask for sub-topic ($ST$).
if no question number is provided by a user, then the next action is ask for question number ($QNR$).

Table 4: Utterance-level F1 for the NAP task. Underlined: best performance for evaluation and test sets for default setting (without previous action context). In bold: best performance for evaluation and test sets on extended setting (with previous action context).

Question: How does the IPA label data after interacting with users?
{"answer": ["Plain dialogues with unique dialogue indexes", "Plain Information Dictionary information (e.g., extracted entities) collected for the whole dialogue", "Pairs of questions (i.e., user requests) and responses (i.e., bot responses)", "Triples in the form of (User Request, Next Action, Response)"], "evidence": ["Plain dialogues with unique dialogue indexes;", "Plain Information Dictionary information (e.g., extracted entities) collected for the whole dialogue;", "Pairs of questions (i.e., user requests) and responses (i.e., bot responses) with the unique dialogue- and turn-indexes;", "Triples in the form of (User Request, Next Action, Response). Information on the next system\u0027s action could be employed to train a Dialogue Manager unit with (deep-) machine learning algorithms;"]}