You will be shown an excerpt from a computer science scientific research paper, followed by a question about the paper. Please answer the question. Do not include any text in your response other than the answer.

Context: So far, we have shown the quantity and diversity of the generated data, but its quality remains uncertain. To investigate this, we randomly sample 200 instructions and randomly select 1 instance per instruction. We asked an expert annotator (co-author of this work) to label whether each instance is correct or not, in terms of the instruction, the instance input, and the instance output.Evaluation results in Table 2 show that most of the generated instructions are meaningful, while the generated instances may contain more noise (to a reasonable extent). However, we found that even though the generations may contain errors, most of them are still in the correct format or even partially correct, which can provide useful guidance for training models to follow instructions. We listed a number of good generations and bad generations in Table 10 and Table 11 respectively. Evaluating models’ performance on this evaluation set of diverse tasks is extremely challenging because different tasks require different expertise. Indeed, many of these tasks cannot be measured by automatic metrics or even be judged by normal crowdworkers (e.g., writing a program, or converting first-order logic into natural language). To get a more faithful evaluation, we asked the authors of the instructions to judge model predictions. The evaluators were asked to rate the output based on whether it accurately and effectively completes the task. We implemented a four-level rating system for categorizing the quality of the models’ outputs, defined as follows: Figure 5 provides the performance of GPT3 model and its instruction-tuned counterparts on this newly written instruction set. As anticipated, the vanilla GPT3 language model is largely unable to respond to instructions, and all instruction-tuned models demonstrate comparatively higher performance,Nonetheless, GPT3{}_{\textsc{Self-Inst}} (i.e., GPT3 model fine-tuned with Self-Instruct) outperforms those counterparts trained on T0 or SuperNI by a large margin, demonstrating the value of the generated data despite the noise.Compared with \text{InstructGPT}_{\text{001}} (c.f. footnote 1), GPT3{}_{\textsc{Self-Inst}} is quite close in the performance—if we count acceptable response with minor imperfections (Rating-3) as valid, GPT3{}_{\textsc{Self-Inst}} is only 5% behind \text{InstructGPT}_{\text{001}}. Lastly, our evaluation confirms the impressive instruction-following ability of \text{InstructGPT}_{\text{002}} & \text{InstructGPT}_{\text{003}} models. Although there are many factors behind this success, we conjecture that future work can largely benefit from improving the quality of our generated data by using human annotators or training a reward model to select better generations, similar to the algorithm used in Ouyang et al. (2022).

Question: Beyond correctness, why did the authors not evaluate the actual quality, meaning, or usefulness of the generated instructions?
One reason to explain why the authors did not perform more comprehensive quality evaluation of the generated outputs is the difficulty in judging the output of the model. Some tasks cannot be quickly verified by the average human (one example the authors provide for this is converting first-order logic into natural language - a task that only experts with the appropriate domain knowledge can perform). However, despite this challenge, the authors do perform some analysis to gauge the overall quality of the generated samples. They randomly select 200 instructions, and for each sample they examine the quality of one instance within each instruction. Doing this evaluation reveals that their approach performs much better than a vanilla GPT3 (i.e. a bare-bones GPT3 with nothing else).