You will be shown an excerpt from a computer science scientific research paper, followed by a question about the paper. Please answer the question. Do not include any text in your response other than the answer.

Context: Specifically, NLPOmaintains a masking policy \pi_{\psi}: the masking policy is a copy of the current policy (\pi_{\theta}), but is updated only every \mu steps.A parameterized-invalid-mask is created from \pi_{\psi} by first selecting the top-p tokens from the vocabulary,444\pi_{\psi} could be trained with alternate sampling techniques like top-k or beam search (or even hard-coded via rules by domain experts), though we find top-p sampling to be most effective in practice. and then applying an invalid-mask to the remaining tokens—i.e. setting their probabilities to zero when sampling actions from \pi_{\theta} during training;this periodic updating policy \pi_{\psi} is inspired by off-policy Q-learning algorithms (Andrychowicz et al., 2017),providing the policy \pi_{\theta} with an additional constraint that balances between the benefits of containing more task relevant information than the KL penalty derived from \pi_{0} and the risk of reward hacking.We provide pseudocode in Algorithm 1 (green portions highlight the differences with PPO). PPO vs. NLPO.Figure 2 shows that NLPO generally outperforms PPO and supervised, especially when applied after supervised training.We hypothesize that the primary reason for NLPO’s improved performance and stability is because the masking policy provides an additional constraint for the current policy.This constraint is not based on the initial untuned policy like the KL penalty but of the policy from \mu iterations ago and likely contains more task-relevant information learned during RL training.Table 3 (and Appendix Table 8) shows how performance increases up to a point and then decreases as p in top-p sampling is increased for the masking policy, relaxing the constraint by eliminating less tokens at each step, implying that there is a balance to be found in how much the model should be constrained during RL training.

Question: Why is it helpful to mask out less relevant tokens if these are less likely to be sampled anyways?
The authors hypothesize that their dynamic masking function helps because it adds a new constraint that the RL algorithm has to abide by. Additionally, since this is a dynamic function, being updated oftenly (every mu steps), it is likely that the masking function ensures that the selected top-p tokens are more relevant to the current state the RL algorithm needs to analyse and decide on.