You will be shown an excerpt from a computer science scientific research paper, followed by a question about the paper. Please answer the question. Do not include any text in your response other than the answer.

Context: To test if the model can effectively learn under a constraint on the computational cost, we also test the model under a sparsity-based regularizer that discourages excessive use of query-key edges. We penalize each sampled edge by adding to the predictive loss a weighted regularization term \lambda\mathcal{L}_{s}, where \mathcal{L}_{s} denotes the average mask density across all attention heads. Table 9 shows the performance of SBM-Transformer across varying regularization weights. Under strong regularization, the model surprisingly retains competitive performance while significantly reducing the average mask density.This indicates that similar local optima are shared across regimes with varying attention density in the loss landscape, and the regularization term is able to drive the model towards finding optimal attention scores with smaller density. We also compare the densities of masks sampled at each layer of SBM-Transformer during test time to examine whether our model is capable of diversifying sparsity across layers for better performance. Recall that this allows models to gather information in different levels, as seen in pretrained BERT where lower layers focus on the overall content via dense attention while upper layers gather syntactic information with tree-like patterns [11]. For each of the five tasks, we pick two highest-performing models (one for unregularized and another for regularized) for measurement. Figure 5 shows the average layer-wise mask densities of unregularized and regularized SBM-Transformers across different tasks. We find that under no regularization, the two layers can differ by more than 10% in tasks such as ListOps and Image. This may be due to the hierarchical and compositional structure of the two tasks. We also find that the variation is relatively low in Text with densities around 25%, indicating that the task requires broad attention overall. Lastly, the standard deviation is extremely large in upper layers for Pathfinder, showing that it samples a wide variety of masks depending on the input.

Question: How is the "average attention sparsity" measured in the experiments?
The average attention sparsity is measured by the densities of masks sampled in SBM-Transformer averaged across all attention heads.