You will be shown an excerpt from a computer science scientific research paper, followed by a question about the paper. Please answer the question. Do not include any text in your response other than the answer.

Context: Model parallelism places certain constraints on the modelarchitectures we can use. For example, we cannot afford to havebi-directional LSTM layers for all the encoder layers, since doing sowould reduce parallelism among subsequent layers, as each layer wouldhave to wait until both forward and backward directions of the previouslayer have finished. This would effectively constrain us to make use ofonly 2 GPUs in parallel (one for the forward direction and one for thebackward direction). For the attention portion of the model, we chose to align thebottom decoder output to the top encoder output to maximizeparallelism when running the decoder network. Had we aligned the top decoderlayer to the top encoder layer, we would have removed all parallelismin the decoder network and would not benefit from using more than oneGPU for decoding. Figure 1: The model architecture of GNMT, Google’s Neural Machine Translation system. On the left is the encoder network, on the right is the decoder network, in the middle is the attention module. The bottom encoder layer is bi-directional: the pink nodes gather information from left to right while the green nodes gather information from right to left. The other layers of the encoder are uni-directional. Residual connections start from the layer third from the bottom in the encoder and decoder. The model is partitioned into multiple GPUs to speed up training. In our setup, we have 8 encoder LSTM layers (1 bi-directional layer and 7 uni-directional layers), and 8 decoder layers. With this setting, one model replica is partitioned 8-ways and is placed on 8 different GPUs typically belonging to one host machine. During training, the bottom bi-directional encoder layers compute in parallel first. Once both finish, the uni-directional encoder layers can start computing, each on a separate GPU. To retain as much parallelism as possible during running the decoder layers, we use the bottom decoder layer output only for obtaining recurrent attention context, which is sent directly to all the remaining decoder layers. The softmax layer is also partitioned and placed on multiple GPUs. Depending on the output vocabulary size we either have them run on the same GPUs as the encoder and decoder networks, or have them run on a separate set of dedicated GPUs.

Question: How can the attention mechanism connecting the bottom layer of the decoder to the top layer of the encoder contribute to improving parallelism?
First we have to establish that LSTM layers reduces parallelism as each layer would have to wait until both forward and backward directions of the previous layer to finish. Then notice in Figure 1, the model architecture consists of 8 LSTM encoder layers (1 bi-directional and 7 uni-directional layers), and 8 decoder layers. During training the bottom bi-directional encoder layers compute in parallelism first, then the uni-directional encoder layers. So to retain retain and much possible parallelism during the decoder layers, the bottom layers of the decoder output only for obtaining the recurrent attention context which is sent directly to all the remaining decoder layers.