You will be shown an excerpt from a computer science scientific research paper, followed by a question about the paper. Please answer the question. Do not include any text in your response other than the answer.

Context: Language representation pre-training Mikolov et al. (2013); Devlin et al. (2018) has been shown effective for improving many natural language processing tasks such as named entity recognition, sentiment analysis, and question answering.In order to get reliable word representation, neural language models are designed to learn word co-occurrence and then obtain word embedding with unsupervised learning.The methods in Word2Vec Mikolov et al. (2013) and Glove Pennington et al. (2014) represent words as vectors, where similar words have similar word representations. These word representations provide an initialization for the word vectors in other deep learning models. Recently, lots of works such as Cove McCann et al. (2017), Elmo Peters et al. (2018), GPT Radford et al. (2018) and BERT Devlin et al. (2018) improved word representation via different strategies, which has been shown to be more effective for down-stream natural language processing tasks. Some other researchers try to add more information based on these models. MT-DNN Liu et al. (2019) combine pre-training learning and multi-task learning to improve the performances over several different tasks in GLUE Wang et al. (2018). GPT-2 Radford et al. (2019) adds task information into the pre-training process and adapt their model to zero-shot tasks. XLM Lample and Conneau (2019) adds language embedding to the pre-training process which achieved better results in cross-lingual tasks. ERNIE use multi-layer Transformer Vaswani et al. (2017) as basic encoder like previous pre-traning model such as GPT, BERT and XLM.The Transformer can capture the contextual information for each token in the sentence via self-attention, and generates a sequence of contextual embeddings.

Question: How did previous models incorporate the knowledge embedding directly to language model?
Existing models incorporate the knowledge embedding by pre-training process.
They use these knowledge embedding as their initialized representation.
After the process, self-attention model is also used to capture the contextual information.
The reason is that GPT-2 and XLM add information by pre-training.