Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion
  Enhances Protein Representations
Protein language models (PLMs) have emerged as powerful tools to detect
complex patterns of protein sequences. However, the capability of PLMs to fully
capture information on protein sequences might be limited by focusing on single
pre-training tasks. Although adding data modalities or supervised objectives
can improve the performance of PLMs, pre-training often remains focused on
denoising corrupted sequences. To push the boundaries of PLMs, our research
investigated a multi-task pre-training strategy. We developed Ankh3, a model
jointly optimized on two objectives: masked language modeling with multiple
masking probabilities and protein sequence completion relying only on protein
sequences as input. This multi-task pre-training demonstrated that PLMs can
learn richer and more generalizable representations solely from protein
sequences. The results demonstrated improved performance in downstream tasks,
such as secondary structure prediction, fluorescence, GB1 fitness, and contact
prediction. The integration of multiple tasks gave the model a more
comprehensive understanding of protein properties, leading to more robust and
accurate predictions.
6836f9a6d44ecc28c70cc038
Hazem Alsamkary
claimed_verified
2025-05-28T12:12:23.522Z
61faa58a11deaebce017da4a
https://cdn-avatars.huggingface.co/v1/production/uploads/1670851610124-61faa58a11deaebce017da4a.jpeg
Hazem Essam
user
hazemessam
6836f9a6d44ecc28c70cc039
Mohamed Elshaffei
admin_assigned
2025-05-28T12:27:38.964Z
63959596401bd73e467e1fc2
/avatars/963d073e9d5ee414d5c185faf9d8d4a6.svg
Mohamed Elshaffei
user
shaffei
6836f9a6d44ecc28c70cc03a
Mohamed Elkerdawy
admin_assigned
2025-05-28T12:27:45.103Z
62f96db84588fe31f43aceb9
/avatars/7577f071b5a030f04f6ab0df46e5e246.svg
Mohamed Elkerdawy
user
melkerdawy
6836f9a6d44ecc28c70cc03b
Ahmed Elnaggar
Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion
  Enhances Protein Representations
Protein language models (PLMs) have emerged as powerful tools to detect
complex patterns of protein sequences. However, the capability of PLMs to fully
capture information on protein sequences might be limited by focusing on single
pre-training tasks. Although adding data modalities or supervised objectives
can improve the performance of PLMs, pre-training often remains focused on
denoising corrupted sequences. To push the boundaries of PLMs, our research
investigated a multi-task pre-training strategy. We developed Ankh3, a model
jointly optimized on two objectives: masked language modeling with multiple
masking probabilities and protein sequence completion relying only on protein
sequences as input. This multi-task pre-training demonstrated that PLMs can
learn richer and more generalizable representations solely from protein
sequences. The results demonstrated improved performance in downstream tasks,
such as secondary structure prediction, fluorescence, GB1 fitness, and contact
prediction. The integration of multiple tasks gave the model a more
comprehensive understanding of protein properties, leading to more robust and
accurate predictions.
A multi-task pre-training strategy for protein language models improves their performance on downstream protein prediction tasks by learning richer representations from sequence data alone.
protein language models pre-training masked language modeling protein sequence completion secondary structure prediction fluorescence fitness contact prediction