Wonderful Matrices: Combining for a More Efficient and Effective
  Foundation Model Architecture
In order to make the foundation model more efficient and effective, our idea
is combining sequence transformation and state transformation. First, we prove
the availability of rotary position embedding in the state space duality
algorithm, which reduces the perplexity of the hybrid quadratic causal
self-attention and state space duality by more than 4%, to ensure that the
combining sequence transformation unifies position encoding. Second, we propose
dynamic mask attention, which maintains 100% accuracy in the more challenging
multi-query associative recall task, improving by more than 150% compared to
quadratic causal self-attention and state space duality, to ensure that the
combining sequence transformation selectively filters relevant information.
Third, we design cross domain mixture of experts, which makes the computational
speed of expert retrieval with more than 1024 experts 8 to 10 times faster than
the mixture of experts, to ensure that the combining state transformation
quickly retrieval mixture. Finally, we summarize these matrix algorithms that
can form the foundation model: Wonderful Matrices, which can be a competitor to
popular model architectures.
6760f7212504d8a0fbf21ac9
Jingze Shi
extracted_confirmed
2025-08-05T01:42:00.947Z
673ab3647afcea17eb4378fd
https://cdn-avatars.huggingface.co/v1/production/uploads/673ab3647afcea17eb4378fd/YQB6zSH1LPxBMUYayIURi.png
Loser Cheems
user
JingzeShi
6760f7212504d8a0fbf21aca
Bingheng Wu
claimed_verified
2024-12-17T08:03:41.783Z
66a0a9e7e81290f90f93b0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/66a0a9e7e81290f90f93b0a6/_HAsUv1eh0Im-aYZaa1C0.png
BinghengWu
user
wubingheng
Wonderful Matrices: Combining for a More Efficient and Effective
  Foundation Model Architecture
In order to make the foundation model more efficient and effective, our idea
is combining sequence transformation and state transformation. First, we prove
the availability of rotary position embedding in the state space duality
algorithm, which reduces the perplexity of the hybrid quadratic causal
self-attention and state space duality by more than 4%, to ensure that the
combining sequence transformation unifies position encoding. Second, we propose
dynamic mask attention, which maintains 100% accuracy in the more challenging
multi-query associative recall task, improving by more than 150% compared to
quadratic causal self-attention and state space duality, to ensure that the
combining sequence transformation selectively filters relevant information.
Third, we design cross domain mixture of experts, which makes the computational
speed of expert retrieval with more than 1024 experts 8 to 10 times faster than
the mixture of experts, to ensure that the combining state transformation
quickly retrieval mixture. Finally, we summarize these matrix algorithms that
can form the foundation model: Wonderful Matrices, which can be a competitor to
popular model architectures.
A foundation model integrates sequence and state transformations using rotary position embedding, dynamic mask attention, and cross-domain mixture of experts, improving efficiency and accuracy.
rotary position embedding state space duality hybrid quadratic causal self-attention dynamic mask attention multi-query associative recall cross domain mixture of experts mixture of experts Wonderful Matrices