Update README.md
Browse files
README.md
CHANGED
|
@@ -140,3 +140,14 @@ This is a **software artifact**, not a hardware limitation.
|
|
| 140 |
|
| 141 |
* **Math:** Monarch Matrices decompose a dense layer into: $\text{BlockDiag}_1 \to \text{Permutation} \to \text{BlockDiag}_2$.
|
| 142 |
* **Solution:** A custom **Fused Triton Kernel** (planned) would fuse these steps, keeping intermediate activations in the GPU's SRAM (L1 Cache). This would drop dynamic VRAM usage significantly below the baseline, matching the FLOPs reduction.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
|
| 141 |
* **Math:** Monarch Matrices decompose a dense layer into: $\text{BlockDiag}_1 \to \text{Permutation} \to \text{BlockDiag}_2$.
|
| 142 |
* **Solution:** A custom **Fused Triton Kernel** (planned) would fuse these steps, keeping intermediate activations in the GPU's SRAM (L1 Cache). This would drop dynamic VRAM usage significantly below the baseline, matching the FLOPs reduction.
|
| 143 |
+
|
| 144 |
+
## Citation
|
| 145 |
+
|
| 146 |
+
@misc{ykae-monarch-bert-mnli-2026,
|
| 147 |
+
author = {Yusuf Kalyoncuoglu, YKAE-Vision},
|
| 148 |
+
title = {Monarch-BERT-MNLI: Extreme Compression via Monarch FFNs},
|
| 149 |
+
year = {2026},
|
| 150 |
+
publisher = {Hugging Face},
|
| 151 |
+
journal = {Hugging Face Model Hub},
|
| 152 |
+
howpublished = {\url{https://huggingface.co/ykae/monarch-bert-base-mnli}}
|
| 153 |
+
}
|