Fill-Mask
Transformers
Safetensors
English
mdlm
diffusion
masked-diffusion
mechanistic-interpretability
custom_code
Instructions to use TheQweaker/mdlm-owt-noflash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TheQweaker/mdlm-owt-noflash with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="TheQweaker/mdlm-owt-noflash", trust_remote_code=True)# Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("TheQweaker/mdlm-owt-noflash", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Fix non-embedding param count (~92M) and drop unsourced token-count claim
Browse files
README.md
CHANGED
|
@@ -107,8 +107,9 @@ count, temperature, and top-k.
|
|
| 107 |
|
| 108 |
- Architecture: 12-layer DiT-style transformer, `d_model` 768, 12 heads, adaLN
|
| 109 |
conditioning, untied embeddings, rotary positions. ≈169.6M total parameters
|
| 110 |
-
(≈
|
| 111 |
-
|
|
|
|
| 112 |
|
| 113 |
## Requirements and environment
|
| 114 |
|
|
|
|
| 107 |
|
| 108 |
- Architecture: 12-layer DiT-style transformer, `d_model` 768, 12 heads, adaLN
|
| 109 |
conditioning, untied embeddings, rotary positions. ≈169.6M total parameters
|
| 110 |
+
(≈92M non-embedding; the input and output embeddings are untied, ≈77M together).
|
| 111 |
+
Context length 1024.
|
| 112 |
+
- Training: OpenWebText, 1M steps (Sahoo et al., 2024).
|
| 113 |
|
| 114 |
## Requirements and environment
|
| 115 |
|