TheQweaker commited on
Commit
77cfbaf
·
verified ·
1 Parent(s): f42d3cf

Fix non-embedding param count (~92M) and drop unsourced token-count claim

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -107,8 +107,9 @@ count, temperature, and top-k.
107
 
108
  - Architecture: 12-layer DiT-style transformer, `d_model` 768, 12 heads, adaLN
109
  conditioning, untied embeddings, rotary positions. ≈169.6M total parameters
110
- (≈130M non-embedding). Context length 1024.
111
- - Training: OpenWebText, 1M steps, ≈33B tokens (Sahoo et al., 2024).
 
112
 
113
  ## Requirements and environment
114
 
 
107
 
108
  - Architecture: 12-layer DiT-style transformer, `d_model` 768, 12 heads, adaLN
109
  conditioning, untied embeddings, rotary positions. ≈169.6M total parameters
110
+ (≈92M non-embedding; the input and output embeddings are untied, ≈77M together).
111
+ Context length 1024.
112
+ - Training: OpenWebText, 1M steps (Sahoo et al., 2024).
113
 
114
  ## Requirements and environment
115