Upload nanochat-1.8B-midtrain model

.gitattributes

@@ -1,35 +1,5 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer/tokenizer.pkl filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,156 @@
+---
+license: mit
+base_model: nanochat
+tags:
+  - nanochat
+  - llm
+  - dgx-spark
+  - grace-blackwell
+  - from-scratch
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# nanochat-1.8B-midtrain
+
+Midtrained model fine-tuned for conversational interactions. Trained on SmolTalk dataset with special tokens for multi-turn conversations.
+
+## Model Details
+
+- **Model Type:** GPT-style transformer trained from scratch
+- **Parameters:** ~1.9 billion
+- **Training Phase:** midtrain
+- **Architecture:** 20 layers, 1280 embedding dimension
+- **Hardware:** NVIDIA DGX Spark (Grace Blackwell GB10)
+- **Framework:** [NanoChat](https://github.com/karpathy/nanochat)
+- **Training Precision:** BFloat16
+
+## Training Details
+
+- **GPU:** NVIDIA Grace Blackwell GB10
+- **Memory:** 128GB unified memory
+- **CUDA:** 13.0
+- **Optimization:** Muon optimizer for matrix parameters, AdamW for others
+- **Checkpoint Step:** 000813
+
+## Usage
+
+### Prerequisites
+
+```bash
+# Clone the NanoChat repository
+git clone https://github.com/karpathy/nanochat.git
+cd nanochat
+
+# Install dependencies (requires CUDA)
+uv sync --extra gpu
+
+# Activate the virtual environment
+source .venv/bin/activate
+```
+
+### Quick Test
+
+Download and test this model from HuggingFace:
+
+```bash
+# Clone the test script
+wget https://raw.githubusercontent.com/jasonacox/dgx-spark/main/nanochat/hf_test.py
+
+# Install dependencies
+pip install huggingface_hub
+
+# Run with this model
+python hf_test.py --model jasonacox/nanochat-1.8B-midtrain
+```
+
+### Example Code
+
+```python
+import sys
+import os
+import glob
+from huggingface_hub import snapshot_download
+import torch
+from contextlib import nullcontext
+
+# Download model from HuggingFace
+print("Downloading model...")
+model_path = snapshot_download(
+    repo_id="jasonacox/nanochat-1.8B-midtrain",
+    cache_dir=os.path.expanduser("~/.cache/nanochat/hf_downloads")
+)
+
+# Setup NanoChat (clone if needed)
+nanochat_path = "nanochat"
+if not os.path.exists(nanochat_path):
+    os.system("git clone https://github.com/karpathy/nanochat.git")
+    os.system("cd nanochat && uv sync --extra gpu")
+
+sys.path.insert(0, nanochat_path)
+
+from nanochat.checkpoint_manager import build_model
+from nanochat.common import compute_init, autodetect_device_type
+from nanochat.engine import Engine
+
+# Initialize
+device_type = autodetect_device_type()
+_, _, _, _, device = compute_init(device_type)
+ptdtype = torch.bfloat16
+autocast_ctx = torch.amp.autocast(device_type=device_type, dtype=ptdtype) if device_type == "cuda" else nullcontext()
+
+# Load model
+checkpoint_files = glob.glob(os.path.join(model_path, "model_*.pt"))
+step = int(os.path.basename(checkpoint_files[0]).split("_")[-1].split(".")[0])
+model, tokenizer, _ = build_model(model_path, step, device, phase="eval")
+engine = Engine(model, tokenizer)
+
+# Generate
+prompt = "Hello, how are you?"
+tokens = tokenizer.encode(prompt)
+print(f"Prompt: {prompt}\nResponse: ", end="", flush=True)
+
+with autocast_ctx:
+    for token_column, _ in engine.generate(tokens, num_samples=1, max_tokens=100, temperature=0.8, top_k=50):
+        print(tokenizer.decode([token_column[0]]), end="", flush=True)
+print()
+```
+
+## Training Pipeline
+
+This model was trained using the DGX Spark optimized training pipeline:
+
+1. **Pretraining:** Base language model on FineWeb-EDU dataset
+2. **Midtraining:** Fine-tuned on conversational data (SmolTalk)
+3. **SFT:** Supervised fine-tuning on curated conversations
+4. **RL:** Reinforcement learning with GRPO
+
+## Limitations
+
+- This is a micro-model (1.9B parameters) - smaller than commercial LLMs
+- May make factual errors or hallucinate
+- Limited knowledge cutoff from training data
+- Best suited for educational purposes and experimentation
+
+## Citation
+
+```bibtex
+@misc{nanochat-1.8B,
+  author = {jasonacox},
+  title = {nanochat-1.8B-midtrain},
+  year = {2025},
+  publisher = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/jasonacox/nanochat-1.8B-midtrain}}
+}
+```
+
+## Acknowledgments
+
+- Andrej Karpathy for [NanoChat](https://github.com/karpathy/nanochat)
+- NVIDIA DGX Spark platform
+- FineWeb-EDU and SmolTalk datasets
+
+## License
+
+MIT License - Free to use for research and educational purposes

config.json

@@ -0,0 +1,13 @@
+{
+  "model_type": "nanochat",
+  "architecture": "gpt",
+  "n_layer": 20,
+  "n_head": 10,
+  "n_kv_head": 10,
+  "n_embd": 1280,
+  "vocab_size": 65536,
+  "sequence_len": 2048,
+  "phase": "midtrain",
+  "checkpoint_step": 000813,
+  "torch_dtype": "bfloat16"
+}

meta_000813.json

@@ -0,0 +1,29 @@
+{
+  "step": 813,
+  "val_bpb": 0.3957736584182098,
+  "model_config": {
+    "sequence_len": 2048,
+    "vocab_size": 65536,
+    "n_layer": 20,
+    "n_head": 10,
+    "n_kv_head": 10,
+    "n_embd": 1280
+  },
+  "user_config": {
+    "run": "nanochat-midtrain",
+    "device_type": "",
+    "dtype": "bfloat16",
+    "num_iterations": -1,
+    "max_seq_len": 2048,
+    "device_batch_size": 32,
+    "unembedding_lr": 0.004,
+    "embedding_lr": 0.2,
+    "matrix_lr": 0.02,
+    "init_lr_frac": 1.0,
+    "weight_decay": 0.0,
+    "eval_every": 150,
+    "eval_tokens": 10485760,
+    "total_batch_size": 524288,
+    "dry_run": 0
+  }
+}

model_000813.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:925e640763858501445eef1bfab5374efcb13d9858d7d0ff05caffe02854c3e7
+size 2076230219

tokenizer/token_bytes.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae39c27aae519d14071efc95f9a558ba0b7ede47e7d83ad4f198422b44c5f70e
+size 263721

tokenizer/tokenizer.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c060565a46fe83b49d99005acba796f2a630daa7970eb49f7513b89f9fb40e0
+size 846208