Upload README.md with huggingface_hub

README.md

@@ -1,3 +1,119 @@
 ---
+language: en
 license: mit
+tags:
+- pytorch
+- text-generation
+- qwen3
+- tinystories
 ---
+
+# Qwen3-0.6B Pre-trained on TinyStories
+
+This is a Qwen3-0.6B model pre-trained on the TinyStories dataset for 200k iterations.
+
+## Model Details
+
+- **Architecture**: Qwen3-0.6B
+- **Training Data**: TinyStories dataset from HuggingFace
+- **Training Iterations**: 200,000
+- **Parameters**: ~596M unique parameters
+- **Tokenizer**: GPT-2 tokenizer (tiktoken)
+- **Training Loss**: Available in training history
+
+## Quick Start
+
+### Download the Model
+
+```python
+from huggingface_hub import hf_hub_download
+import torch
+
+# Download model weights
+model_path = hf_hub_download(
+    repo_id="vuminhtue/qwen3-200k-tinystories",
+    filename="Qwen3_200k_model_params.pt"
+)
+
+# Download config
+config_path = hf_hub_download(
+    repo_id="vuminhtue/qwen3-200k-tinystories",
+    filename="config.json"
+)
+```
+
+### Load and Use
+
+```python
+import torch
+import tiktoken
+from Qwen3_model import Qwen3Model  # You need this file from the original code
+
+# Set up configuration
+QWEN3_CONFIG = {
+    "vocab_size": 151936,
+    "context_length": 40960,
+    "emb_dim": 1024,
+    "n_heads": 16,
+    "n_layers": 28,
+    "hidden_dim": 3072,
+    "head_dim": 128,
+    "qk_norm": True,
+    "n_kv_groups": 8,
+    "rope_base": 1000000.0,
+    "dtype": torch.bfloat16,
+}
+
+# Load model
+model = Qwen3Model(QWEN3_CONFIG)
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model.load_state_dict(torch.load(model_path, map_location=device))
+model = model.to(device)
+model.eval()
+
+# Generate text
+tokenizer = tiktoken.get_encoding("gpt2")
+# Your generation code here...
+```
+
+## Training Details
+
+- **Optimizer**: AdamW with weight decay (0.1)
+- **Learning Rate**: 1e-4 with warmup and cosine decay
+- **Batch Size**: 32 with gradient accumulation (32 steps)
+- **Context Length**: 128 tokens
+- **Mixed Precision**: bfloat16 training
+
+## Model Architecture
+
+- Grouped Query Attention (GQA) with 8 KV groups
+- RoPE (Rotary Position Embeddings)
+- RMSNorm for normalization
+- SiLU activation function
+- 28 transformer layers
+
+## Performance
+
+The model was trained on TinyStories, a dataset of simple stories for children. It can generate coherent short stories in a similar style.
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{qwen3-tinystories-2025,
+  author = {Tue Vu},
+  title = {Qwen3-0.6B Pre-trained on TinyStories},
+  year = {2025},
+  publisher = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/vuminhtue/qwen3-200k-tinystories}},
+}
+```
+
+## License
+
+MIT License
+
+## Contact
+
+For questions or issues, please open an issue on the HuggingFace model page.