GPT-Usenet
An 81-million parameter LLM using GPT-2 encodings. Trained using 10GB of USENET posts along with over 1 GB of miscellaneous BBS posts, digitized books, and text documents. Supervised fine-tuning should be performed before use.
Purpose of GPT-Usenet
LLMs are all currently focused on becoming larger and larger, able to do more and more. However, this just makes them jack of all trades, master of none. GPT-Usenet takes a different approach. Instead of trying to do everything perfectly, GPT-Usenet offers a digital stem cell, which can then be finetuned into a single, specialized role and run in parallel with copies of itself.
Technical Information
| Layers | 10 |
| Heads | 10 |
| Embeddings | 640 |
| Context Window | 1024 tokens |
| Tokenizer | GPT-2 BPE |
Training Information
| Training Loss | 2.3256 |
| Validation Loss | 2.3651 |
| Device | Google Colab L4 |
| Training Time | 16 Hours |
Example Syntax
| uucp: | The path of reasoning you want GPT-Usenet to use when thinking. Use lowercase words separated by exclamation points. |
| Internet: | The system calls relevant to this email |
| Path: | The path of reasoning you want GPT-Usenet to use when writing. Use lowercase words separated by exclamation points. |
| From: | The username who sent this message |
| Sender: | The group that username belongs to |
| Newsgroups: | The broad subject field of the email. |
| Subject: | The prompt |
| Message-ID: | The type of message this is. |
| Date: | Use this field to simulate urgency or moods. |
| Organization: | The system GPT-Usenet is running on.(testing... deployment... simulation) |
| Lines: | How long the message is. |
| Write the SFT response here. First, Prefix the first sentence with > to signify that it is a Reasoning sentence. | |
| -- | The stop tokens |
uucp:!field1!field2!
Internet:simulation
Path:!field1!field2!
From:user
Sender:usergroup
Newsgroups:motorskills.papercraft
Subject:Build a paper airplane
Message-ID:Command
Date:01 Jan 01 00:00:01 GMT
Organization:deployment
Lines: 1
>Provide detailed steps on building a paper airplane.
--
For finetuning, your data should be in the .mbox format.