Create README.md

README.md

@@ -0,0 +1,129 @@
+---
+language:
+- en
+tags:
+- pytorch
+- causal-lm
+license: apache-2.0
+datasets:
+- the Pile
+---
+
+# Genji-python 6B
+
+## Model Description
+
+Genji is a transformer model finetuned on EleutherAI's GPT-J 6B model. This particular model is trained on python only code approaching 4GB in size.
+Split model has the checkpoints splitted, which makes it use less system RAM while loading and makes it faster to load.
+This model needs more effort to set up as you need to install git-lfs and pull the repo.
+
+| Hyperparameter    | Value  | 
+|-------------------|--------|
+| n_parameters      | 6,053,381,344 |
+| n_layers          | 28*    |
+| d_model           | 4,096  |
+| d_ff              | 16,384 |
+| n_heads           | 16     |
+| d_head            | 256    |
+| n_ctx             | 2,048  |
+| n_vocab           | 50,400 (same tokenizer as GPT-2/3)  |
+| position encoding | [Rotary position encodings (RoPE)](https://arxiv.org/abs/2104.09864) |
+| RoPE dimensions   | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |
+
+`*` each layer consists of one feedforward block and one self attention block
+
+The model consists of 28 layers with a model dimension of 4096, and a feedforward dimension of 16384. The model
+dimension is split into 16 heads, each with a dimension of 256. Rotary position encodings (RoPE) was applied to 64
+dimensions of each head. The model is trained with a tokenization vocabulary of 50257, using the same set of BPEs as
+GPT-2/GPT-3.
+
+## Training data
+
+GPT-J 6B was pretrained on the [Pile](pile.eleuther.ai), a large scale curated dataset created by EleutherAI for the purpose of training this model. After the pre-training, it's finetuned on the python code that was taken from the Pile.
+
+## Training procedure
+
+Genji-python-6B is trained for 20k steps on around 655 million tokens with learning rate of 2e-06
+
+## Intended Use
+
+This model is trained for assistence on writing python code and having fun trying weird stuff with it. 
+
+### How to use
+
+This model is only usable with our fork because GPT-J is not merged to the main transformers repo yet. When it's merged, we will make this model easily loadable.
+For now, you need to use this fork:
+[Fork](https://github.com/finetuneanon/transformers)
+
+to install with pip:
+```bash
+pip install git+https://github.com/finetuneanon/transformers@gpt-neo-localattention3-rp-b
+```
+
+**git-lfs** also needs to be installed, on ubuntu:
+```bash
+apt install git-lfs
+```
+
+after it's installed, initialize git-lfs:
+```bash
+git lfs install
+```
+
+then clone this repo:
+```bash
+git clone https://huggingface.co/NovelAI/genji-python-6B-split
+```
+
+Now we can load the model.
+
+We recommend the usage of the model as FP16. That way, it fits in 16GB VRAM cards.
+
+How to use:
+```python
+from transformers import (
+    AutoTokenizer,
+    AutoModelForCausalLM,
+    GPTNeoForCausalLM,
+)
+
+model = AutoModelForCausalLM.from_pretrained("genji-python-6B-split/model").half().eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")
+
+text = '''def print_customer_name'''
+
+tokens = tokenizer(text, return_tensors="pt").input_ids
+generated_tokens = model.generate(tokens.long().cuda(), use_cache=True, do_sample=True, top_k=50, temperature=0.3, top_p=0.9, repetition_penalty=1.125, min_length=1, max_length=len(tokens[0]) + 400, pad_token_id=tokenizer.eos_token_id)
+last_tokens = generated_tokens[0][len(tokens[0]):]
+generated_text = tokenizer.decode(last_tokens)
+print("Generation:\n" + generated_text)
+```
+When ran, this code generates:
+```python
+Prompt:
+def print_customer_name
+Generation:
+(self, customer):
+        """Print the name of a customer."""
+        if not self.is_valid():
+            return
+
+        print("Customer: {}".format(customer))
+```
+
+For example usage, you can see our colab notebook as well:
+[Notebook](https://colab.research.google.com/drive/1PnWpx02IEUkY8jhLKd_NewUGEXahAska?usp=sharing)
+
+## Eval results
+
+TBD
+
+## Acknowledgements
+
+This project was possible because of the compute provided by the
+[TPU Research Cloud](https://sites.research.google/trc/) and [EleutherAI](https://eleuther.ai/) for pretraining of the GPT-J 6B.
+
+Thanks to everyone who contributed to this project:
+- [Aero](https://github.com/AeroScripts)
+- [Finetune](https://github.com/finetuneanon)
+- [Kurumuz](https://github.com/kurumuz)