#
Using PEFT at Hugging Face

🤗 Parameter-Efficient Fine-Tuning (PEFT) is a library for efficiently adapting pre-trained language models to various downstream applications without fine-tuning all the model’s parameters.

##
Exploring PEFT on the Hub

You can find PEFT models by filtering at the left of the models page.

##
Installation

To get started, you can check out the Quick Tour in the PEFT docs. To install, follow the PEFT installation guide. You can also use the following one-line install through pip:

    $ pip install peft

##
Using existing models

All PEFT models can be loaded from the Hub. To use a PEFT model you also need to load the base model that was fine-tuned, as shown below. Every fine-tuned model has the base model in it’s model card.

    from transformers import AutoModelForCausalLM, AutoTokenizer
    from peft import PeftModel, PeftConfig

    base_model = "mistralai/Mistral-7B-v0.1"
    adapter_model = "dfurman/Mistral-7B-Instruct-v0.2"

    model = AutoModelForCausalLM.from_pretrained(base_model)
    model = PeftModel.from_pretrained(model, adapter_model)
    tokenizer = AutoTokenizer.from_pretrained(base_model)

    model = model.to("cuda")
    model.eval()

Once loaded, you can pass your inputs to the tokenizer to prepare them, and call `model.generate()` in regular `transformers` fashion.

    inputs = tokenizer("Tell me the recipe for chocolate chip cookie", return_tensors="pt")

    with torch.no_grad():
        outputs = model.generate(input_ids=inputs["input_ids"].to("cuda"), max_new_tokens=10)
        print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0])

It outputs the following:

    Tell me the recipe for chocolate chip cookie dough.

    1. Preheat oven to 375 degrees F (190 degrees C).
    2. In a large bowl, cream together 1/2 cup (1 stick) of butter or margarine, 1/2 cup granulated sugar, and 1/2 cup packed brown sugar.
    3. Beat in 1 egg and 1 teaspoon vanilla extract.
    4. Mix in 1 1/4 cups all-purpose flour.
    5. Stir in 1/2 teaspoon baking soda and 1/2 teaspoon salt.
    6. Fold in 3/4 cup semisweet chocolate chips.
    7. Drop by

If you want to load a specific PEFT model, you can click `Use in PEFT` in the model card and you will be given a working snippet!

##
Additional resources

- PEFT repository
- PEFT docs
- PEFT models