#
Using OpenCLIP at Hugging Face

OpenCLIP is an open-source implementation of OpenAI’s CLIP.

##
Exploring OpenCLIP on the Hub

You can find OpenCLIP models by filtering at the left of the models page.

OpenCLIP models hosted on the Hub have a model card with useful information about the models. Thanks to OpenCLIP Hugging Face Hub integration, you can load OpenCLIP models with a few lines of code. You can also deploy these models using Inference Endpoints.

##
Installation

To get started, you can follow the OpenCLIP installation guide. You can also use the following one-line install through pip:

    $ pip install open_clip_torch

##
Using existing models

All OpenCLIP models can easily be loaded from the Hub:

    import open_clip

    model, preprocess = open_clip.create_model_from_pretrained('hf-hub:laion/CLIP-ViT-g-14-laion2B-s12B-b42K')
    tokenizer = open_clip.get_tokenizer('hf-hub:laion/CLIP-ViT-g-14-laion2B-s12B-b42K')

Once loaded, you can encode the image and text to do zero-shot image classification:

    import torch
    from PIL import Image
    import requests

    url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
    image = Image.open(requests.get(url, stream=True).raw)
    image = preprocess(image).unsqueeze(0)
    text = tokenizer(["a diagram", "a dog", "a cat"])

    with torch.no_grad(), torch.cuda.amp.autocast():
        image_features = model.encode_image(image)
        text_features = model.encode_text(text)
        image_features /= image_features.norm(dim=-1, keepdim=True)
        text_features /= text_features.norm(dim=-1, keepdim=True)

        text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

    print("Label probs:", text_probs)

It outputs the probability of each possible class:

    Label probs: tensor([[0.0020, 0.0034, 0.9946]])

If you want to load a specific OpenCLIP model, you can click `Use in OpenCLIP` in the model card and you will be given a working snippet!

##
Additional resources

- OpenCLIP repository
- OpenCLIP docs
- OpenCLIP models in the Hub

Update on GitHub