#
AltCLIP

##
概要

AltCLIPモデルは、「AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities」という論文でZhongzhi Chen、Guang Liu、Bo-Wen Zhang、Fulong Ye、Qinghong Yang、Ledell Wuによって提案されました。AltCLIP（CLIPの言語エンコーダーの代替）は、様々な画像-テキストペアおよびテキスト-テキストペアでトレーニングされたニューラルネットワークです。CLIPのテキストエンコーダーを事前学習済みの多言語テキストエンコーダーXLM-Rに置き換えることで、ほぼ全てのタスクでCLIPに非常に近い性能を得られ、オリジナルのCLIPの能力を多言語理解などに拡張しました。

論文の要旨は以下の通りです：

この研究では、強力なバイリンガルマルチモーダル表現モデルを訓練するための概念的に単純で効果的な方法を提案します。OpenAIによってリリースされたマルチモーダル表現モデルCLIPから開始し、そのテキストエンコーダを事前学習済みの多言語テキストエンコーダXLM-Rに交換し、教師学習と対照学習からなる2段階のトレーニングスキーマを用いて言語と画像の表現を整合させました。幅広いタスクの評価を通じて、我々の方法を検証します。ImageNet-CN、Flicker30k-CN、COCO-CNを含む多くのタスクで新たな最先端の性能を達成しました。さらに、ほぼすべてのタスクでCLIPに非常に近い性能を得ており、これはCLIPのテキストエンコーダを変更するだけで、多言語理解などの拡張を実現できることを示唆しています。

このモデルはjongjyhにより提供されました。

##
使用上のヒントと使用例

AltCLIPの使用方法はCLIPに非常に似ています。CLIPとの違いはテキストエンコーダーにあります。私たちはカジュアルアテンションではなく双方向アテンションを使用し、XLM-Rの[CLS]トークンをテキスト埋め込みを表すものとして取ることに留意してください。

AltCLIPはマルチモーダルな視覚言語モデルです。これは画像とテキストの類似度や、ゼロショット画像分類に使用できます。AltCLIPはViTのようなTransformerを使用して視覚的特徴を、双方向言語モデルを使用してテキスト特徴を取得します。テキストと視覚の両方の特徴は、同一の次元を持つ潜在空間に射影されます。射影された画像とテキスト特徴間のドット積が類似度スコアとして使用されます。

Transformerエンコーダーに画像を与えるには、各画像を固定サイズの重複しないパッチの系列に分割し、それらを線形に埋め込みます。画像全体を表現するための[CLS]トークンが追加されます。著者は絶対位置埋め込みも追加し、結果として得られるベクトルの系列を標準的なTransformerエンコーダーに供給します。CLIPImageProcessorを使用して、モデルのために画像のサイズ変更（または拡大縮小）と正規化を行うことができます。

AltCLIPProcessorは、テキストのエンコードと画像の前処理を両方行うために、CLIPImageProcessorと`XLMRobertaTokenizer`を単一のインスタンスにラップします。以下の例は、AltCLIPProcessorとAltCLIPModelを使用して画像-テキスト類似スコアを取得する方法を示しています。

    >>> from PIL import Image
    >>> import requests

    >>> from transformers import AltCLIPModel, AltCLIPProcessor

    >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
    >>> processor = AltCLIPProcessor.from_pretrained("BAAI/AltCLIP")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

    >>> outputs = model(**inputs)
    >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
    >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

このモデルは`CLIPModel`をベースにしており、オリジナルのCLIPと同じように使用してください。

##
AltCLIPConfig

###
class transformers.AltCLIPConfig

< source >

( text_config = None vision_config = None projection_dim = 768 logit_scale_init_value = 2.6592 **kwargs )

Parameters

- text_config (`dict`, optional) — Dictionary of configuration options used to initialize AltCLIPTextConfig.
- vision_config (`dict`, optional) — Dictionary of configuration options used to initialize AltCLIPVisionConfig.
- projection_dim (`int`, optional, defaults to 768) — Dimensionality of text and vision projection layers.
- logit_scale_init_value (`float`, optional, defaults to 2.6592) — The initial value of the logit_scale parameter. Default is used as per the original CLIP implementation.
- kwargs (optional) — Dictionary of keyword arguments.

This is the configuration class to store the configuration of a AltCLIPModel. It is used to instantiate an AltCLIP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AltCLIP BAAI/AltCLIP architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import AltCLIPConfig, AltCLIPModel

    >>> # Initializing a AltCLIPConfig with BAAI/AltCLIP style configuration
    >>> configuration = AltCLIPConfig()

    >>> # Initializing a AltCLIPModel (with random weights) from the BAAI/AltCLIP style configuration
    >>> model = AltCLIPModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

    >>> # We can also initialize a AltCLIPConfig from a AltCLIPTextConfig and a AltCLIPVisionConfig

    >>> # Initializing a AltCLIPText and AltCLIPVision configuration
    >>> config_text = AltCLIPTextConfig()
    >>> config_vision = AltCLIPVisionConfig()

    >>> config = AltCLIPConfig.from_text_vision_configs(config_text, config_vision)

####
from_text_vision_configs

< source >

( text_config: AltCLIPTextConfig vision_config: AltCLIPVisionConfig **kwargs ) →
AltCLIPConfig

Returns

AltCLIPConfig

An instance of a configuration object

Instantiate a AltCLIPConfig (or a derived class) from altclip text model configuration and altclip vision model configuration.

##
AltCLIPTextConfig

###
class transformers.AltCLIPTextConfig

< source >

( vocab_size = 250002 hidden_size = 1024 num_hidden_layers = 24 num_attention_heads = 16 intermediate_size = 4096 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 514 type_vocab_size = 1 initializer_range = 0.02 initializer_factor = 0.02 layer_norm_eps = 1e-05 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True project_dim = 768 **kwargs )

Parameters

- vocab_size (`int`, optional, defaults to 250002) — Vocabulary size of the AltCLIP model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling AltCLIPTextModel.
- hidden_size (`int`, optional, defaults to 1024) — Dimensionality of the encoder layers and the pooler layer.
- num_hidden_layers (`int`, optional, defaults to 24) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 16) — Number of attention heads for each attention layer in the Transformer encoder.
- intermediate_size (`int`, optional, defaults to 4096) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
- hidden_act (`str` or `Callable`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"silu"` and `"gelu_new"` are supported.
- hidden_dropout_prob (`float`, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
- attention_probs_dropout_prob (`float`, optional, defaults to 0.1) — The dropout ratio for the attention probabilities.
- max_position_embeddings (`int`, optional, defaults to 514) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
- type_vocab_size (`int`, optional, defaults to 1) — The vocabulary size of the `token_type_ids` passed when calling AltCLIPTextModel
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- initializer_factor (`float`, optional, defaults to 0.02) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- layer_norm_eps (`float`, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
- pad_token_id (`int`, optional, defaults to 1) — The id of the padding token.
- bos_token_id (`int`, optional, defaults to 0) — The id of the beginning-of-sequence token.
- eos_token_id (`Union[int, List[int]]`, optional, defaults to 2) — The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.
- position_embedding_type (`str`, optional, defaults to `"absolute"`) — Type of position embedding. Choose one of `"absolute"`, `"relative_key"`, `"relative_key_query"`. For positional embeddings use `"absolute"`. For more information on `"relative_key"`, please refer to Self-Attention with Relative Position Representations (Shaw et al.). For more information on `"relative_key_query"`, please refer to Method 4 in Improve Transformer Models with Better Relative Position Embeddings (Huang et al.).
- use_cache (`bool`, optional, defaults to `True`) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`.
- project_dim (`int`, optional, defaults to 768) — The dimensions of the teacher model before the mapping layer.

This is the configuration class to store the configuration of a AltCLIPTextModel. It is used to instantiate a AltCLIP text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AltCLIP BAAI/AltCLIP architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Examples:

    >>> from transformers import AltCLIPTextModel, AltCLIPTextConfig

    >>> # Initializing a AltCLIPTextConfig with BAAI/AltCLIP style configuration
    >>> configuration = AltCLIPTextConfig()

    >>> # Initializing a AltCLIPTextModel (with random weights) from the BAAI/AltCLIP style configuration
    >>> model = AltCLIPTextModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
AltCLIPVisionConfig

###
class transformers.AltCLIPVisionConfig

< source >

( hidden_size = 768 intermediate_size = 3072 projection_dim = 512 num_hidden_layers = 12 num_attention_heads = 12 num_channels = 3 image_size = 224 patch_size = 32 hidden_act = 'quick_gelu' layer_norm_eps = 1e-05 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 **kwargs )

Parameters

- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- intermediate_size (`int`, optional, defaults to 3072) — Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.
- projection_dim (`int`, optional, defaults to 512) — Dimensionality of text and vision projection layers.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- num_channels (`int`, optional, defaults to 3) — The number of input channels.
- image_size (`int`, optional, defaults to 224) — The size (resolution) of each image.
- patch_size (`int`, optional, defaults to 32) — The size (resolution) of each patch.
- hidden_act (`str` or `function`, optional, defaults to `"quick_gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported.
- layer_norm_eps (`float`, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
- attention_dropout (`float`, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- initializer_factor (`float`, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).

This is the configuration class to store the configuration of a AltCLIPModel. It is used to instantiate an AltCLIP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AltCLIP BAAI/AltCLIP architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import AltCLIPVisionConfig, AltCLIPVisionModel

    >>> # Initializing a AltCLIPVisionConfig with BAAI/AltCLIP style configuration
    >>> configuration = AltCLIPVisionConfig()

    >>> # Initializing a AltCLIPVisionModel (with random weights) from the BAAI/AltCLIP style configuration
    >>> model = AltCLIPVisionModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
AltCLIPProcessor

###
class transformers.AltCLIPProcessor

< source >

( image_processor = None tokenizer = None )

Parameters

- image_processor (CLIPImageProcessor, optional) — The image processor is a required input.
- tokenizer (`XLMRobertaTokenizerFast`, optional) — The tokenizer is a required input.

Constructs a AltCLIP processor which wraps a CLIP image processor and a XLM-Roberta tokenizer into a single processor.

AltCLIPProcessor offers all the functionalities of CLIPImageProcessor and `XLMRobertaTokenizerFast`. See the `__call__()` and decode() for more information.

####
batch_decode

< source >

( *args **kwargs )

This method forwards all its arguments to XLMRobertaTokenizerFast’s batch_decode(). Please refer to the docstring of this method for more information.

####
decode

< source >

( *args **kwargs )

This method forwards all its arguments to XLMRobertaTokenizerFast’s decode(). Please refer to the docstring of this method for more information.

##
AltCLIPModel

###
class transformers.AltCLIPModel

< source >

( config: AltCLIPConfig )

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None token_type_ids: typing.Optional[torch.Tensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None interpolate_pos_encoding: bool = False return_dict: typing.Optional[bool] = None ) →
`transformers.models.altclip.modeling_altclip.AltCLIPOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, optional) — The tensors corresponding to the input images. Pixel values can be obtained using `{image_processor_class}`. See `{image_processor_class}.__call__` for details (`{processor_class}` uses `{image_processor_class}` for processing images).
- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- token_type_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- return_loss (`bool`, optional) — Whether or not to return the contrastive loss.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.altclip.modeling_altclip.AltCLIPOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.altclip.modeling_altclip.AltCLIPOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (AltCLIPConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `return_loss` is `True`) — Contrastive loss for image-text similarity.
- logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`) — The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores.
- logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`) — The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores.
- text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`) — The text embeddings obtained by applying the projection layer to the pooled output of AltCLIPTextModel.
- image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`) — The image embeddings obtained by applying the projection layer to the pooled output of AltCLIPVisionModel.
- text_model_output (`BaseModelOutputWithPooling`) — The output of the AltCLIPTextModel.
- vision_model_output (`BaseModelOutputWithPooling`) — The output of the AltCLIPVisionModel.

The AltCLIPModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, AltCLIPModel

    >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
    >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)
    >>> inputs = processor(
    ...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
    ... )
    >>> outputs = model(**inputs)
    >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
    >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

####
get_text_features

< source >

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None token_type_ids = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

Parameters

- input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- position_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- token_type_ids (``of shape`(batch_size, sequence_length)`) -- Segment token indices to indicate first and second portions of the inputs. Indices are selected in` [0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The text embeddings obtained by applying the projection layer to the pooled output of AltCLIPTextModel.

Examples:

    >>> from transformers import AutoProcessor, AltCLIPModel

    >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
    >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
    >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
    >>> text_features = model.get_text_features(**inputs)

####
get_image_features

< source >

( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None interpolate_pos_encoding: bool = False return_dict: typing.Optional[bool] = None ) →
image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, optional) — The tensors corresponding to the input images. Pixel values can be obtained using `{image_processor_class}`. See `{image_processor_class}.__call__` for details (`{processor_class}` uses `{image_processor_class}` for processing images).
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The image embeddings obtained by applying the projection layer to the pooled output of AltCLIPVisionModel.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, AltCLIPModel

    >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
    >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)
    >>> inputs = processor(images=image, return_tensors="pt")
    >>> image_features = model.get_image_features(**inputs)

##
AltCLIPTextModel

###
class transformers.AltCLIPTextModel

< source >

( config )

####
forward

< source >

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None ) →
`transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- encoder_hidden_states (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.
- encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.

Returns

`transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection` or `tuple(torch.FloatTensor)`

A `transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (AltCLIPConfig) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- projection_state (`tuple(torch.FloatTensor)`, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` of shape `(batch_size,config.project_dim)`.

  Text embeddings before the projection layer, used to mimic the last hidden state of the teacher encoder.

The AltCLIPTextModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoProcessor, AltCLIPTextModel

    >>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP")
    >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")

    >>> texts = ["it's a cat", "it's a dog"]

    >>> inputs = processor(text=texts, padding=True, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> last_hidden_state = outputs.last_hidden_state
    >>> pooled_output = outputs.pooler_output  # pooled CLS states

##
AltCLIPVisionModel

###
class transformers.AltCLIPVisionModel

< source >

( config: AltCLIPVisionConfig )

####
forward

< source >

( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None interpolate_pos_encoding: bool = False return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.BaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, optional) — The tensors corresponding to the input images. Pixel values can be obtained using `{image_processor_class}`. See `{image_processor_class}.__call__` for details (`{processor_class}` uses `{image_processor_class}` for processing images).
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.BaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (AltCLIPConfig) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The AltCLIPVisionModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, AltCLIPVisionModel

    >>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP")
    >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> inputs = processor(images=image, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> last_hidden_state = outputs.last_hidden_state
    >>> pooled_output = outputs.pooler_output  # pooled CLS states

< > Update on GitHub