GRACE: Generative Representation Learning via Contrastive Policy
  Optimization
Prevailing methods for training Large Language Models (LLMs) as text encoders
rely on contrastive losses that treat the model as a black box function,
discarding its generative and reasoning capabilities in favor of static
embeddings. We introduce GRACE (Generative Representation Learning via
Contrastive Policy Optimization), a novel framework that reimagines contrastive
signals not as losses to be minimized, but as rewards that guide a generative
policy. In GRACE, the LLM acts as a policy that produces explicit,
human-interpretable rationales--structured natural language explanations of its
semantic understanding. These rationales are then encoded into high-quality
embeddings via mean pooling. Using policy gradient optimization, we train the
model with a multi-component reward function that maximizes similarity between
query positive pairs and minimizes similarity with negatives. This transforms
the LLM from an opaque encoder into an interpretable agent whose reasoning
process is transparent and inspectable. On MTEB benchmark, GRACE yields broad
cross category gains: averaged over four backbones, the supervised setting
improves overall score by 11.5% over base models, and the unsupervised variant
adds 6.9%, while preserving general capabilities. This work treats contrastive
objectives as rewards over rationales, unifying representation learning with
generation to produce stronger embeddings and transparent rationales. The
model, data and code are available at https://github.com/GasolSun36/GRACE.
68e68dbc975ac4c405ef23a0
Jiashuo Sun
68e68dbc975ac4c405ef23a1
Shixuan Liu
68e68dbc975ac4c405ef23a2
Zhaochen Su
68e68dbc975ac4c405ef23a3
Xianrui Zhong
68e68dbc975ac4c405ef23a4
Pengcheng Jiang
68e68dbc975ac4c405ef23a5
Bowen Jin
68e68dbc975ac4c405ef23a6
Peiran Li
68e68dbc975ac4c405ef23a7
Weijia Shi
68e68dbc975ac4c405ef23a8
Jiawei Han
GRACE: Generative Representation Learning via Contrastive Policy
  Optimization
Prevailing methods for training Large Language Models (LLMs) as text encoders
rely on contrastive losses that treat the model as a black box function,
discarding its generative and reasoning capabilities in favor of static
embeddings. We introduce GRACE (Generative Representation Learning via
Contrastive Policy Optimization), a novel framework that reimagines contrastive
signals not as losses to be minimized, but as rewards that guide a generative
policy. In GRACE, the LLM acts as a policy that produces explicit,
human-interpretable rationales--structured natural language explanations of its
semantic understanding. These rationales are then encoded into high-quality
embeddings via mean pooling. Using policy gradient optimization, we train the
model with a multi-component reward function that maximizes similarity between
query positive pairs and minimizes similarity with negatives. This transforms
the LLM from an opaque encoder into an interpretable agent whose reasoning
process is transparent and inspectable. On MTEB benchmark, GRACE yields broad
cross category gains: averaged over four backbones, the supervised setting
improves overall score by 11.5% over base models, and the unsupervised variant
adds 6.9%, while preserving general capabilities. This work treats contrastive
objectives as rewards over rationales, unifying representation learning with
generation to produce stronger embeddings and transparent rationales. The
model, data and code are available at https://github.com/GasolSun36/GRACE.
GRACE uses contrastive policy optimization to train LLMs as generative agents that produce interpretable rationales, improving embeddings and transparency.
contrastive losses generative representation learning contrastive policy optimization generative policy human-interpretable rationales mean pooling policy gradient optimization multi-component reward function MTEB benchmark supervised setting unsupervised variant representation learning generation