CogCoM: Train Large Vision-Language Models Diving into Details through
  Chain of Manipulations
Vision-Language Models (VLMs) have demonstrated their widespread viability
thanks to extensive training in aligning visual instructions to answers.
However, this conclusive alignment leads models to ignore critical visual
reasoning, and further result in failures on meticulous visual problems and
unfaithful responses. In this paper, we propose Chain of Manipulations, a
mechanism that enables VLMs to solve problems with a series of manipulations,
where each manipulation refers to an operation on the visual input, either from
intrinsic abilities (e.g., grounding) acquired through prior training or from
imitating human-like behaviors (e.g., zoom in). This mechanism encourages VLMs
to generate faithful responses with evidential visual reasoning, and permits
users to trace error causes in the interpretable paths. We thus train CogCoM, a
general 17B VLM with a memory-based compatible architecture endowed this
reasoning mechanism. Experiments show that our model achieves the
state-of-the-art performance across 8 benchmarks from 3 categories, and a
limited number of training steps with the data swiftly gains a competitive
performance. The code and data are publicly available at
https://github.com/THUDM/CogCoM.
65c2f87dd14c096f8a69bd08
Ji Qi
claimed_verified
2025-07-02T08:24:27.962Z
6864b9f90a269bace8c92164
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zSeWHZRCZtzZ13CzFrr5u.png
Ji Qi
user
miracle11121
65c2f87dd14c096f8a69bd09
Ming Ding
65c2f87dd14c096f8a69bd0a
Weihan Wang
claimed_verified
2025-07-02T12:23:41.021Z
63403c75b8b51e0098d3555c
/avatars/85725280b63788e387fe73319a54164d.svg
王维汉
user
mactavish91
65c2f87dd14c096f8a69bd0b
Yushi Bai
claimed_verified
2024-06-19T20:22:22.146Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
65c2f87dd14c096f8a69bd0c
Qingsong Lv
65c2f87dd14c096f8a69bd0d
Wenyi Hong
claimed_verified
2025-07-02T08:55:52.307Z
62ecd24cb8764c7738ef2793
/avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
Wenyi Hong
user
wenyi
65c2f87dd14c096f8a69bd0e
Bin Xu
65c2f87dd14c096f8a69bd0f
Lei Hou
65c2f87dd14c096f8a69bd10
Juanzi Li
65c2f87dd14c096f8a69bd11
Yuxiao Dong
65c2f87dd14c096f8a69bd12
Jie Tang
CogCoM: Train Large Vision-Language Models Diving into Details through
  Chain of Manipulations
Vision-Language Models (VLMs) have demonstrated their widespread viability
thanks to extensive training in aligning visual instructions to answers.
However, this conclusive alignment leads models to ignore critical visual
reasoning, and further result in failures on meticulous visual problems and
unfaithful responses. In this paper, we propose Chain of Manipulations, a
mechanism that enables VLMs to solve problems with a series of manipulations,
where each manipulation refers to an operation on the visual input, either from
intrinsic abilities (e.g., grounding) acquired through prior training or from
imitating human-like behaviors (e.g., zoom in). This mechanism encourages VLMs
to generate faithful responses with evidential visual reasoning, and permits
users to trace error causes in the interpretable paths. We thus train CogCoM, a
general 17B VLM with a memory-based compatible architecture endowed this
reasoning mechanism. Experiments show that our model achieves the
state-of-the-art performance across 8 benchmarks from 3 categories, and a
limited number of training steps with the data swiftly gains a competitive
performance. The code and data are publicly available at
https://github.com/THUDM/CogCoM.
Chain of Manipulations, an intrinsic and imitation-based reasoning mechanism, is introduced to enhance visual reasoning in Vision-Language Models, achieving state-of-the-art performance across benchmarks.
Vision-Language Models Chain of Manipulations visual reasoning grounding zoom in CogCoM memory-based architecture state-of-the-art performance benchmarks