Iterative Graph Alignment
By compressing diverse narratives, LLMs go beyond memorization, achieving
intelligence by capturing generalizable causal relationships. However, they
suffer from local 'representation gaps' due to insufficient training data
diversity, limiting their real-world utility, especially in tasks requiring
strict alignment to rules. Traditional alignment methods relying on heavy human
annotations are inefficient and unscalable. Recent self-alignment techniques
also fall short, as they often depend on self-selection based prompting and
memorization-based learning. To address these issues, we introduce Iterative
Graph Alignment (IGA), an annotation-free rule-based alignment algorithm. A
teacher model (VLM) employs Iterative Graph Prompting (IGP) to create logical
graphs and reference answers. The student model (LLM) identifies local
knowledge gaps by attempting to align its responses with these references,
collaborating with helper models to generate diverse answers. These aligned
responses are then used for iterative supervised fine-tuning (SFT). Our
evaluations across five rule-based scenarios demonstrate IGP's effectiveness,
with a 73.12\% alignment improvement in Claude Sonnet 3.5, and
Llama3-8B-Instruct achieving an 86.20\% improvement, outperforming Claude
Sonnet 3.5 in rule-based alignment.
66d11f2a092cc13d0f36e075
Fangyuan Yu
claimed_verified
2024-08-30T07:18:05.215Z
64d98ef7a4839890b25eb78b
https://cdn-avatars.huggingface.co/v1/production/uploads/64d98ef7a4839890b25eb78b/215-CSVLl81z6CAq0ECWU.jpeg
Fangyuan Yu
user
Ksgk-fy
66d11f2a092cc13d0f36e076
Hardeep Singh Arora
66d11f2a092cc13d0f36e077
Matt Johnson
Iterative Graph Alignment
By compressing diverse narratives, LLMs go beyond memorization, achieving
intelligence by capturing generalizable causal relationships. However, they
suffer from local 'representation gaps' due to insufficient training data
diversity, limiting their real-world utility, especially in tasks requiring
strict alignment to rules. Traditional alignment methods relying on heavy human
annotations are inefficient and unscalable. Recent self-alignment techniques
also fall short, as they often depend on self-selection based prompting and
memorization-based learning. To address these issues, we introduce Iterative
Graph Alignment (IGA), an annotation-free rule-based alignment algorithm. A
teacher model (VLM) employs Iterative Graph Prompting (IGP) to create logical
graphs and reference answers. The student model (LLM) identifies local
knowledge gaps by attempting to align its responses with these references,
collaborating with helper models to generate diverse answers. These aligned
responses are then used for iterative supervised fine-tuning (SFT). Our
evaluations across five rule-based scenarios demonstrate IGP's effectiveness,
with a 73.12\% alignment improvement in Claude Sonnet 3.5, and
Llama3-8B-Instruct achieving an 86.20\% improvement, outperforming Claude
Sonnet 3.5 in rule-based alignment.
Iterative Graph Alignment (IGA) improves rule-based alignment in large language models by iteratively generating and fine-tuning responses without requiring human annotations.
LLMs representation gaps alignment methods Iterative Graph Alignment (IGA) Iterative Graph Prompting (IGP) teacher model student model helper models logical graphs reference answers iterative supervised fine-tuning (SFT) rule-based scenarios