Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic
  Agents
Flowcharts are a critical tool for visualizing decision-making processes.
However, their non-linear structure and complex visual-textual relationships
make it challenging to interpret them using LLMs, as vision-language models
frequently hallucinate nonexistent connections and decision paths when
analyzing these diagrams. This leads to compromised reliability for automated
flowchart processing in critical domains such as logistics, health, and
engineering. We introduce the task of Fine-grained Flowchart Attribution, which
traces specific components grounding a flowchart referring LLM response.
Flowchart Attribution ensures the verifiability of LLM predictions and improves
explainability by linking generated responses to the flowchart's structure. We
propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post
hoc attribution through graph-based reasoning. It first segments the flowchart,
then converts it into a structured symbolic graph, and then employs an agentic
approach to dynamically interact with the graph, to generate attribution paths.
Additionally, we present FlowExplainBench, a novel benchmark for evaluating
flowchart attributions across diverse styles, domains, and question types.
Experimental results show that FlowPathAgent mitigates visual hallucinations in
LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our
proposed FlowExplainBench dataset.
6841009bdf863485e04879c8
Manan Suri
6841009bdf863485e04879c9
Puneet Mathur
extracted_pending
2025-06-05T02:27:41.018Z
65c16444d4c3b8dff2f0d78d
/avatars/4ed764c1657bd260d2a12ba61c111062.svg
Puneet Mathur
user
puneetm
6841009bdf863485e04879ca
Nedim Lipka
6841009bdf863485e04879cb
Franck Dernoncourt
claimed_verified
2025-06-05T07:26:52.205Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
6841009bdf863485e04879cc
Ryan A. Rossi
6841009bdf863485e04879cd
Vivek Gupta
6841009bdf863485e04879ce
Dinesh Manocha
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic
  Agents
Flowcharts are a critical tool for visualizing decision-making processes.
However, their non-linear structure and complex visual-textual relationships
make it challenging to interpret them using LLMs, as vision-language models
frequently hallucinate nonexistent connections and decision paths when
analyzing these diagrams. This leads to compromised reliability for automated
flowchart processing in critical domains such as logistics, health, and
engineering. We introduce the task of Fine-grained Flowchart Attribution, which
traces specific components grounding a flowchart referring LLM response.
Flowchart Attribution ensures the verifiability of LLM predictions and improves
explainability by linking generated responses to the flowchart's structure. We
propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post
hoc attribution through graph-based reasoning. It first segments the flowchart,
then converts it into a structured symbolic graph, and then employs an agentic
approach to dynamically interact with the graph, to generate attribution paths.
Additionally, we present FlowExplainBench, a novel benchmark for evaluating
flowchart attributions across diverse styles, domains, and question types.
Experimental results show that FlowPathAgent mitigates visual hallucinations in
LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our
proposed FlowExplainBench dataset.
FlowPathAgent, a neurosymbolic agent, enhances the reliability of LLM predictions for flowchart interpretation by tracing specific components and generating accurate attribution paths.
Flowcharts Fine-grained Flowchart Attribution FlowPathAgent graph-based reasoning symbolic graph neurosymbolic agent flowExplainBench flowchart QA