Text-to-CAD Generation Through Infusing Visual Feedback in Large
  Language Models
Creating Computer-Aided Design (CAD) models requires significant expertise
and effort. Text-to-CAD, which converts textual descriptions into CAD
parametric sequences, is crucial in streamlining this process. Recent studies
have utilized ground-truth parametric sequences, known as sequential signals,
as supervision to achieve this goal. However, CAD models are inherently
multimodal, comprising parametric sequences and corresponding rendered visual
objects. Besides,the rendering process from parametric sequences to visual
objects is many-to-one. Therefore, both sequential and visual signals are
critical for effective training. In this work, we introduce CADFusion, a
framework that uses Large Language Models (LLMs) as the backbone and alternates
between two training stages: the sequential learning (SL) stage and the visual
feedback (VF) stage. In the SL stage, we train LLMs using ground-truth
parametric sequences, enabling the generation of logically coherent parametric
sequences. In the VF stage, we reward parametric sequences that render into
visually preferred objects and penalize those that do not, allowing LLMs to
learn how rendered visual objects are perceived and evaluated. These two stages
alternate throughout the training, ensuring balanced learning and preserving
benefits of both signals. Experiments demonstrate that CADFusion significantly
improves performance, both qualitatively and quantitatively.
67a33e60b793ca5296f2a6d1
Ruiyu Wang
claimed_verified
2025-02-05T13:37:40.966Z
67a30a243d5a32b36c7d7d0b
/avatars/4aa43f0d79797be1063f246d63638a85.svg
Ruiyu Wang
user
rywang37
67a33e60b793ca5296f2a6d2
Yu Yuan
67a33e60b793ca5296f2a6d3
Shizhao Sun
claimed_verified
2025-02-06T14:15:14.093Z
63eb00a191a1b8ec4fbba2a9
/avatars/0cc7cf9b6d05337603f700e0d592edf5.svg
ShizhaoSun
user
ShizhaoSun
67a33e60b793ca5296f2a6d4
Jiang Bian
claimed_verified
2025-06-19T09:11:15.411Z
683843c8a9bdf98f92657042
/avatars/4983876f615fa830cfc8f2cb5a00fa08.svg
Jiang Bian
user
jiangbian22
Text-to-CAD Generation Through Infusing Visual Feedback in Large
  Language Models
Creating Computer-Aided Design (CAD) models requires significant expertise
and effort. Text-to-CAD, which converts textual descriptions into CAD
parametric sequences, is crucial in streamlining this process. Recent studies
have utilized ground-truth parametric sequences, known as sequential signals,
as supervision to achieve this goal. However, CAD models are inherently
multimodal, comprising parametric sequences and corresponding rendered visual
objects. Besides,the rendering process from parametric sequences to visual
objects is many-to-one. Therefore, both sequential and visual signals are
critical for effective training. In this work, we introduce CADFusion, a
framework that uses Large Language Models (LLMs) as the backbone and alternates
between two training stages: the sequential learning (SL) stage and the visual
feedback (VF) stage. In the SL stage, we train LLMs using ground-truth
parametric sequences, enabling the generation of logically coherent parametric
sequences. In the VF stage, we reward parametric sequences that render into
visually preferred objects and penalize those that do not, allowing LLMs to
learn how rendered visual objects are perceived and evaluated. These two stages
alternate throughout the training, ensuring balanced learning and preserving
benefits of both signals. Experiments demonstrate that CADFusion significantly
improves performance, both qualitatively and quantitatively.
CADFusion uses alternating sequential and visual training stages with LLMs to enhance the generation of parametric sequences for CAD models from textual descriptions.
Large Language Models (LLMs) sequential learning visual feedback CADFusion parametric sequences CAD models text-to-CAD