Executable Knowledge Graphs for Replicating AI Research
Replicating AI research is a crucial yet challenging task for large language
model (LLM) agents. Existing approaches often struggle to generate executable
code, primarily due to insufficient background knowledge and the limitations of
retrieval-augmented generation (RAG) methods, which fail to capture latent
technical details hidden in referenced papers. Furthermore, previous approaches
tend to overlook valuable implementation-level code signals and lack structured
knowledge representations that support multi-granular retrieval and reuse. To
overcome these challenges, we propose Executable Knowledge Graphs (xKG), a
modular and pluggable knowledge base that automatically integrates technical
insights, code snippets, and domain-specific knowledge extracted from
scientific literature. When integrated into three agent frameworks with two
different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on
PaperBench, demonstrating its effectiveness as a general and extensible
solution for automated AI research replication. Code will released at
https://github.com/zjunlp/xKG.
68f6f0d824c4489363111892
Yujie Luo
68f6f0d824c4489363111893
Zhuoyun Yu
68f6f0d824c4489363111894
Xuehai Wang
68f6f0d824c4489363111895
Yuqi Zhu
68f6f0d824c4489363111896
Ningyu Zhang
claimed_verified
2025-10-22T15:46:00.599Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
68f6f0d824c4489363111897
Lanning Wei
68f6f0d824c4489363111898
Lun Du
68f6f0d824c4489363111899
Da Zheng
68f6f0d824c448936311189a
Huajun Chen
Executable Knowledge Graphs for Replicating AI Research
Replicating AI research is a crucial yet challenging task for large language
model (LLM) agents. Existing approaches often struggle to generate executable
code, primarily due to insufficient background knowledge and the limitations of
retrieval-augmented generation (RAG) methods, which fail to capture latent
technical details hidden in referenced papers. Furthermore, previous approaches
tend to overlook valuable implementation-level code signals and lack structured
knowledge representations that support multi-granular retrieval and reuse. To
overcome these challenges, we propose Executable Knowledge Graphs (xKG), a
modular and pluggable knowledge base that automatically integrates technical
insights, code snippets, and domain-specific knowledge extracted from
scientific literature. When integrated into three agent frameworks with two
different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on
PaperBench, demonstrating its effectiveness as a general and extensible
solution for automated AI research replication. Code will released at
https://github.com/zjunlp/xKG.
Executable Knowledge Graphs (xKG) enhance AI research replication by integrating technical insights and code snippets from scientific literature, improving performance in automated replication tasks.
large language model (LLM) retrieval-augmented generation (RAG) Executable Knowledge Graphs (xKG) technical insights code snippets domain-specific knowledge PaperBench o3-mini