StructLM: Towards Building Generalist Models for Structured Knowledge
  Grounding
Structured data sources, such as tables, graphs, and databases, are
ubiquitous knowledge sources. Despite the demonstrated capabilities of large
language models (LLMs) on plain text, their proficiency in interpreting and
utilizing structured data remains limited. Our investigation reveals a notable
deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags
behind state-of-the-art (SoTA) model by an average of 35%. To augment the
Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a
comprehensive instruction tuning dataset comprising 1.1 million examples.
Utilizing this dataset, we train a series of models, referred to as StructLM,
based on the Code-LLaMA architecture, ranging from 7B to 34B parameters. Our
StructLM series surpasses task-specific models on 14 out of 18 evaluated
datasets and establishes new SoTA achievements on 7 SKG tasks. Furthermore,
StructLM demonstrates exceptional generalization across 6 novel SKG tasks.
Contrary to expectations, we observe that scaling model size offers marginal
benefits, with StructLM-34B showing only slight improvements over StructLM-7B.
This suggests that structured knowledge grounding is still a challenging task
and requires more innovative design to push to a new level.
65dd73ce3ae2cf0ab2138a19
Alex Zhuang
admin_assigned
2024-02-27T10:35:06.206Z
6498eecc3756a91498f08f60
/avatars/d92a98535d212c73b9249aa48c717f5e.svg
Zhuang
user
AlexZhuang
65dd73ce3ae2cf0ab2138a1a
Ge Zhang
claimed_verified
2024-02-27T08:11:45.125Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
65dd73ce3ae2cf0ab2138a1b
Tianyu Zheng
claimed_verified
2024-02-27T08:11:37.158Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
65dd73ce3ae2cf0ab2138a1c
Xinrun Du
claimed_verified
2024-05-31T06:53:56.569Z
654907a4a1faff97850c4eff
/avatars/458c90151614bc7f116943b6e67d6b8a.svg
du
user
dododododo
65dd73ce3ae2cf0ab2138a1d
Junjie Wang
claimed_verified
2024-02-28T08:28:13.932Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
65dd73ce3ae2cf0ab2138a1e
Weiming Ren
admin_assigned
2024-02-27T10:35:43.147Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
65dd73ce3ae2cf0ab2138a1f
Stephen W. Huang
claimed_verified
2024-10-13T20:26:13.768Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
65dd73ce3ae2cf0ab2138a20
Jie Fu
claimed_verified
2024-02-27T09:04:26.312Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
65dd73ce3ae2cf0ab2138a21
Xiang Yue
admin_assigned
2024-02-27T10:36:34.221Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
65dd73ce3ae2cf0ab2138a22
Wenhu Chen
admin_assigned
2024-02-27T10:36:40.228Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
StructLM: Towards Building Generalist Models for Structured Knowledge
  Grounding
Structured data sources, such as tables, graphs, and databases, are
ubiquitous knowledge sources. Despite the demonstrated capabilities of large
language models (LLMs) on plain text, their proficiency in interpreting and
utilizing structured data remains limited. Our investigation reveals a notable
deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags
behind state-of-the-art (SoTA) model by an average of 35%. To augment the
Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a
comprehensive instruction tuning dataset comprising 1.1 million examples.
Utilizing this dataset, we train a series of models, referred to as StructLM,
based on the Code-LLaMA architecture, ranging from 7B to 34B parameters. Our
StructLM series surpasses task-specific models on 14 out of 18 evaluated
datasets and establishes new SoTA achievements on 7 SKG tasks. Furthermore,
StructLM demonstrates exceptional generalization across 6 novel SKG tasks.
Contrary to expectations, we observe that scaling model size offers marginal
benefits, with StructLM-34B showing only slight improvements over StructLM-7B.
This suggests that structured knowledge grounding is still a challenging task
and requires more innovative design to push to a new level.
A new series of models called StructLM improves LLMs' interpretation of structured data by surpassing existing models on 14 datasets and achieving SoTA on 7 tasks, indicating that structured knowledge grounding is still challenging.
large language models LLMs structured data Structured Knowledge Grounding SKG instruction tuning Code-LLaMA architecture model size scaling training dataset