The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in
  LLMs with Camlang
Large Language Models (LLMs) achieve gold-medal performance across many
benchmarks, yet it remains unclear whether such success reflects genuine
reasoning or pattern matching. From a cognitive science perspective, an
informative test is whether models can master an unfamiliar language through
explicit metalinguistic deductive learning, a paradigm where human learners can
reliably internalise grammatical systems through metalinguistic reasoning. We
address this question with Camlang, a novel constructed language that exhibits
naturalistic yet unattested feature combinations. Camlang consists of two
explicit resources, a grammar book and a bilingual dictionary, which mirror
adult second-language learning via explicit grammar rules and lexical lookup,
and enable us to disentangle errors in morpho-syntax, lexical semantics, and
sentence-level reasoning. Human experiments show that these resources are
sufficient for participants to acquire Camlang and successfully solve Camlang
tasks. To operationalise evaluation, we adapt CommonsenseQA into Camlang,
creating Camlang-CSQA-v0, the first task in a broader suite where solving
questions requires applying grammar rules and lexical mappings. Experimental
results show that GPT-5 achieves 98\% EM accuracy in English but only 47\% in
Camlang, far below human performance at 87\%, while other state-of-the-art
reasoning LLMs perform even worse. Human verification further reveals that most
model successes stem from shallow lexical alignment while GPT-5 shows emerging
metalinguistic awareness to a limited extent but not systematic grammatical
mastery as humans. Camlang establishes a cognitively grounded evaluation
paradigm that exposes fundamental gaps between current models and human
metalinguistic competence.
68b7c228295f15ff609113e0
Fenghua Liu
68b7c228295f15ff609113e1
Yulong Chen
claimed_verified
2025-09-03T08:49:48.933Z
627e07f953494d140baea7fd
/avatars/e6c4cc361f0cbcd5e27840d477cd06f3.svg
cyl
user
yulongchen
68b7c228295f15ff609113e2
Yixuan Liu
68b7c228295f15ff609113e3
Zhujun Jin
68b7c228295f15ff609113e4
Solomon Tsai
68b7c228295f15ff609113e5
Ming Zhong
claimed_verified
2025-09-04T08:44:17.240Z
61d53df2062444ea769d3b79
/avatars/fa771202368b6b2626a8fdf1c4369239.svg
Ming Zhong
user
MingZhong
The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in
  LLMs with Camlang
Large Language Models (LLMs) achieve gold-medal performance across many
benchmarks, yet it remains unclear whether such success reflects genuine
reasoning or pattern matching. From a cognitive science perspective, an
informative test is whether models can master an unfamiliar language through
explicit metalinguistic deductive learning, a paradigm where human learners can
reliably internalise grammatical systems through metalinguistic reasoning. We
address this question with Camlang, a novel constructed language that exhibits
naturalistic yet unattested feature combinations. Camlang consists of two
explicit resources, a grammar book and a bilingual dictionary, which mirror
adult second-language learning via explicit grammar rules and lexical lookup,
and enable us to disentangle errors in morpho-syntax, lexical semantics, and
sentence-level reasoning. Human experiments show that these resources are
sufficient for participants to acquire Camlang and successfully solve Camlang
tasks. To operationalise evaluation, we adapt CommonsenseQA into Camlang,
creating Camlang-CSQA-v0, the first task in a broader suite where solving
questions requires applying grammar rules and lexical mappings. Experimental
results show that GPT-5 achieves 98\% EM accuracy in English but only 47\% in
Camlang, far below human performance at 87\%, while other state-of-the-art
reasoning LLMs perform even worse. Human verification further reveals that most
model successes stem from shallow lexical alignment while GPT-5 shows emerging
metalinguistic awareness to a limited extent but not systematic grammatical
mastery as humans. Camlang establishes a cognitively grounded evaluation
paradigm that exposes fundamental gaps between current models and human
metalinguistic competence.
Camlang, a constructed language, is used to evaluate whether LLMs can master unfamiliar languages through metalinguistic reasoning, revealing that current models lack systematic grammatical mastery compared to humans.
Large Language Models LLMs metalinguistic deductive learning constructed language grammar book bilingual dictionary morpho-syntax lexical semantics sentence-level reasoning CommonsenseQA GPT-5 metalinguistic awareness