Grammar Prompting for Domain-Specific Language Generation with Large
  Language Models
Large language models (LLMs) can learn to perform a wide range of natural
language tasks from just a handful of in-context examples. However, for
generating strings from highly structured languages (e.g., semantic parsing to
complex domain-specific languages), it is challenging for the LLM to generalize
from just a few exemplars. We explore grammar prompting as a simple
approach for enabling LLMs to use external knowledge and domain-specific
constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
during in-context learning. Grammar prompting augments each demonstration
example with a specialized grammar that is minimally sufficient for generating
the particular output example, where the specialized grammar is a subset of the
full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
test input, and then generates the output according to the rules of the
grammar. Experiments demonstrate that grammar prompting can enable LLMs to
perform competitively on a diverse set of DSL generation tasks, including
semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
molecule generation (SMILES).
6476c72ea0bde42d3f94b45c
Bailin Wang
claimed_verified
2023-06-01T05:54:35.403Z
642f4bfbc953ca48ccd162f2
/avatars/390662485bbf2bbe3454ba9f416d1536.svg
Bailin
user
bailin28
6476c72ea0bde42d3f94b45d
Zi Wang
6476c72ea0bde42d3f94b45e
Xuezhi Wang
6476c72ea0bde42d3f94b45f
Yuan Cao
claimed_verified
2023-06-01T05:54:30.844Z
6433a8c37b8247480105cf7f
/avatars/a19536f3f3b8eae3609648bea112b381.svg
Yuan Cao
user
caoyuan33
6476c72ea0bde42d3f94b460
Rif A. Saurous
6476c72ea0bde42d3f94b461
Yoon Kim
Grammar Prompting for Domain-Specific Language Generation with Large
  Language Models
Large language models (LLMs) can learn to perform a wide range of natural
language tasks from just a handful of in-context examples. However, for
generating strings from highly structured languages (e.g., semantic parsing to
complex domain-specific languages), it is challenging for the LLM to generalize
from just a few exemplars. We explore grammar prompting as a simple
approach for enabling LLMs to use external knowledge and domain-specific
constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
during in-context learning. Grammar prompting augments each demonstration
example with a specialized grammar that is minimally sufficient for generating
the particular output example, where the specialized grammar is a subset of the
full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
test input, and then generates the output according to the rules of the
grammar. Experiments demonstrate that grammar prompting can enable LLMs to
perform competitively on a diverse set of DSL generation tasks, including
semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
molecule generation (SMILES).
Grammar prompting enhances LLMs' performance on DSL generation tasks by incorporating domain-specific grammars expressed in BNF.
grammar prompting Backus--Naur Form (BNF) semantic parsing DSL generation PDDL planning molecule generation SMCalFlow Overnight GeoQuery SMILES