Grammar Prompting for Domain-Specific Language Generation with Large
  Language Models
Large language models (LLMs) can learn to perform a wide range of natural
language tasks from just a handful of in-context examples. However, for
generating strings from highly structured languages (e.g., semantic parsing to
complex domain-specific languages), it is challenging for the LLM to generalize
from just a few exemplars. We explore grammar prompting as a simple
approach for enabling LLMs to use external knowledge and domain-specific
constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
during in-context learning. Grammar prompting augments each demonstration
example with a specialized grammar that is minimally sufficient for generating
the particular output example, where the specialized grammar is a subset of the
full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
test input, and then generates the output according to the rules of the
grammar. Experiments demonstrate that grammar prompting can enable LLMs to
perform competitively on a diverse set of DSL generation tasks, including
semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
molecule generation (SMILES).
6476c72ea0bde42d3f94b45c
Bailin Wang
claimed_verified
2023-06-01T05:54:35.403Z
642f4bfbc953ca48ccd162f2
/avatars/390662485bbf2bbe3454ba9f416d1536.svg
Bailin
user
bailin28
6476c72ea0bde42d3f94b45d
Zi Wang
6476c72ea0bde42d3f94b45e
Xuezhi Wang
6476c72ea0bde42d3f94b45f
Yuan Cao
claimed_verified
2023-06-01T05:54:30.844Z
6433a8c37b8247480105cf7f
/avatars/a19536f3f3b8eae3609648bea112b381.svg
Yuan Cao
user
caoyuan33
6476c72ea0bde42d3f94b460
Rif A. Saurous
6476c72ea0bde42d3f94b461
Yoon Kim
Grammar Prompting for Domain-Specific Language Generation with Large
  Language Models
Large language models (LLMs) can learn to perform a wide range of natural
language tasks from just a handful of in-context examples. However, for
generating strings from highly structured languages (e.g., semantic parsing to
complex domain-specific languages), it is challenging for the LLM to generalize
from just a few exemplars. We explore grammar prompting as a simple
approach for enabling LLMs to use external knowledge and domain-specific
constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
during in-context learning. Grammar prompting augments each demonstration
example with a specialized grammar that is minimally sufficient for generating
the particular output example, where the specialized grammar is a subset of the
full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
test input, and then generates the output according to the rules of the
grammar. Experiments demonstrate that grammar prompting can enable LLMs to
perform competitively on a diverse set of DSL generation tasks, including
semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
molecule generation (SMILES).
grammar prompting Large language models (LLMs) pre-context examples semantic parsing domain-specific languages (DSL) Backus--Naur Form (BNF) specialized grammar DSL grammar SMCalFlow Overnight GeoQuery PDDL planning molecule generation (SMILES)