DSPy: Compiling Declarative Language Model Calls into Self-Improving
  Pipelines
The ML community is rapidly exploring techniques for prompting language
models (LMs) and for stacking them into pipelines that solve complex tasks.
Unfortunately, existing LM pipelines are typically implemented using hard-coded
"prompt templates", i.e. lengthy strings discovered via trial and error. Toward
a more systematic approach for developing and optimizing LM pipelines, we
introduce DSPy, a programming model that abstracts LM pipelines as text
transformation graphs, i.e. imperative computational graphs where LMs are
invoked through declarative modules. DSPy modules are parameterized, meaning
they can learn (by creating and collecting demonstrations) how to apply
compositions of prompting, finetuning, augmentation, and reasoning techniques.
We design a compiler that will optimize any DSPy pipeline to maximize a given
metric. We conduct two case studies, showing that succinct DSPy programs can
express and optimize sophisticated LM pipelines that reason about math word
problems, tackle multi-hop retrieval, answer complex questions, and control
agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and
llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot
prompting (generally by over 25% and 65%, respectively) and pipelines with
expert-created demonstrations (by up to 5-46% and 16-40%, respectively). On top
of that, DSPy programs compiled to open and relatively small LMs like
770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely
on expert-written prompt chains for proprietary GPT-3.5. DSPy is available at
https://github.com/stanfordnlp/dspy
651f6316e3c86280559d2eb2
Omar Khattab
admin_assigned
2023-10-09T08:47:57.705Z
5f764762f1e7ef6e919a1e0f
/avatars/9445f6efa3f78a870e338988d9ce5573.svg
Omar Khattab
user
okhattab
651f6316e3c86280559d2eb3
Arnav Singhvi
admin_assigned
2023-10-09T08:48:05.597Z
64b835e3d2176850e0abadcd
/avatars/eb81b20d70fb9d467ba610f6a1733fbd.svg
Arnav Singhvi
user
arnavs11
651f6316e3c86280559d2eb4
Paridhi Maheshwari
admin_assigned
2023-10-09T08:48:13.348Z
640e34432f9c7b364d157408
/avatars/59552e26a13183607ebb668669476b39.svg
Paridhi Maheshwari
user
paridhi
651f6316e3c86280559d2eb5
Zhiyuan Zhang
651f6316e3c86280559d2eb6
Keshav Santhanam
admin_assigned
2023-10-09T08:48:44.241Z
633328b6f695c3df38d97434
/avatars/017e15cad6406653a9340cdbbee7dc51.svg
Keshav Santhanam
user
santhnm2
651f6316e3c86280559d2eb7
Sri Vardhamanan
651f6316e3c86280559d2eb8
Saiful Haq
651f6316e3c86280559d2eb9
Ashutosh Sharma
651f6316e3c86280559d2eba
Thomas T. Joshi
651f6316e3c86280559d2ebb
Hanna Moazam
admin_assigned
2023-10-09T08:49:44.690Z
6505eb335aa2ad0324c205b5
/avatars/982a90c601c9d5eb4f2c3828ebc106ee.svg
Hanna Moazam
user
hmoazam
651f6316e3c86280559d2ebc
Heather Miller
claimed_verified
2023-10-09T07:51:21.809Z
6520235b94f3a1a209c1bcc1
/avatars/49a4815846825cd1334fa080c6e71c5d.svg
Heather Miller
user
heathercmiller
651f6316e3c86280559d2ebd
Matei Zaharia
admin_assigned
2023-10-09T08:49:52.303Z
642509c6d476e4ad5568dd31
https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
Matei Zaharia
user
mateiz
651f6316e3c86280559d2ebe
Christopher Potts
admin_assigned
2023-10-09T08:50:04.621Z
62606790e3dd8b49f360f050
https://cdn-avatars.huggingface.co/v1/production/uploads/1650485126923-noauth.png
Chris Potts
user
cgpotts
DSPy: Compiling Declarative Language Model Calls into Self-Improving
  Pipelines
The ML community is rapidly exploring techniques for prompting language
models (LMs) and for stacking them into pipelines that solve complex tasks.
Unfortunately, existing LM pipelines are typically implemented using hard-coded
"prompt templates", i.e. lengthy strings discovered via trial and error. Toward
a more systematic approach for developing and optimizing LM pipelines, we
introduce DSPy, a programming model that abstracts LM pipelines as text
transformation graphs, i.e. imperative computational graphs where LMs are
invoked through declarative modules. DSPy modules are parameterized, meaning
they can learn (by creating and collecting demonstrations) how to apply
compositions of prompting, finetuning, augmentation, and reasoning techniques.
We design a compiler that will optimize any DSPy pipeline to maximize a given
metric. We conduct two case studies, showing that succinct DSPy programs can
express and optimize sophisticated LM pipelines that reason about math word
problems, tackle multi-hop retrieval, answer complex questions, and control
agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and
llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot
prompting (generally by over 25% and 65%, respectively) and pipelines with
expert-created demonstrations (by up to 5-46% and 16-40%, respectively). On top
of that, DSPy programs compiled to open and relatively small LMs like
770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely
on expert-written prompt chains for proprietary GPT-3.5. DSPy is available at
https://github.com/stanfordnlp/dspy
DSPy is a programming model for developing and optimizing language model pipelines by abstracting them as text transformation graphs with parameterized modules that learn through demonstrations.
language models LM pipelines prompt templates textual transformation graphs parameterized modules compiler reasoning techniques few-shot prompting agent loops math word problems multi-hop retrieval complex questions T5 llama2-13b-chat standard few-shot prompting