EcoAssistant: Using LLM Assistant More Affordably and Accurately
Today, users ask Large language models (LLMs) as assistants to answer queries
that require external knowledge; they ask about the weather in a specific city,
about stock prices, and even about where specific locations are within their
neighborhood. These queries require the LLM to produce code that invokes
external APIs to answer the user's question, yet LLMs rarely produce correct
code on the first try, requiring iterative code refinement upon execution
results. In addition, using LLM assistants to support high query volumes can be
expensive. In this work, we contribute a framework, EcoAssistant, that enables
LLMs to answer code-driven queries more affordably and accurately. EcoAssistant
contains three components. First, it allows the LLM assistants to converse with
an automatic code executor to iteratively refine code or to produce answers
based on the execution results. Second, we use a hierarchy of LLM assistants,
which attempts to answer the query with weaker, cheaper LLMs before backing off
to stronger, expensive ones. Third, we retrieve solutions from past successful
queries as in-context demonstrations to help subsequent queries. Empirically,
we show that EcoAssistant offers distinct advantages for affordability and
accuracy, surpassing GPT-4 by 10 points of success rate with less than 50% of
GPT-4's cost.
651f60545ff73c3cec5678ee
Jieyu Zhang
admin_assigned
2023-10-09T10:02:53.745Z
640131b08ba76abe4b71b5d0
/avatars/2288b96a9a0ae8f584768f54e098def1.svg
Jieyu Zhang
user
jieyuz2
651f60545ff73c3cec5678ef
Ranjay Krishna
651f60545ff73c3cec5678f0
Ahmed H. Awadallah
admin_assigned
2023-10-10T08:28:33.399Z
6398c705cc8635e998d3c874
/avatars/3f3f56021cd74c561bde118d6ad7439d.svg
Ahmed Awadallah
user
aawadall
651f60545ff73c3cec5678f1
Chi Wang
admin_assigned
2023-10-10T08:29:15.329Z
640baf807047b6c332a912e1
/avatars/830fd35aff114eedf68c4f9eeb9fe944.svg
Chi Wang
user
sonichi
EcoAssistant: Using LLM Assistant More Affordably and Accurately
Today, users ask Large language models (LLMs) as assistants to answer queries
that require external knowledge; they ask about the weather in a specific city,
about stock prices, and even about where specific locations are within their
neighborhood. These queries require the LLM to produce code that invokes
external APIs to answer the user's question, yet LLMs rarely produce correct
code on the first try, requiring iterative code refinement upon execution
results. In addition, using LLM assistants to support high query volumes can be
expensive. In this work, we contribute a framework, EcoAssistant, that enables
LLMs to answer code-driven queries more affordably and accurately. EcoAssistant
contains three components. First, it allows the LLM assistants to converse with
an automatic code executor to iteratively refine code or to produce answers
based on the execution results. Second, we use a hierarchy of LLM assistants,
which attempts to answer the query with weaker, cheaper LLMs before backing off
to stronger, expensive ones. Third, we retrieve solutions from past successful
queries as in-context demonstrations to help subsequent queries. Empirically,
we show that EcoAssistant offers distinct advantages for affordability and
accuracy, surpassing GPT-4 by 10 points of success rate with less than 50% of
GPT-4's cost.
EcoAssistant framework enables more affordable and accurate responses from LLMs through iterative code refinement, LLM hierarchy, and solution retrieval.
Large language models LLMs automatic code executor hierarchical LLMs in-context demonstrations