Guiding Language Models of Code with Global Context using Monitors
Language models of code (LMs) work well when the surrounding code in the
vicinity of generation provides sufficient context. This is not true when it
becomes necessary to use types or functionality defined in another module or
library, especially those not seen during training. LMs suffer from limited
awareness of such global context and end up hallucinating, e.g., using types
defined in other files incorrectly. Recent work tries to overcome this issue by
retrieving global information to augment the local context. However, this
bloats the prompt or requires architecture modifications and additional
training.
  Integrated development environments (IDEs) assist developers by bringing the
global context at their fingertips using static analysis. We extend this
assistance, enjoyed by developers, to the LMs. We propose a notion of monitors
that use static analysis in the background to guide the decoding. Unlike a
priori retrieval, static analysis is invoked iteratively during the entire
decoding process, providing the most relevant suggestions on demand. We
demonstrate the usefulness of our proposal by monitoring for type-consistent
use of identifiers whenever an LM generates code for object dereference.
  To evaluate our approach, we curate PragmaticCode, a dataset of open-source
projects with their development environments. On models of varying parameter
scale, we show that monitor-guided decoding consistently improves the ability
of an LM to not only generate identifiers that match the ground truth but also
improves compilation rates and agreement with ground truth. We find that LMs
with fewer parameters, when guided with our monitor, can outperform larger LMs.
With monitor-guided decoding, SantaCoder-1.1B achieves better compilation rate
and next-identifier match than the much larger text-davinci-003 model. The
datasets and code will be released at https://aka.ms/monitors4codegen .
6492611c7bcb175b58b69046
Lakshya A Agrawal
claimed_verified
2023-06-21T09:16:58.013Z
63bf0cbb24e75d1efdfdf8bd
/avatars/3bb13e989db31820734cbe6b13be130a.svg
Lakshya A Agrawal
user
LakshyAAAgrawal
6492611c7bcb175b58b69047
Aditya Kanade
extracted_pending
2025-05-02T12:51:57.419Z
649e62e953777d8950749c72
https://cdn-avatars.huggingface.co/v1/production/uploads/649e62e953777d8950749c72/PHigcJ-TQ7HXwriZkBhLV.jpeg
AK
user
adityakanade
6492611c7bcb175b58b69048
Navin Goyal
6492611c7bcb175b58b69049
Shuvendu K. Lahiri
6492611c7bcb175b58b6904a
Sriram K. Rajamani
extracted_confirmed
2023-06-21T03:48:09.988Z
63fb54480aab060792f71151
/avatars/97edbaed65e87a555bf05f770fa60b23.svg
Sriram Rajamani
user
sriramkr
Guiding Language Models of Code with Global Context using Monitors
Language models of code (LMs) work well when the surrounding code in the
vicinity of generation provides sufficient context. This is not true when it
becomes necessary to use types or functionality defined in another module or
library, especially those not seen during training. LMs suffer from limited
awareness of such global context and end up hallucinating, e.g., using types
defined in other files incorrectly. Recent work tries to overcome this issue by
retrieving global information to augment the local context. However, this
bloats the prompt or requires architecture modifications and additional
training.
  Integrated development environments (IDEs) assist developers by bringing the
global context at their fingertips using static analysis. We extend this
assistance, enjoyed by developers, to the LMs. We propose a notion of monitors
that use static analysis in the background to guide the decoding. Unlike a
priori retrieval, static analysis is invoked iteratively during the entire
decoding process, providing the most relevant suggestions on demand. We
demonstrate the usefulness of our proposal by monitoring for type-consistent
use of identifiers whenever an LM generates code for object dereference.
  To evaluate our approach, we curate PragmaticCode, a dataset of open-source
projects with their development environments. On models of varying parameter
scale, we show that monitor-guided decoding consistently improves the ability
of an LM to not only generate identifiers that match the ground truth but also
improves compilation rates and agreement with ground truth. We find that LMs
with fewer parameters, when guided with our monitor, can outperform larger LMs.
With monitor-guided decoding, SantaCoder-1.1B achieves better compilation rate
and next-identifier match than the much larger text-davinci-003 model. The
datasets and code will be released at https://aka.ms/monitors4codegen .
Using static analysis monitors during code generation improves the accuracy and compilation rates of language models by providing context-aware guidance.
language models of code static analysis decoding identifiers object dereference PragmaticCode compilation rates next-identifier match monitor-guided decoding SantaCoder-1.1B text-davinci-003