Creating General User Models from Computer Use
Human-computer interaction has long imagined technology that understands
us-from our preferences and habits, to the timing and purpose of our everyday
actions. Yet current user models remain fragmented, narrowly tailored to
specific apps, and incapable of the flexible reasoning required to fulfill
these visions. This paper presents an architecture for a general user model
(GUM) that learns about you by observing any interaction you have with your
computer. The GUM takes as input any unstructured observation of a user (e.g.,
device screenshots) and constructs confidence-weighted propositions that
capture that user knowledge and preferences. GUMs can infer that a user is
preparing for a wedding they're attending from messages with a friend. Or
recognize that a user is struggling with a collaborator's feedback on a draft
by observing multiple stalled edits and a switch to reading related work. GUMs
introduce an architecture that infers new propositions about a user from
multimodal observations, retrieves related propositions for context, and
continuously revises existing propositions. To illustrate the breadth of
applications that GUMs enable, we demonstrate how they augment chat-based
assistants with context, manage OS notifications to selectively surface
important information, and enable interactive agents that adapt to preferences
across apps. We also instantiate proactive assistants (GUMBOs) that discover
and execute useful suggestions on a user's behalf using their GUM. In our
evaluations, we find that GUMs make calibrated and accurate inferences about
users, and that assistants built on GUMs proactively identify and perform
actions that users wouldn't think to request explicitly. Altogether, GUMs
introduce methods that leverage multimodal models to understand unstructured
context, enabling long-standing visions of HCI and entirely new interactive
systems that anticipate user needs.
682acae37cb97ecc4a4cd8f0
Omar Shaikh
682acae37cb97ecc4a4cd8f1
Shardul Sapkota
682acae37cb97ecc4a4cd8f2
Shan Rizvi
682acae37cb97ecc4a4cd8f3
Eric Horvitz
682acae37cb97ecc4a4cd8f4
Joon Sung Park
682acae37cb97ecc4a4cd8f5
Diyi Yang
682acae37cb97ecc4a4cd8f6
Michael S. Bernstein
Creating General User Models from Computer Use
Human-computer interaction has long imagined technology that understands
us-from our preferences and habits, to the timing and purpose of our everyday
actions. Yet current user models remain fragmented, narrowly tailored to
specific apps, and incapable of the flexible reasoning required to fulfill
these visions. This paper presents an architecture for a general user model
(GUM) that learns about you by observing any interaction you have with your
computer. The GUM takes as input any unstructured observation of a user (e.g.,
device screenshots) and constructs confidence-weighted propositions that
capture that user knowledge and preferences. GUMs can infer that a user is
preparing for a wedding they're attending from messages with a friend. Or
recognize that a user is struggling with a collaborator's feedback on a draft
by observing multiple stalled edits and a switch to reading related work. GUMs
introduce an architecture that infers new propositions about a user from
multimodal observations, retrieves related propositions for context, and
continuously revises existing propositions. To illustrate the breadth of
applications that GUMs enable, we demonstrate how they augment chat-based
assistants with context, manage OS notifications to selectively surface
important information, and enable interactive agents that adapt to preferences
across apps. We also instantiate proactive assistants (GUMBOs) that discover
and execute useful suggestions on a user's behalf using their GUM. In our
evaluations, we find that GUMs make calibrated and accurate inferences about
users, and that assistants built on GUMs proactively identify and perform
actions that users wouldn't think to request explicitly. Altogether, GUMs
introduce methods that leverage multimodal models to understand unstructured
context, enabling long-standing visions of HCI and entirely new interactive
systems that anticipate user needs.
GUMs, a multimodal user model architecture, learns user preferences and habits from unstructured observations to enhance chat-based assistants and interactive agents by inferring context and making proactive suggestions.
multimodal models unstructured context user model propositions inferring context proactive assistants interactive agents