Sharingan: Extract User Action Sequence from Desktop Recordings
Video recordings of user activities, particularly desktop recordings, offer a
rich source of data for understanding user behaviors and automating processes.
However, despite advancements in Vision-Language Models (VLMs) and their
increasing use in video analysis, extracting user actions from desktop
recordings remains an underexplored area. This paper addresses this gap by
proposing two novel VLM-based methods for user action extraction: the Direct
Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and
the Differential Frame-Based Approach (DiffF), which incorporates explicit
frame differences detected via computer vision techniques. We evaluate these
methods using a basic self-curated dataset and an advanced benchmark adapted
from prior work. Our results show that the DF approach achieves an accuracy of
70% to 80% in identifying user actions, with the extracted action sequences
being re-playable though Robotic Process Automation. We find that while VLMs
show potential, incorporating explicit UI changes can degrade performance,
making the DF approach more reliable. This work represents the first
application of VLMs for extracting user action sequences from desktop
recordings, contributing new methods, benchmarks, and insights for future
research.
6736fbe36eee0fb7dad3f678
Yanting Chen
admin_assigned
2024-11-15T14:07:00.889Z
670d85fe6ee84c30545ea558
/avatars/516b00b81603e9a67bd2ba104f375ddb.svg
Yanting Chen
user
yantc
6736fbe36eee0fb7dad3f679
Yi Ren
admin_assigned
2024-11-15T14:09:21.884Z
6632323d2a4742700e976553
/avatars/fb6e017afe268f09e0028135d3b7dc7a.svg
Yi Ren
user
yiren0724
6736fbe36eee0fb7dad3f67a
Xiaoting Qin
admin_assigned
2024-11-15T14:09:14.845Z
662b0bc9c709a61df8291c0f
/avatars/16dd4d945e9fbef5ac889a8087101ded.svg
Xiaoting Qin
user
XiaotingQin
6736fbe36eee0fb7dad3f67b
Jue Zhang
admin_assigned
2024-11-15T14:09:07.328Z
65f40e43653c231cbaf7d1e4
/avatars/a42ac5454cbe175f04c3420fce90cad2.svg
Jue Zhang
user
JueZhang
6736fbe36eee0fb7dad3f67c
Kehong Yuan
6736fbe36eee0fb7dad3f67d
Lu Han
admin_assigned
2024-11-15T14:08:52.787Z
671f6ae374ec77d5788ad487
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wYkPCbD7kXrZgn426ozXP.png
Han
user
luhan-microsoft
6736fbe36eee0fb7dad3f67e
Qingwei Lin
admin_assigned
2024-11-15T14:08:34.679Z
652fc9f39bc50a6c0e435224
https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
Lin Qingwei
user
Eliblo1969
6736fbe36eee0fb7dad3f67f
Dongmei Zhang
admin_assigned
2024-11-15T14:08:22.520Z
66473d2c7abe6ad66e81a3dd
/avatars/82f40244806c06ffeaa1c4265e9725ea.svg
ZHANGDONGMEI
user
ZDM6426
6736fbe36eee0fb7dad3f680
Saravan Rajmohan
6736fbe36eee0fb7dad3f681
Qi Zhang
Sharingan: Extract User Action Sequence from Desktop Recordings
Video recordings of user activities, particularly desktop recordings, offer a
rich source of data for understanding user behaviors and automating processes.
However, despite advancements in Vision-Language Models (VLMs) and their
increasing use in video analysis, extracting user actions from desktop
recordings remains an underexplored area. This paper addresses this gap by
proposing two novel VLM-based methods for user action extraction: the Direct
Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and
the Differential Frame-Based Approach (DiffF), which incorporates explicit
frame differences detected via computer vision techniques. We evaluate these
methods using a basic self-curated dataset and an advanced benchmark adapted
from prior work. Our results show that the DF approach achieves an accuracy of
70% to 80% in identifying user actions, with the extracted action sequences
being re-playable though Robotic Process Automation. We find that while VLMs
show potential, incorporating explicit UI changes can degrade performance,
making the DF approach more reliable. This work represents the first
application of VLMs for extracting user action sequences from desktop
recordings, contributing new methods, benchmarks, and insights for future
research.
VLM-based methods, Direct Frame-Based Approach and Differential Frame-Based Approach, for extracting user actions from desktop recordings are proposed and evaluated.
Vision-Language Models (VLMs) Direct Frame-Based Approach Differential Frame-Based Approach computer vision techniques self-curated dataset advanced benchmark action sequences Robotic Process Automation