PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task
  Completion
Recent evaluations of Large Language Models (LLMs) have centered around
testing their zero-shot/few-shot capabilities for basic natural language tasks
and their ability to translate instructions into tool APIs. However, the
evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal
instructions in a complex multi-modal environment has not been investigated. To
address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark
to assess LLMs' ability to create and edit PPT files based on user
instructions. It contains 279 multi-turn sessions covering diverse topics and
hundreds of instructions involving multi-modal operations. We also propose the
PPTX-Match Evaluation System that evaluates if LLMs finish the instruction
based on the prediction file rather than the label API sequence, thus it
supports various LLM-generated API sequences. We measure 3 closed LLMs and 6
open-source LLMs. The results show that GPT-4 outperforms other LLMs with
75.1\% accuracy in single-turn dialogue testing but faces challenges in
completing entire sessions, achieving just 6\% session accuracy. We find three
main error causes in our benchmark: error accumulation in the multi-turn
session, long PPT template processing, and multi-modality perception. These
pose great challenges for future LLM and agent systems. We release the data,
code, and evaluation system of PPTC at https://github.com/gydpku/PPTC.
654874d85cd5692b3af9d354
Yiduo Guo
admin_assigned
2023-11-06T13:59:37.823Z
64e3282156b920ef00cf7d94
/avatars/8a76c27fe6c71eb98a28b4ebfb90336d.svg
Guo
user
YiDuo1999
654874d85cd5692b3af9d355
Zekai Zhang
654874d85cd5692b3af9d356
Yaobo Liang
admin_assigned
2023-11-06T14:00:19.782Z
64ddde772402348182664c2a
/avatars/21798dde3df06f57288dc9dc2e044abc.svg
Yaobo Liang
user
Eureka6174
654874d85cd5692b3af9d357
Dongyan Zhao
admin_assigned
2023-11-06T14:01:22.213Z
629ee8171fec321a2d3c458d
/avatars/a8ae78b193ba2a43c6e36e185e0ef2b9.svg
Zhaodong Yan
user
Anhforth
654874d85cd5692b3af9d358
Duan Nan
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task
  Completion
Recent evaluations of Large Language Models (LLMs) have centered around
testing their zero-shot/few-shot capabilities for basic natural language tasks
and their ability to translate instructions into tool APIs. However, the
evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal
instructions in a complex multi-modal environment has not been investigated. To
address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark
to assess LLMs' ability to create and edit PPT files based on user
instructions. It contains 279 multi-turn sessions covering diverse topics and
hundreds of instructions involving multi-modal operations. We also propose the
PPTX-Match Evaluation System that evaluates if LLMs finish the instruction
based on the prediction file rather than the label API sequence, thus it
supports various LLM-generated API sequences. We measure 3 closed LLMs and 6
open-source LLMs. The results show that GPT-4 outperforms other LLMs with
75.1\% accuracy in single-turn dialogue testing but faces challenges in
completing entire sessions, achieving just 6\% session accuracy. We find three
main error causes in our benchmark: error accumulation in the multi-turn
session, long PPT template processing, and multi-modality perception. These
pose great challenges for future LLM and agent systems. We release the data,
code, and evaluation system of PPTC at https://github.com/gydpku/PPTC.
The PowerPoint Task Completion (PPTC) benchmark assesses LLMs' ability to complete multi-turn, multi-modal instructions for editing PPT files and identifies challenges such as error accumulation, template processing, and multi-modality perception.
LLMs zero-shot few-shot natural language tasks tool APIs multi-turn multi-modal PowerPoint Task Completion PPTC benchmark PPTX-Match Evaluation System single-turn dialogue testing session accuracy error accumulation PPT template processing multi-modality perception