PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining
  Tasks
Large Language Models (LLMs) have the potential to semi-automate some process
mining (PM) analyses. While commercial models are already adequate for many
analytics tasks, the competitive level of open-source LLMs in PM tasks is
unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive
benchmark for PM focusing on domain knowledge (process-mining-specific and
process-specific) and on different implementation strategies. We focus also on
the challenges in creating such a benchmark, related to the public availability
of the data and on evaluation biases by the LLMs. Overall, we observe that most
of the considered LLMs can perform some process mining tasks at a satisfactory
level, but tiny models that would run on edge devices are still inadequate. We
also conclude that while the proposed benchmark is useful for identifying LLMs
that are adequate for process mining tasks, further research is needed to
overcome the evaluation biases and perform a more thorough ranking of the
competitive LLMs.
6699eeebfbc9ba8a122714f8
Alessandro Berti
6699eeebfbc9ba8a122714f9
Humam Kourani
6699eeebfbc9ba8a122714fa
Wil M. P. van der Aalst
PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining
  Tasks
Large Language Models (LLMs) have the potential to semi-automate some process
mining (PM) analyses. While commercial models are already adequate for many
analytics tasks, the competitive level of open-source LLMs in PM tasks is
unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive
benchmark for PM focusing on domain knowledge (process-mining-specific and
process-specific) and on different implementation strategies. We focus also on
the challenges in creating such a benchmark, related to the public availability
of the data and on evaluation biases by the LLMs. Overall, we observe that most
of the considered LLMs can perform some process mining tasks at a satisfactory
level, but tiny models that would run on edge devices are still inadequate. We
also conclude that while the proposed benchmark is useful for identifying LLMs
that are adequate for process mining tasks, further research is needed to
overcome the evaluation biases and perform a more thorough ranking of the
competitive LLMs.
Large Language Models (LLMs) process mining (PM) analyses PM-LLM-Benchmark domain knowledge process-mining-specific process-specific implementation strategies public availability evaluation biases edge devices