MIRAI: Evaluating LLM Agents for Event Forecasting
Recent advancements in Large Language Models (LLMs) have empowered LLM agents
to autonomously collect world information, over which to conduct reasoning to
solve complex problems. Given this capability, increasing interests have been
put into employing LLM agents for predicting international events, which can
influence decision-making and shape policy development on an international
scale. Despite such a growing interest, there is a lack of a rigorous benchmark
of LLM agents' forecasting capability and reliability. To address this gap, we
introduce MIRAI, a novel benchmark designed to systematically evaluate LLM
agents as temporal forecasters in the context of international events. Our
benchmark features an agentic environment with tools for accessing an extensive
database of historical, structured events and textual news articles. We refine
the GDELT event database with careful cleaning and parsing to curate a series
of relational prediction tasks with varying forecasting horizons, assessing LLM
agents' abilities from short-term to long-term forecasting. We further
implement APIs to enable LLM agents to utilize different tools via a code-based
interface. In summary, MIRAI comprehensively evaluates the agents' capabilities
in three dimensions: 1) autonomously source and integrate critical information
from large global databases; 2) write codes using domain-specific APIs and
libraries for tool-use; and 3) jointly reason over historical knowledge from
diverse formats and time to accurately predict future events. Through
comprehensive benchmarking, we aim to establish a reliable framework for
assessing the capabilities of LLM agents in forecasting international events,
thereby contributing to the development of more accurate and trustworthy models
for international relation analysis.
66836a75f771146afb3b4486
Chenchen Ye
claimed_verified
2024-07-02T07:39:01.011Z
661e2aac940272b8edc5a5dd
/avatars/000f7c8fae4ed38af3c616135a3e8d2d.svg
Chenchen Ye
user
chenchenye
66836a75f771146afb3b4487
Ziniu Hu
66836a75f771146afb3b4488
Yihe Deng
claimed_verified
2024-07-02T07:38:58.926Z
642f4c789b2484d7d8551a93
https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
Yihe Deng
user
ydeng9
66836a75f771146afb3b4489
Zijie Huang
66836a75f771146afb3b448a
Mingyu Derek Ma
claimed_verified
2024-07-02T07:38:56.904Z
644c1df68c51ddbe0ea6ddd7
https://cdn-avatars.huggingface.co/v1/production/uploads/644c1df68c51ddbe0ea6ddd7/KAsOhnOSbrOK54OHtJpiQ.jpeg
Mingyu Derek Ma
user
derekma
66836a75f771146afb3b448b
Yanqiao Zhu
claimed_verified
2024-07-02T07:38:54.633Z
65dda5295eea01ee95e21f64
/avatars/f935e7db6038661bef2305b7aeda8c9e.svg
Yanqiao Zhu
user
sxkdz
66836a75f771146afb3b448c
Wei Wang
MIRAI: Evaluating LLM Agents for Event Forecasting
Recent advancements in Large Language Models (LLMs) have empowered LLM agents
to autonomously collect world information, over which to conduct reasoning to
solve complex problems. Given this capability, increasing interests have been
put into employing LLM agents for predicting international events, which can
influence decision-making and shape policy development on an international
scale. Despite such a growing interest, there is a lack of a rigorous benchmark
of LLM agents' forecasting capability and reliability. To address this gap, we
introduce MIRAI, a novel benchmark designed to systematically evaluate LLM
agents as temporal forecasters in the context of international events. Our
benchmark features an agentic environment with tools for accessing an extensive
database of historical, structured events and textual news articles. We refine
the GDELT event database with careful cleaning and parsing to curate a series
of relational prediction tasks with varying forecasting horizons, assessing LLM
agents' abilities from short-term to long-term forecasting. We further
implement APIs to enable LLM agents to utilize different tools via a code-based
interface. In summary, MIRAI comprehensively evaluates the agents' capabilities
in three dimensions: 1) autonomously source and integrate critical information
from large global databases; 2) write codes using domain-specific APIs and
libraries for tool-use; and 3) jointly reason over historical knowledge from
diverse formats and time to accurately predict future events. Through
comprehensive benchmarking, we aim to establish a reliable framework for
assessing the capabilities of LLM agents in forecasting international events,
thereby contributing to the development of more accurate and trustworthy models
for international relation analysis.
MIRAI introduces a benchmark to evaluate LLM agents' forecasting capabilities for international events using historical databases and APIs.
Large Language Models (LLMs) GDELT event database temporal forecasting forecasting horizons relational prediction tasks domain-specific APIs tool-use historical knowledge international relation analysis