Ai-Facilitated Analysis of Abstracts and Conclusions: Flagging
  Unsubstantiated Claims and Ambiguous Pronouns
We present and evaluate a suite of proof-of-concept (PoC), structured
workflow prompts designed to elicit human-like hierarchical reasoning while
guiding Large Language Models (LLMs) in high-level semantic and linguistic
analysis of scholarly manuscripts. The prompts target two non-trivial
analytical tasks: identifying unsubstantiated claims in summaries
(informational integrity) and flagging ambiguous pronoun references (linguistic
clarity). We conducted a systematic, multi-run evaluation on two frontier
models (Gemini Pro 2.5 Pro and ChatGPT Plus o3) under varied context
conditions. Our results for the informational integrity task reveal a
significant divergence in model performance: while both models successfully
identified an unsubstantiated head of a noun phrase (95% success), ChatGPT
consistently failed (0% success) to identify an unsubstantiated adjectival
modifier that Gemini correctly flagged (95% success), raising a question
regarding potential influence of the target's syntactic role. For the
linguistic analysis task, both models performed well (80-90% success) with full
manuscript context. In a summary-only setting, however, ChatGPT achieved a
perfect (100%) success rate, while Gemini's performance was substantially
degraded. Our findings suggest that structured prompting is a viable
methodology for complex textual analysis but show that prompt performance may
be highly dependent on the interplay between the model, task type, and context,
highlighting the need for rigorous, model-specific testing.
6850fc485e07650ecce8918b
Evgeny Markhasin
claimed_verified
2025-06-17T07:20:37.574Z
68264aa0e6a0ae8670403081
https://cdn-avatars.huggingface.co/v1/production/uploads/68264aa0e6a0ae8670403081/a6V9yE1cf6-lFf7G8Ih-H.png
Evgeny Markhasin
user
PChemGuy
Ai-Facilitated Analysis of Abstracts and Conclusions: Flagging
  Unsubstantiated Claims and Ambiguous Pronouns
We present and evaluate a suite of proof-of-concept (PoC), structured
workflow prompts designed to elicit human-like hierarchical reasoning while
guiding Large Language Models (LLMs) in high-level semantic and linguistic
analysis of scholarly manuscripts. The prompts target two non-trivial
analytical tasks: identifying unsubstantiated claims in summaries
(informational integrity) and flagging ambiguous pronoun references (linguistic
clarity). We conducted a systematic, multi-run evaluation on two frontier
models (Gemini Pro 2.5 Pro and ChatGPT Plus o3) under varied context
conditions. Our results for the informational integrity task reveal a
significant divergence in model performance: while both models successfully
identified an unsubstantiated head of a noun phrase (95% success), ChatGPT
consistently failed (0% success) to identify an unsubstantiated adjectival
modifier that Gemini correctly flagged (95% success), raising a question
regarding potential influence of the target's syntactic role. For the
linguistic analysis task, both models performed well (80-90% success) with full
manuscript context. In a summary-only setting, however, ChatGPT achieved a
perfect (100%) success rate, while Gemini's performance was substantially
degraded. Our findings suggest that structured prompting is a viable
methodology for complex textual analysis but show that prompt performance may
be highly dependent on the interplay between the model, task type, and context,
highlighting the need for rigorous, model-specific testing.
Structured workflow prompts improve hierarchical reasoning in LLMs for scholarly manuscript analysis, but their effectiveness varies with the model, task type, and context.
Large Language Models LLMs Gemini Pro 2.5 Pro ChatGPT Plus o3 unsubstantiated claims informational integrity ambiguous pronoun references linguistic clarity hierarchical reasoning textual analysis