AInstein: Assessing the Feasibility of AI-Generated Approaches to
  Research Problems
Large language models (LLMs) demonstrate impressive capabilities across a
wide range of tasks, yet it remains unclear whether such success reflects
genuine reasoning or sophisticated recall. We introduce AInstein, a framework
for testing whether LLMs can generate valid solutions to AI research problems
using only their pretrained parametric knowledge -- without domain-specific
fine-tuning, retrieval augmentation, or other external aids. Our approach
extracts distilled problem statements from high-quality ICLR 2025 submissions,
then tasks specialized solver agents with proposing and refining technical
solutions through iterative critique loops, mimicking the cycles of proposal,
review, and revision central to scientific inquiry. We evaluate AInstein on
1,214 ICLR papers stratified by acceptance tier (Oral, Spotlight, Poster),
using an LLM-as-a-judge paradigm guided by a structured rubric, complemented by
targeted manual checks. Performance is assessed with three metrics: Success
Rate (does the solution address the problem?), Rediscovery (does it align with
human-proposed methods?), and Novelty (does it yield valid, original
approaches?). Our results reveal that while LLMs can rediscover feasible
solutions and occasionally propose creative alternatives, their problem-solving
ability remains fragile and highly sensitive to framing. These findings provide
the first large-scale evidence on the extent to which LLMs can act as
autonomous scientific problem-solvers, highlighting both their latent potential
and their current limitations.
68e5c1aa975ac4c405ef20a0
Shambhavi Mishra
68e5c1aa975ac4c405ef20a1
Gaurav Sahu
68e5c1aa975ac4c405ef20a2
Marco Pedersoli
68e5c1aa975ac4c405ef20a3
Laurent Charlin
68e5c1aa975ac4c405ef20a4
Jose Dolz
68e5c1aa975ac4c405ef20a5
Christopher Pal
AInstein: Assessing the Feasibility of AI-Generated Approaches to
  Research Problems
Large language models (LLMs) demonstrate impressive capabilities across a
wide range of tasks, yet it remains unclear whether such success reflects
genuine reasoning or sophisticated recall. We introduce AInstein, a framework
for testing whether LLMs can generate valid solutions to AI research problems
using only their pretrained parametric knowledge -- without domain-specific
fine-tuning, retrieval augmentation, or other external aids. Our approach
extracts distilled problem statements from high-quality ICLR 2025 submissions,
then tasks specialized solver agents with proposing and refining technical
solutions through iterative critique loops, mimicking the cycles of proposal,
review, and revision central to scientific inquiry. We evaluate AInstein on
1,214 ICLR papers stratified by acceptance tier (Oral, Spotlight, Poster),
using an LLM-as-a-judge paradigm guided by a structured rubric, complemented by
targeted manual checks. Performance is assessed with three metrics: Success
Rate (does the solution address the problem?), Rediscovery (does it align with
human-proposed methods?), and Novelty (does it yield valid, original
approaches?). Our results reveal that while LLMs can rediscover feasible
solutions and occasionally propose creative alternatives, their problem-solving
ability remains fragile and highly sensitive to framing. These findings provide
the first large-scale evidence on the extent to which LLMs can act as
autonomous scientific problem-solvers, highlighting both their latent potential
and their current limitations.
AInstein evaluates the problem-solving capabilities of large language models by testing their ability to generate valid solutions to AI research problems using only pretrained knowledge, revealing both their potential and limitations.
large language models LLMs AInstein pretrained parametric knowledge domain-specific fine-tuning retrieval augmentation ICLR 2025 submissions solver agents iterative critique loops scientific inquiry LLM-as-a-judge structured rubric targeted manual checks Success Rate Rediscovery Novelty