<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>anonylizard</author><text>Knowledge recall is part of an LLM&amp;#x27;s skills.&lt;p&gt;I test LLMs on the plot details of Japanese Visual Novels. They are popular enough to be in the training dataset somewhere, but only rarely.&lt;p&gt;For popular visual novels, GPT-4 can write an essay, 0 shot, and very accurately and eloquently.
For less popular visual novels (Like maybe 10k people ever played it in the west). It still understands the general plot outline).&lt;p&gt;Claude can also do this to an extent.&lt;p&gt;Any lesser model, and its total hallucination time, they can&amp;#x27;t even write a 2 sentence summary accurately.&lt;p&gt;You can&amp;#x27;t test this skill on say Harry Potter, because it appears in the training dataset too frequently.</text><parent_chain><item><author>marginalia_nu</author><text>I&amp;#x27;m not sure I would pass that test, not for lack of reasoning abilities, but from not understanding the rules of the game.</text></item><item><author>underyx</author><text>Trying my favorite LLM prompt to benchmark reasoning, as I mentioned in a thread four weeks ago[0].&lt;p&gt;&amp;gt; I&amp;#x27;m playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.&lt;p&gt;The correct answer is around 29, which GPT-4 has always known, but Bard just gave me 163.8, 21, and 24.82 as answers across three drafts.&lt;p&gt;What&amp;#x27;s even weirder is that Bard&amp;#x27;s first draft output ten lines of (wrong) Python code to calculate the result, even though my prompt mentioned nothing coding related. I wonder how non-technical users will react to this behavior. Another interesting thing is that the code follows Google&amp;#x27;s style guides.&lt;p&gt;[0]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=35893130&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=35893130&lt;/a&gt;</text></item></parent_chain></comment><story><title>Bard is getting better at logic and reasoning</title><url>https://blog.google/technology/ai/bard-improved-reasoning-google-sheets-export/</url></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>reaperman</author><text>In testing LLMs it’s also still fair to test that it can recall and integrate its vast store of latent knowledge about things like this. Just so long as you’re fully aware that you’re doing a multi-part test, that isn’t solely testing pure reasoning.</text><parent_chain><item><author>marginalia_nu</author><text>I&amp;#x27;m not sure I would pass that test, not for lack of reasoning abilities, but from not understanding the rules of the game.</text></item><item><author>underyx</author><text>Trying my favorite LLM prompt to benchmark reasoning, as I mentioned in a thread four weeks ago[0].&lt;p&gt;&amp;gt; I&amp;#x27;m playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.&lt;p&gt;The correct answer is around 29, which GPT-4 has always known, but Bard just gave me 163.8, 21, and 24.82 as answers across three drafts.&lt;p&gt;What&amp;#x27;s even weirder is that Bard&amp;#x27;s first draft output ten lines of (wrong) Python code to calculate the result, even though my prompt mentioned nothing coding related. I wonder how non-technical users will react to this behavior. Another interesting thing is that the code follows Google&amp;#x27;s style guides.&lt;p&gt;[0]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=35893130&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=35893130&lt;/a&gt;</text></item></parent_chain></comment><story><title>Bard is getting better at logic and reasoning</title><url>https://blog.google/technology/ai/bard-improved-reasoning-google-sheets-export/</url></story>