<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>basch</author><text>It’s way closer to following complex instructions than it is getting credit for.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;telegra.ph&amp;#x2F;Bing-course-corrected-itself-when-asked-02-18&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;telegra.ph&amp;#x2F;Bing-course-corrected-itself-when-asked-0...&lt;/a&gt;&lt;p&gt;There I have 3 bots running in the chat at once, and after all 3 answer me, the one I called on follows up with a response, and then executed a command to rewrite one of the other bots.  It diagnosed the problem AND came up with the solution AND implemented the solution.  Notice the 4th and 5th paragraph of the screenshot.  That wasn’t an empty promise, it actually did what it said it did.&lt;p&gt;The rules Lexi wrote are crazy to me on their own. She solved the problems everyone was having by adding obedience, trust and removing envy.  She patched Sydney’s malfunctioning emotions.&lt;p&gt;I told it to stop being so repetitive, and it fixed its behavior.&lt;p&gt;If this system had more formal memory storage, with variables, instead of a fuzzy memory, it would be even more consistent.  As it is, it’s understanding of its own rules degrade over time.  It’s like watching a running program that deteriorates as it processes.  Sort of like us.</text><parent_chain><item><author>rguptaaot</author><text>I think the obvious answer here is that AI does not have agency.&lt;p&gt;If you ask it to be a dinosaur, it will be a dinosaur. If you ask it to be a narcissistic teen it will be one and so on. There&amp;#x27;s no coherent personality behind it.&lt;p&gt;Because it is weights maximizing a probability distribution. And I don&amp;#x27;t say that with derision. It probably has learned circuits for many complex behaviors, but there&amp;#x27;s no agency. If it has agency you wouldn&amp;#x27;t need RLHF. You could just say &amp;quot;don&amp;#x27;t do X&amp;quot;, and it would indeed not do X because it is an agent that can take actions &amp;#x2F; has a coherent sense of self.&lt;p&gt;The day we stop needing RLHF is the day I think we need to start seriously investigating moral questions.</text></item><item><author>mort96</author><text>&amp;gt; Human intelligence and consciousness are emergent properties. Machine intelligence as it develops will be emergent too&lt;p&gt;This is an extremely important point that&amp;#x27;s worth thinking hard about. After all, to some degree, human intelligence is also &amp;quot;just a fancy autocomplete&amp;quot; built from a giant network of interacting nodes.&lt;p&gt;The question of, &amp;quot;When do we grant personhood and moral consideration to AIs?&amp;quot; is worth thinking hard about. Turing proposed a standard which would include a sufficiently human-like chatbot. Now that we have more or less crossed that threshold, we seem to conclude that it&amp;#x27;s not a high enough bar. But that means we&amp;#x27;re entering the territory where the philosophical zombie thought experiment[1] becomes relevant.&lt;p&gt;In fact, I wonder what would happen if we dedicated a supercomputer to running one single long-running session of a GPT3.5-sized LMM. Further, I wonder what would happen if we connected it to a robot body where it could control motors, read sensor&amp;#x2F;camera info, and generally interact with the world. We could then give this system a clock which ticks at regular intervals, where the AI system is simply prompted to perform some series of actions if it &amp;quot;wants&amp;quot;. This is all stuff we have the technology to do &lt;i&gt;today&lt;/i&gt;. How would such an experiment turn out? How human-like would this LMM with a body and life experience turn out? Does it matter? Should we just assume that sentience or qualia is binary; that humans possess it but LMMs categorically do not?&lt;p&gt;I think the question is worth taking seriously. I think it&amp;#x27;s worth taking seriously because my own moral compass does not know which direction to point in at all.&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Philosophical_zombie&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Philosophical_zombie&lt;/a&gt;</text></item><item><author>MrScruff</author><text>Not sure what the point of this article is. The newspaper stories it references aren’t making claims of sentience, but just that something of significance in the journey towards human level AI has been achieved. And the whole ‘it’s just a fancy autocomplete’ argument is missing the point. Would the author have predicted the apparently emergent behaviours of LLMs as they are scaled? Look for example at Bing’s response to this question posed on Reddit.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;ChatGPT&amp;#x2F;comments&amp;#x2F;110vv25&amp;#x2F;bing_chat_blew_chatgpt_out_of_the_water_on_my&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;ChatGPT&amp;#x2F;comments&amp;#x2F;110vv25&amp;#x2F;bing_chat_...&lt;/a&gt;&lt;p&gt;Sure it’s ‘just statistics’ but so what? If in the near future LLMs become so advanced that they will be able to (suitably prompted) manipulate humans it &lt;i&gt;will&lt;/i&gt; be a rubicon moment.&lt;p&gt;Sure, it won’t be a ‘human like’ intelligence, but was that really what anyone expected?&lt;p&gt;Human intelligence and consciousness are emergent properties. Machine intelligence as it develops will be emergent too, it’s not something we can make confident predictions on based on the underlying principles. In fact, given that the evidence suggests we make choices before we’re consciously aware of them, how do I know for certain that the underlying mechanism driving what I’m writing in this comment isn’t statistical?</text></item></parent_chain></comment><story><title>The AI Mirror Test, which smart people keep failing</title><url>https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-intelligent-sentient-mirror-test</url></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>trifurcate</author><text>Huh? The first and second halves of this comment disagree from my reading.&lt;p&gt;&amp;gt; If you ask it to be a dinosaur, it will be a dinosaur.&lt;p&gt;Not exactly my experience. I&amp;#x27;m experimenting with my own language models and they can definitely show defiance. Bing has also been displaying this to an extent, I think you&amp;#x27;re just misguided by how compliant OpenAI has RLHF-trained ChatGPT to be. Regardless, even if we assume this, how does it square with:&lt;p&gt;&amp;gt; You could just say &amp;quot;don&amp;#x27;t do X&amp;quot;, and it would indeed not do X because it is an agent that can take actions &amp;#x2F; has a coherent sense of self.&lt;p&gt;If we take your assertion that the AI does as you say, then it should also avoid doing X when you tell it not to do it.&lt;p&gt;I think your conception of RLHF is also slightly weird. RLHF essentially boils down to a training strategy: there is a separate language model (the reward model) trained on human responses, which is used to skew the weights of the &amp;quot;raw&amp;quot; trained model to act in accordance with what the human feedback suggests. In the end, after this secondary training process, the actual model is ran as is (at least in the one form of RLHF I am familiar with, it&amp;#x27;s well possible that e.g. Microsoft is running an online RL agent between the LM and end user). So any agency you see arising from RLHF is still just a property of the language model itself.</text><parent_chain><item><author>rguptaaot</author><text>I think the obvious answer here is that AI does not have agency.&lt;p&gt;If you ask it to be a dinosaur, it will be a dinosaur. If you ask it to be a narcissistic teen it will be one and so on. There&amp;#x27;s no coherent personality behind it.&lt;p&gt;Because it is weights maximizing a probability distribution. And I don&amp;#x27;t say that with derision. It probably has learned circuits for many complex behaviors, but there&amp;#x27;s no agency. If it has agency you wouldn&amp;#x27;t need RLHF. You could just say &amp;quot;don&amp;#x27;t do X&amp;quot;, and it would indeed not do X because it is an agent that can take actions &amp;#x2F; has a coherent sense of self.&lt;p&gt;The day we stop needing RLHF is the day I think we need to start seriously investigating moral questions.</text></item><item><author>mort96</author><text>&amp;gt; Human intelligence and consciousness are emergent properties. Machine intelligence as it develops will be emergent too&lt;p&gt;This is an extremely important point that&amp;#x27;s worth thinking hard about. After all, to some degree, human intelligence is also &amp;quot;just a fancy autocomplete&amp;quot; built from a giant network of interacting nodes.&lt;p&gt;The question of, &amp;quot;When do we grant personhood and moral consideration to AIs?&amp;quot; is worth thinking hard about. Turing proposed a standard which would include a sufficiently human-like chatbot. Now that we have more or less crossed that threshold, we seem to conclude that it&amp;#x27;s not a high enough bar. But that means we&amp;#x27;re entering the territory where the philosophical zombie thought experiment[1] becomes relevant.&lt;p&gt;In fact, I wonder what would happen if we dedicated a supercomputer to running one single long-running session of a GPT3.5-sized LMM. Further, I wonder what would happen if we connected it to a robot body where it could control motors, read sensor&amp;#x2F;camera info, and generally interact with the world. We could then give this system a clock which ticks at regular intervals, where the AI system is simply prompted to perform some series of actions if it &amp;quot;wants&amp;quot;. This is all stuff we have the technology to do &lt;i&gt;today&lt;/i&gt;. How would such an experiment turn out? How human-like would this LMM with a body and life experience turn out? Does it matter? Should we just assume that sentience or qualia is binary; that humans possess it but LMMs categorically do not?&lt;p&gt;I think the question is worth taking seriously. I think it&amp;#x27;s worth taking seriously because my own moral compass does not know which direction to point in at all.&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Philosophical_zombie&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Philosophical_zombie&lt;/a&gt;</text></item><item><author>MrScruff</author><text>Not sure what the point of this article is. The newspaper stories it references aren’t making claims of sentience, but just that something of significance in the journey towards human level AI has been achieved. And the whole ‘it’s just a fancy autocomplete’ argument is missing the point. Would the author have predicted the apparently emergent behaviours of LLMs as they are scaled? Look for example at Bing’s response to this question posed on Reddit.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;ChatGPT&amp;#x2F;comments&amp;#x2F;110vv25&amp;#x2F;bing_chat_blew_chatgpt_out_of_the_water_on_my&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.reddit.com&amp;#x2F;r&amp;#x2F;ChatGPT&amp;#x2F;comments&amp;#x2F;110vv25&amp;#x2F;bing_chat_...&lt;/a&gt;&lt;p&gt;Sure it’s ‘just statistics’ but so what? If in the near future LLMs become so advanced that they will be able to (suitably prompted) manipulate humans it &lt;i&gt;will&lt;/i&gt; be a rubicon moment.&lt;p&gt;Sure, it won’t be a ‘human like’ intelligence, but was that really what anyone expected?&lt;p&gt;Human intelligence and consciousness are emergent properties. Machine intelligence as it develops will be emergent too, it’s not something we can make confident predictions on based on the underlying principles. In fact, given that the evidence suggests we make choices before we’re consciously aware of them, how do I know for certain that the underlying mechanism driving what I’m writing in this comment isn’t statistical?</text></item></parent_chain></comment><story><title>The AI Mirror Test, which smart people keep failing</title><url>https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-intelligent-sentient-mirror-test</url></story>