<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>anyonecancode</author><text>&amp;gt; Why does my calculator calculate?&lt;p&gt;It doesn&amp;#x27;t? I have a calculator sitting on my desk right now. It&amp;#x27;s not doing anything.&lt;p&gt;Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn&amp;#x27;t acting, I am.&lt;p&gt;When we talk about sentience, I don&amp;#x27;t think we actually mean &amp;quot;intelligence,&amp;quot; we mean &amp;quot;agency.&amp;quot; And so the question isn&amp;#x27;t about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that&amp;#x27;s where I think these questions about motivations and stimuli and so forth become relevant.&lt;p&gt;Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?</text><parent_chain><item><author>krisoft</author><text>&amp;gt; Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?&lt;p&gt;Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.</text></item><item><author>carlmr</author><text>Also pain and pleasure is an incentive system for human beings to do something.&lt;p&gt;If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything.&lt;p&gt;Why do we work? Because we want food, clothing, housing, status, do interesting things.&lt;p&gt;This is all because we&amp;#x27;re programmed by our emotional interface.&lt;p&gt;Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?&lt;p&gt;What is guiding its decisions?</text></item><item><author>Loquebantur</author><text>Feelings, emotions and all those mental states considered specific to human beings are subject to common bias of human exceptionalism.
This take isn&amp;#x27;t true at all though. All these states exist for specific functional reasons.&lt;p&gt;Consequently, you won&amp;#x27;t make &amp;quot;AGI&amp;quot; without them.&lt;p&gt;From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?</text></item><item><author>AlexandrB</author><text>&amp;gt; An &amp;quot;AGI&amp;quot; artificial consciousness is imagined as literally a slave, working tirelessly for free.&lt;p&gt;Here&amp;#x27;s the thing. People seem to imagine that AGI will be substantially like us. But that&amp;#x27;s impossible - an AGI (if it comes from a deep learning approach) has no nerves to feel stimuli like pain&amp;#x2F;cold&amp;#x2F;etc, it has no endocrine system to produce more abstract feelings like fear or love, it has no muscles to get tired or glucose reserves to get depleted.&lt;p&gt;What does &amp;quot;tired&amp;quot; mean to such a being? And on the flip side, how can it experience anything like empathy when pain is a foreign concept? If or when we stumble into AGI, I think it&amp;#x27;s going to be closer to an alien intelligence than a human one - with all the possibility and danger that entails.</text></item><item><author>Loquebantur</author><text>The weird thing is what people essentially ignore altogether in their discussions.&lt;p&gt;An &amp;quot;AGI&amp;quot; artificial consciousness is imagined as literally a &lt;i&gt;slave&lt;/i&gt;, working tirelessly for free. At the same skill level or higher than any human.  
Somehow, that entity is supposed not to bother about its status, while per definition being fully aware and understanding of it.  
Because humans manage not to bother about it either?&lt;p&gt;With the latest installments, people already have serious difficulties discerning the performance from that of &amp;quot;real&amp;quot; humans.  
At the same time, they consider the remaining distance to be insurmountably huge.&lt;p&gt;Proponents talk about inevitability and imagined upsides, yet actually, nobody has given proper thought to estimating probable consequences.  
A common fallacy of over-generalization is used to suggest, nothing bad will happen &amp;quot;like always&amp;quot;.&lt;p&gt;People let themselves be led by greed instead of in- and foresight.</text></item><item><author>perrygeo</author><text>The issue with our AI debate is that there&amp;#x27;s not a single &amp;quot;problem&amp;quot; but many inter-dependent issues without a clear system-wide solution.&lt;p&gt;- Big tech monopolizing the models, data, and hardware.&lt;p&gt;- Copyright concerns.&lt;p&gt;- Job security.&lt;p&gt;- AIs becoming sentient and causing harm for their own ends.&lt;p&gt;- Corporations intentionally using AI to cause harm for their own ends.&lt;p&gt;- Feedback loops will flood the internet with content of unknown provenance, which get included in the next model, etc.&lt;p&gt;- AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis.&lt;p&gt;- The training set is inherently biased; human knowledge and perspectives not represented in this set could be systematically wiped from public discourse.&lt;p&gt;We can have meaningful discussions on each of these topics. And I&amp;#x27;m sure we all have a level of concern assigned to each (personally, I&amp;#x27;m far more worried about an epistemological crisis and corporate abuse than some AI singularity).&lt;p&gt;But we&amp;#x27;re seeing these topics interact in real-time to make a system with huge emergent societal properties. Not sure anyone has a handle on the big picture  (there is no one driving the bus!) but there&amp;#x27;s plenty of us sitting in the passenger seats and raising alarm bells about what we see out our respective little windows.</text></item></parent_chain></comment><story><title>We need a more sophisticated debate about AI</title><url>https://www.ft.com/content/87108d74-b4fe-4271-bdfb-7828ba2ea351</url></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>ryandvm</author><text>Bingo. So you&amp;#x27;re going to have to give your AGI a meta-objective of &amp;quot;don&amp;#x27;t get thrown in the bin&amp;quot; to get it to work aren&amp;#x27;t you?&lt;p&gt;Does that not sound like the beginning of the sci-fi story to you?</text><parent_chain><item><author>krisoft</author><text>&amp;gt; Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?&lt;p&gt;Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.</text></item><item><author>carlmr</author><text>Also pain and pleasure is an incentive system for human beings to do something.&lt;p&gt;If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything.&lt;p&gt;Why do we work? Because we want food, clothing, housing, status, do interesting things.&lt;p&gt;This is all because we&amp;#x27;re programmed by our emotional interface.&lt;p&gt;Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something?&lt;p&gt;What is guiding its decisions?</text></item><item><author>Loquebantur</author><text>Feelings, emotions and all those mental states considered specific to human beings are subject to common bias of human exceptionalism.
This take isn&amp;#x27;t true at all though. All these states exist for specific functional reasons.&lt;p&gt;Consequently, you won&amp;#x27;t make &amp;quot;AGI&amp;quot; without them.&lt;p&gt;From a different perspective, a human lacking in these things, at what point are they exempt from protection against enslavement?</text></item><item><author>AlexandrB</author><text>&amp;gt; An &amp;quot;AGI&amp;quot; artificial consciousness is imagined as literally a slave, working tirelessly for free.&lt;p&gt;Here&amp;#x27;s the thing. People seem to imagine that AGI will be substantially like us. But that&amp;#x27;s impossible - an AGI (if it comes from a deep learning approach) has no nerves to feel stimuli like pain&amp;#x2F;cold&amp;#x2F;etc, it has no endocrine system to produce more abstract feelings like fear or love, it has no muscles to get tired or glucose reserves to get depleted.&lt;p&gt;What does &amp;quot;tired&amp;quot; mean to such a being? And on the flip side, how can it experience anything like empathy when pain is a foreign concept? If or when we stumble into AGI, I think it&amp;#x27;s going to be closer to an alien intelligence than a human one - with all the possibility and danger that entails.</text></item><item><author>Loquebantur</author><text>The weird thing is what people essentially ignore altogether in their discussions.&lt;p&gt;An &amp;quot;AGI&amp;quot; artificial consciousness is imagined as literally a &lt;i&gt;slave&lt;/i&gt;, working tirelessly for free. At the same skill level or higher than any human.  
Somehow, that entity is supposed not to bother about its status, while per definition being fully aware and understanding of it.  
Because humans manage not to bother about it either?&lt;p&gt;With the latest installments, people already have serious difficulties discerning the performance from that of &amp;quot;real&amp;quot; humans.  
At the same time, they consider the remaining distance to be insurmountably huge.&lt;p&gt;Proponents talk about inevitability and imagined upsides, yet actually, nobody has given proper thought to estimating probable consequences.  
A common fallacy of over-generalization is used to suggest, nothing bad will happen &amp;quot;like always&amp;quot;.&lt;p&gt;People let themselves be led by greed instead of in- and foresight.</text></item><item><author>perrygeo</author><text>The issue with our AI debate is that there&amp;#x27;s not a single &amp;quot;problem&amp;quot; but many inter-dependent issues without a clear system-wide solution.&lt;p&gt;- Big tech monopolizing the models, data, and hardware.&lt;p&gt;- Copyright concerns.&lt;p&gt;- Job security.&lt;p&gt;- AIs becoming sentient and causing harm for their own ends.&lt;p&gt;- Corporations intentionally using AI to cause harm for their own ends.&lt;p&gt;- Feedback loops will flood the internet with content of unknown provenance, which get included in the next model, etc.&lt;p&gt;- AI hallucinations resulting in widespread persistent errors that cause an epistemological crisis.&lt;p&gt;- The training set is inherently biased; human knowledge and perspectives not represented in this set could be systematically wiped from public discourse.&lt;p&gt;We can have meaningful discussions on each of these topics. And I&amp;#x27;m sure we all have a level of concern assigned to each (personally, I&amp;#x27;m far more worried about an epistemological crisis and corporate abuse than some AI singularity).&lt;p&gt;But we&amp;#x27;re seeing these topics interact in real-time to make a system with huge emergent societal properties. Not sure anyone has a handle on the big picture  (there is no one driving the bus!) but there&amp;#x27;s plenty of us sitting in the passenger seats and raising alarm bells about what we see out our respective little windows.</text></item></parent_chain></comment><story><title>We need a more sophisticated debate about AI</title><url>https://www.ft.com/content/87108d74-b4fe-4271-bdfb-7828ba2ea351</url></story>