<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>tomrod</author><text>This highlights the major downside to &amp;quot;data-driven&amp;quot; policy and decisions.&lt;p&gt;Data can &amp;quot;lie&amp;quot;. What is observed is not always reality, simply what we can see of it.&lt;p&gt;Consider auctions. You never actually &amp;quot;see&amp;quot; the bidder&amp;#x27;s demand or utility. Yes, there are some ways to structure auctions that in theory show willingness to pay and such (ignoring confounding factors and irrationality), but you don&amp;#x27;t actually observe anything beyond the bid.&lt;p&gt;Similarly, on websites, you don&amp;#x27;t always know the causal reasons people click here or there. You know perhaps enough to predict a step-wise behavior, but don&amp;#x27;t (usually) understand the full behavioral lifecycle -- especially if a metric improves but at the hidden cost of decrements to conversion and similar.</text><parent_chain><item><author>causi</author><text>The problem with AB testing is that it&amp;#x27;s a short-term strategy. For example, if a news site runs AB testing with headlines, they&amp;#x27;ll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles get the most alluring headlines.</text></item></parent_chain></comment><story><title>A/B testing gets misused to juice metrics in the short term</title><url>https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/</url></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>goodside</author><text>There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made the statistics more complicated.</text><parent_chain><item><author>causi</author><text>The problem with AB testing is that it&amp;#x27;s a short-term strategy. For example, if a news site runs AB testing with headlines, they&amp;#x27;ll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles get the most alluring headlines.</text></item></parent_chain></comment><story><title>A/B testing gets misused to juice metrics in the short term</title><url>https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/</url></story>