<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>1vuio0pswjnm7</author><text>Unless the originating IP address is a Google-controlled one, using Googlebot as a User-Agent header is (IME) generally no better than not sending a UA header at all.^1  If the goal is to make a server believe a request is coming from Google, then the request needs to be sent from a publicised Google-controlled IP address.^2&lt;p&gt;1. For many years I have had great results with not sending a UA header. It is also, IMO, an effective means to discover the true number of websites that refuse to fulfill a request in the absence of a UA header, which IME is extremely small.  For that small handful of sites, one can send a &amp;quot;fake&amp;quot; UA header of one&amp;#x27;s choosing.  sec.gov is an example of such a site.&lt;p&gt;2. &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;developers.google.com&amp;#x2F;static&amp;#x2F;search&amp;#x2F;apis&amp;#x2F;ipranges&amp;#x2F;googlebot.json&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;developers.google.com&amp;#x2F;static&amp;#x2F;search&amp;#x2F;apis&amp;#x2F;ipranges&amp;#x2F;goo...&lt;/a&gt;</text><parent_chain><item><author>hutrdvnj</author><text>I use Googlebot as my fake browsers user agent for years. It&amp;#x27;s really interested to explore the web, when everyone thinks you&amp;#x27;re Google.</text></item><item><author>cj</author><text>There are legitimate use cases for bypassing cloudflare&amp;#x27;s bot protection.&lt;p&gt;I discovered our company&amp;#x27;s help documentation (and integration guides), hosted by readme.com, were completely de-indexed from Google for the past 3 months.&lt;p&gt;Our Readme docs were formerly our #1 source of organic (free) leads.&lt;p&gt;After investigating, Cloudflare (as configured by Readme) was blocking Googlebot when using Cloudflare Workers. Cloudflare was returning a 403 for Googlebot, but returning pages as usual for regular users.&lt;p&gt;The cause: we were using Workers to rewrite some URLs at the edge (replacing Readme&amp;#x27;s default images with optimized + compressed images, using Cloudflare&amp;#x27;s own image optimization service).&lt;p&gt;By using Workers to do this, it resulted in Readme&amp;#x27;s Cloudflare account receiving requests from our domain with &amp;quot;googlebot&amp;quot; useragent, but from an IP that wasn&amp;#x27;t verified as a googlebot IP address (I assume the Worker was requesting the Readme site using the Googlebot user agent but with whatever IP address is used when using CF Workers).&lt;p&gt;I emailed Cloudflare support but it was clear it would take a lot of time to get them to understand the issue (and probably longer to fix it).&lt;p&gt;So, we had to spend a lot of time figuring out how to allow Googlebot requests past Cloudflare&amp;#x27;s &amp;quot;fake bot&amp;quot; firewall rule.&lt;p&gt;In our own Cloudflare account, we have all security settings at the lowest sensitivity possible (or turned off completely). We serve over 500 billion requests a month (10+ TB of bandwidth), and the amount of blocked traffic to seemingly legitimate clients was surprisingly high.&lt;p&gt;I love Cloudflare (and own quite a bit of their stock) but I&amp;#x27;m beginning to rethink my stance on their service. They make it extremely easy to enable powerful features with little visibility or control over the details of how those features work.&lt;p&gt;Another SEO nightmare is their &amp;quot;Crawler Hints&amp;quot; service. I highly recommend no one uses this if you are ever the target of automated security scanners (e.g. ones used by bug bounty white hat hackers). With &amp;quot;crawler hints&amp;quot; enabled and with a white hat hacker running a scan of your site hitting random URLs... results in bingbot, yandex, and other search engines attempting to index every single one of the URLs hit by the security scanners used by hackers.&lt;p&gt;Basically, it&amp;#x27;s a mess, and the only way to really fix it is to bypass cloudflare or spend a lot of time and money with Cloudflare debugging.&lt;p&gt;Next quarter I&amp;#x27;m faced with the decision of either doubling down of Cloudflare and getting an Enterprise plan with them ($20k+) or just ripping them out of our stack and going back to our old AWS Cloudfront set up which has fewer POPs, but was much less of a hassle.</text></item></parent_chain></comment><story><title>How to Bypass Cloudflare: A Comprehensive Guide</title><url>https://www.zenrows.com/blog/bypass-cloudflare</url></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>TrickyRick</author><text>What are some of the most interesting differences you&amp;#x27;ve seen?</text><parent_chain><item><author>hutrdvnj</author><text>I use Googlebot as my fake browsers user agent for years. It&amp;#x27;s really interested to explore the web, when everyone thinks you&amp;#x27;re Google.</text></item><item><author>cj</author><text>There are legitimate use cases for bypassing cloudflare&amp;#x27;s bot protection.&lt;p&gt;I discovered our company&amp;#x27;s help documentation (and integration guides), hosted by readme.com, were completely de-indexed from Google for the past 3 months.&lt;p&gt;Our Readme docs were formerly our #1 source of organic (free) leads.&lt;p&gt;After investigating, Cloudflare (as configured by Readme) was blocking Googlebot when using Cloudflare Workers. Cloudflare was returning a 403 for Googlebot, but returning pages as usual for regular users.&lt;p&gt;The cause: we were using Workers to rewrite some URLs at the edge (replacing Readme&amp;#x27;s default images with optimized + compressed images, using Cloudflare&amp;#x27;s own image optimization service).&lt;p&gt;By using Workers to do this, it resulted in Readme&amp;#x27;s Cloudflare account receiving requests from our domain with &amp;quot;googlebot&amp;quot; useragent, but from an IP that wasn&amp;#x27;t verified as a googlebot IP address (I assume the Worker was requesting the Readme site using the Googlebot user agent but with whatever IP address is used when using CF Workers).&lt;p&gt;I emailed Cloudflare support but it was clear it would take a lot of time to get them to understand the issue (and probably longer to fix it).&lt;p&gt;So, we had to spend a lot of time figuring out how to allow Googlebot requests past Cloudflare&amp;#x27;s &amp;quot;fake bot&amp;quot; firewall rule.&lt;p&gt;In our own Cloudflare account, we have all security settings at the lowest sensitivity possible (or turned off completely). We serve over 500 billion requests a month (10+ TB of bandwidth), and the amount of blocked traffic to seemingly legitimate clients was surprisingly high.&lt;p&gt;I love Cloudflare (and own quite a bit of their stock) but I&amp;#x27;m beginning to rethink my stance on their service. They make it extremely easy to enable powerful features with little visibility or control over the details of how those features work.&lt;p&gt;Another SEO nightmare is their &amp;quot;Crawler Hints&amp;quot; service. I highly recommend no one uses this if you are ever the target of automated security scanners (e.g. ones used by bug bounty white hat hackers). With &amp;quot;crawler hints&amp;quot; enabled and with a white hat hacker running a scan of your site hitting random URLs... results in bingbot, yandex, and other search engines attempting to index every single one of the URLs hit by the security scanners used by hackers.&lt;p&gt;Basically, it&amp;#x27;s a mess, and the only way to really fix it is to bypass cloudflare or spend a lot of time and money with Cloudflare debugging.&lt;p&gt;Next quarter I&amp;#x27;m faced with the decision of either doubling down of Cloudflare and getting an Enterprise plan with them ($20k+) or just ripping them out of our stack and going back to our old AWS Cloudfront set up which has fewer POPs, but was much less of a hassle.</text></item></parent_chain></comment><story><title>How to Bypass Cloudflare: A Comprehensive Guide</title><url>https://www.zenrows.com/blog/bypass-cloudflare</url></story>