<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>rpalmaotero</author><text>For everyone that wants to start working with Spark and Big Data, I recommend them to enrole into this MOOC published by UC Berkeley at EDX: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.edx.org&amp;#x2F;course&amp;#x2F;introduction-big-data-apache-spark-uc-berkeleyx-cs100-1x&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.edx.org&amp;#x2F;course&amp;#x2F;introduction-big-data-apache-spar...&lt;/a&gt;</text><parent_chain></parent_chain></comment><story><title>IBM Invests to Help Apache Spark</title><url>http://bits.blogs.nytimes.com/2015/06/15/ibm-invests-to-help-open-source-big-data-software-and-itself/</url><text></text></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>zaroth</author><text>I read about an interesting technique, an &amp;quot;all-or-nothing tracker&amp;quot; in a blog post from an Apache Spark engineer.&lt;p&gt;You dispatch n jobs, where n is quite large, and you want to know; have all n jobs have completed, or has less than n jobs have completed. How to do so with a small fixed number of bytes with very high probability?&lt;p&gt;Give each job a random 128-bit ID number. XOR each ID number together as you start each job, and XOR into the same value as each job completes. If all the jobs have completed, the result is 0. The chance of zero turning up randomly if all jobs are not complete is negligible.&lt;p&gt;The technique is mentioned here under &amp;#x27;Lineage Tracking&amp;#x27;: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;highlyscalable.wordpress.com&amp;#x2F;2013&amp;#x2F;08&amp;#x2F;20&amp;#x2F;in-stream-big-data-processing&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;highlyscalable.wordpress.com&amp;#x2F;2013&amp;#x2F;08&amp;#x2F;20&amp;#x2F;in-stream-bi...&lt;/a&gt;  but there&amp;#x27;s a better blog post I remembered reading but can&amp;#x27;t find at the moment...</text><parent_chain></parent_chain></comment><story><title>IBM Invests to Help Apache Spark</title><url>http://bits.blogs.nytimes.com/2015/06/15/ibm-invests-to-help-open-source-big-data-software-and-itself/</url><text></text></story>