<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>AmazingTurtle</author><text>Most likely a small retrieval optimized model with focus on JSON tokens and a combination of the technique involved in grammar&amp;#x2F;guidance&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;guidance-ai&amp;#x2F;guidance&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;guidance-ai&amp;#x2F;guidance&lt;/a&gt;&lt;p&gt;Same principle</text><parent_chain></parent_chain></comment><story><title>Inversion: Fast, Reliable Structured LLMs</title><url>https://rysana.com/inversion</url></story>
<instructions>Your goal is to analyze the following comment and estimate how highly it will be upvoted by the Hacker News community.</instructions><comment><author>fzysingularity</author><text>I was initially impressed with the landing page, but it does look a bit suspect when things are claimed to be 100x faster without much info on the HW acceleration or the model sizes.&lt;p&gt;My best guess is that they&amp;#x27;re using two approaches to get this running faster:&lt;p&gt;- structured generation techniques from sglang (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;sgl-project&amp;#x2F;sglang&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;sgl-project&amp;#x2F;sglang&lt;/a&gt;) that allow them to generate faster JSON (with look-ahead &amp;#x2F; pre-fill) with strong guarantees on the output (i.e. 100% reliable, without requiring any retries).&lt;p&gt;- distilling a gpt-3.5 turbo-esque model from GPT-4 JSON outputs, and using it in conjuction with above to give the additional performance boosts on inference.&lt;p&gt;It doesn&amp;#x27;t seem like they&amp;#x27;re deploying on any custom silicon, nor have they optimized GPU kernels to suggest that the speed ups came there.</text><parent_chain></parent_chain></comment><story><title>Inversion: Fast, Reliable Structured LLMs</title><url>https://rysana.com/inversion</url></story>