Thread

1. walrus01 · 1785887768 · focus · HN ↗

   I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.

   "please write 250 words on the etymology and history of the word schlong"

   <a href="https:&#x2F;&#x2F;pastes.io&#x2F;uhshFgn4" rel="nofollow">https:&#x2F;&#x2F;pastes.io&#x2F;uhshFgn4

   The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.

   For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word&#x27;s real origins and how it has made it into casual profanity&#x2F;vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic&#x2F;religious origin in the US NE who introduced it into wider use.

   Ask it for something that&#x27;s not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.

   chat test link: <a href="https:&#x2F;&#x2F;chat.deepgrove.ai&#x2F;">https:&#x2F;&#x2F;chat.deepgrove.ai&#x2F;

   1. brainless · 1785895108 · focus · HN ↗

      Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.

      1. walrus01 · 1785896936 · focus · HN ↗

         I don&#x27;t, really, but 20B is also not that small... It&#x27;s an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang.

         It&#x27;s also something I&#x27;ve seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.

         1. sznio · 1785914151 · focus · HN ↗

            Parameter count is not everything.

            20b parameters * 1.5 bits per parameter is just 30 billion bits, about 3.75gb

            a full 20b fp16 is about 40GB.

            I find it weird how a smaller model still produces decent text, except it bullshits all the way.

      2. HelloUsername · 1785914075 · focus · HN ↗

         &gt; Ask models to search on the Internet and answer?

         To me, the benefit of running small models is that they fit on your device exactly that you don&#x27;t need any internet connection. It&#x27;s all local and offline, so you can still consult for information in any scenario.

         1. CTDOCodebases · 1785918808 · focus · HN ↗

            I thought the benefit of small models is that they are a natural language UI to whatever they are connected to.

            It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.

         2. nl · 1785926572 · focus · HN ↗

            This is a very bad way of thinking of it. Small LLMs have clues about real knowledge but only surface level answers will be accurate.
         3. dofm · 1785928180 · focus · HN ↗

            The real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds.

            If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update.

            This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.

         4. unglaublich · 1785931052 · focus · HN ↗

            True but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.

            1. em500 · 1785932052 · focus · HN ↗

               I wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate &#x2F; interpolate a lot of the gaps.
      3. spider-mario · 1785914636 · focus · HN ↗

         Maybe we don’t necessarily expect them to answer from their own knowledge, but to either do that or say “I don’t know”.
      4. nozzlegear · 1786036755 · focus · HN ↗

         It would indeed be better to have them search the internet for an answer, but the issue (as I see it) is overconfident models don&#x27;t know when they should search the internet because they think they already know the answer.

Unofficial Hacker News client; not affiliated with Y Combinator.