Thread

Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

169 points · 52 comments · edwardbzhang

walrus01 · 1785887768 · focus · HN ↗
I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.
"please write 250 words on the etymology and history of the word schlong"
<a href="https://pastes.io/uhshFgn4" rel="nofollow">https://pastes.io/uhshFgn4
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
chat test link: <a href="https://chat.deepgrove.ai/">https://chat.deepgrove.ai/
1. johnsmith1840 · 1785892419 · focus · HN ↗
  lol I like the first one though. Reads like a great sarcasm response.
  I wonder if kids will do this to their parents.
2. brainless · 1785895108 · focus · HN ↗
  Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
  1. walrus01 · 1785896936 · focus · HN ↗
    I don't, really, but 20B is also not that small... It's an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang.
    It's also something I've seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.
    sznio · 1785914151 · focus · HN ↗
    Parameter count is not everything.
    20b parameters * 1.5 bits per parameter is just 30 billion bits, about 3.75gb
    a full 20b fp16 is about 40GB.
    I find it weird how a smaller model still produces decent text, except it bullshits all the way.
  2. HelloUsername · 1785914075 · focus · HN ↗
    > Ask models to search on the Internet and answer?
    To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
    CTDOCodebases · 1785918808 · focus · HN ↗
    I thought the benefit of small models is that they are a natural language UI to whatever they are connected to.
    It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.
    nl · 1785926572 · focus · HN ↗
    This is a very bad way of thinking of it. Small LLMs have clues about real knowledge but only surface level answers will be accurate.
    dofm · 1785928180 · focus · HN ↗
    The real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds.
    If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update.
    This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.
    unglaublich · 1785931052 · focus · HN ↗
    True but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.
    em500 · 1785932052 · focus · HN ↗
    I wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate / interpolate a lot of the gaps.
  3. spider-mario · 1785914636 · focus · HN ↗
    Maybe we don’t necessarily expect them to answer from their own knowledge, but to either do that or say “I don’t know”.
  4. nozzlegear · 1786036755 · focus · HN ↗
    It would indeed be better to have them search the internet for an answer, but the issue (as I see it) is overconfident models don't know when they should search the internet because they think they already know the answer.
3. api · 1785895377 · focus · HN ↗
  Small or overly quantized LLMs are a genre of humor. Same goes for small image generators. Janky generative AI is like the Geocities web pages of today.
4. getpokedagain · 1785897463 · focus · HN ↗
  The schlong test is nearly as funny as drawing shit on bicycles test
  1. walrus01 · 1785897557 · focus · HN ↗
    I've also been asking LLMs to draw SVGs of literal pelican cases and the results can be more amusing than pelicans on bicycles. You can get pelican cases with cameras, firearms, long cases, square cases, cases that look nothing like pelicans (but more like Zero Halliburton aluminum briefcases), etc. You also get cases that are open or closed depending on the whims of the LLM.
5. boomlinde · 1785916385 · focus · HN ↗
  Seems like less of a problem in smaller models where bullshit tends to become very obvious to anyone with half a clue about the given subject than it is in larger models where the illusion is complete enough that the confidently stated answers are very incorrect in more subtle ways.
6. sajithdilshan · 1785917321 · focus · HN ↗
  I think for smaller models, they need to be more defensive on unknown information and frontier model level tool calling capabilities.
  LLMs are kind of a compact knowledge box of its training data and it's understandable it would not have information about every topic and in that case just do a web search or a proper tool invocation to get the data and then synthesize.
  1. miohtama · 1785935164 · focus · HN ↗
    You need a small LLM that can reason very well and use tools like web search very well.
    No one is going to compress human knowledge into few bits.
    sajithdilshan · 1785937073 · focus · HN ↗
    Not at the moment, but who knows what types of models and storage types we would have in 100 years.
    jasonjmcghee · 1785939915 · focus · HN ↗
    It's far more likely basic consumer devices will advance such that much larger models efficiently run than finding novel ways to compress all of human experience to fit on today's mobile hardware.
    Information can only be compressed so much
    wolttam · 1785940632 · focus · HN ↗
    We’re already seeing incredible knowledge compression out of the models the GP mentioned like Qwen 35B-A3B, which feels well within the realm of “runs on a phone” in the next handful of years.
    And by then we’ll probably have been further surprised by just how much information and capacity for reasoning can be crammed into a few gigs of weights. Models just keep getting better for a given size, it’ll be interesting to see where the limit of that is.
7. hrmon · 1785921344 · focus · HN ↗
  English Wikipedia: "As of 16 October 2024, the size of the current version including all articles compressed is about 24.7 GB without media."
  Models in this size range should aim for correct tool calling and avoiding hallucinations, not universal knowledge. (apparently they don't.)
8. thomastjeffery · 1785937929 · focus · HN ↗
  They are not incorrect. They don't even hold a position in the first place.
  An LLM continues the prompt it is given. What is more likely to come after a question? An answer, not an "oh sorry I'm not sure". Sure, you could make the latter more likely, but then the model would be unusable. Larger models simply contain more answers, more ways to stumble into them, and a granular enough geography to stay on the trail.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.