Thread

ESP32S3 cluster running 1.58-bit (BitNet) Language model

148 points · 31 comments · nkko

Loading the complete thread in the background. This saved snapshot is available now. Refresh

cameron_b · 1790642885 · focus · HN ↗
It is a bit of a bummer to see that the degree of 'compression' makes it a fancy llm noise-maker. It is still charming.
sjakati98 · 1790646112 · focus · HN ↗
Gemma 4 when?
1. nkozyra · 1790655104 · focus · HN ↗
  We're gonna need a bigger ESP.
  1. pantalaimon · 1790675087 · focus · HN ↗
    <a href="https://www.espressif.com/en/products/socs/esp32-p4" rel="nofollow">https://www.espressif.com/en/products/socs/esp32-p4
  2. nkko · 1790679240 · focus · HN ↗
    looking forward to try this one <a href="https://www.espressif.com/en/products/socs/esp32-s31" rel="nofollow">https://www.espressif.com/en/products/socs/esp32-s31
tdhz77 · 1790649025 · focus · HN ↗
Soon ai in every lightbulb running Kubernetes
1. tombert · 1790656187 · focus · HN ↗
  You know, I've always liked Futurama but I always kind of thought it was silly that literally everything has an AI and a personality.
  But, you know, I actually think that there might be a logic to it. Economies of scale might mean that almost-literally every computer you buy in the year 3000 has some kind of AI-assistance chip in there, and sure maybe it will have full AI with a personality spitting out one-liners.
  1. abroadwin · 1790659047 · focus · HN ↗
    Kind of like how disposable vape pens often have a 24 MHz Cortex-M0+ with 3 kB SRAM and 24 kB flash, which would have seemed ludicrous a while back.
  2. KeplerBoy · 1790662090 · focus · HN ↗
    Change the year 3000 to the 2030s and it might be just as accurate.
2. oneZergArmy · 1790662048 · focus · HN ↗
  Praise the Omnissiah.
  1. Tade0 · 1790664958 · focus · HN ↗
    With the proliferation of Abominable Intelligence? Quite the contrary!
3. _joel · 1790681000 · focus · HN ↗
  How many rollingupdate pods does it take to change a lightbulb?
  1. jagged-chisel · 1790684074 · focus · HN ↗
    Eventually.
matthewfcarlson · 1790651141 · focus · HN ↗
I’m actually working on a small project that’s exactly this! Less quant so it’s only 150M parameters but this is amazing.
NDlurker · 1790651214 · focus · HN ↗
I'm curious how this would handle grammar checking on a basic word processor. Or maybe generate worlds for small text based games. I have no idea what the capabilities are of a cluster like this.
nonasking_ · 1790655204 · focus · HN ↗
Thanks for sharing. It's fascinating to see a 0.5B LLM being split across seven ESP32s like this.
aidiscoverywire · 1790661664 · focus · HN ↗
[dead]
librasteve · 1790662201 · focus · HN ↗
haha … this is precisely the kind of project that <a href="https://bil-lang.org" rel="nofollow">https://bil-lang.org is aimed at: Go for parallel (ie in this case pipeline processing).
don’t get too excited until we get the TinyGo backend built though ;-)
ladyanita22 · 1790663623 · focus · HN ↗
This is something I've been fantasizing about for long.
Let's say we took Rust, a language that makes parallelization easier than others (as it helps you avoid some common footguns). How difficult would it be to have a massively parallel computer system made out of many tiny, simple microcontroller-like chips? Let's say we picked many little Risc-V's. Surely this would be an interesting experiment (though I'm not sure whether it'd make economic sense or not...)
1. sigmoid10 · 1790664903 · focus · HN ↗
  It would certainly not make any economic sense, and I guess that's also why noone is seriously looking into stuff like volunteer/enthusiast clusters of home computers to do inference in the same way that e.g. LHC@home works. The main bottleneck for LLMs is still memory bandwidth. Any memory bus not directly soldered on your GPU is terribly slow. That's why one big GPU with twice the VRAM will always perform significantly better than two ones with half the VRAM each. If you spread it out over dozens or hundreds of tiny chips, you'll be wasting most processing time.
2. ur-whale · 1790669388 · focus · HN ↗
  > How difficult would it be to have a massively parallel computer system made out of many tiny, simple microcontroller-like chips?
  It's scaling the communication that becomes hard.
  In this project they daisy-chain SPI. I don't believe that would scale very far.
3. akavel · 1790672728 · focus · HN ↗
  See GreenArrays' 144-core Forth chips by Chuck Moore.
4. alfanick · 1790688023 · focus · HN ↗
  Look at Xmos, founded by a transputers-dad. Small uCs, that can be connected together into a massive cluster, while being first-citizen of their xC language.
5. tyingq · 1790688199 · focus · HN ↗
  Not exactly what you're describing, and not shipping yet, but a cluster in a box. 8 cores/node, 8 nodes.
  <a href="https://milkv.io/cluster-08" rel="nofollow">https://milkv.io/cluster-08
6. godojo · 1790689269 · focus · HN ↗
  Man I miss Slashdot's Beowolf culsters
7. [deleted] · 1790700870 · focus · HN ↗
  [deleted]
8. giancarlostoro · 1790701128 · focus · HN ↗
  Mojo might be what you want, especially with "MAX" which is their AI modeling framework, where in other languages you need NVidia's libraries, or AMDs, etc the Max libraries just let you talk directly to the GPU / CPU / ASIC with Mojo. I think Mojo is very underrated in this space right now, but assuming they don't mess it up, it could be a major contender in AI.
  <a href="https://max.modular.com/" rel="nofollow">https://max.modular.com/
sneak · 1790665787 · focus · HN ↗
Seriously though, what are the low cost chips that can usefully run LLMs? Is a Mac Mini the lowest we can go? Are there iGPUs on mini-itx that can do it, or are there dedicated AI chips that one could turn into a pi HAT?
1. Risse · 1790669436 · focus · HN ↗
  Depends on what you consider to "usefully run LLMs".
  Earlier this year, I bought a mini pc from Aliexpress, specs are roughly Ryzen H255, 24GB LPDDR5, 1TB SSD. This was around 350€ including VAT, customs, shipping etc. I would personally consider this somewhat of a lowest class of useful LLM box. It can run 8B models well, up to somewhere around 24B. I currently run Gemma 4 26B A4B Q5 on it, with MTP, and it is quite slow, but smaller models would run okay on it.
  1. bahmboo · 1790680788 · focus · HN ↗
    Let's teleport back 10 years and what you have is a magic box that could make you billions.
2. chorylee · 1790686952 · focus · HN ↗
  An Orange Pi 5 Max does this job for real. It's an RK3588 board — $75 for 4GB, $95 for 8GB on AliExpress. A community test got Qwen2.5-0.5B at about 12 tok/s on the CPU via llama.cpp. The chip also has a 6 INT8 TOPS NPU if you'd rather go the RKNN route. Won't beat a Mac Mini, but it's an actual computer for under a hundred bucks.
  1. cameron_b · 1790694757 · focus · HN ↗
    I regret to inform you that prices have long departed the lower atmosphere. Even on AliExpress, you're looking at a few hundred bucks for those. Still less than a Mac Mini, but less less.
AmazingTurtle · 1790686239 · focus · HN ↗
"Your scientists were so preoccupied with whether they could, they didn't stop to think if they should"

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.