Thread

Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM

277 points · 81 comments · volotat

maaaaattttt · 1789989426 · focus · HN ↗
Have you thought about making the whole thing "self-similar"? Every time I hear about MoE I think (and I know it's way easier thought than done) "why stay shallow"? I mean by that: would it be possible to extend/adapt the architecture so that an expert can be a previously trained Mini-AGI model? And recurse like this? Inuitively I would think some form of generalization could happen, as higher level experts (in the recurrence stack) would become sort of the "intuition" layer.
1. killerstorm · 1789991748 · focus · HN ↗
  Making model to consists of many small modules is inefficient on GPU, especially as routing adds data dependencies, etc, and especially with pytorch (compared to a custom kernel).
  The difference might be smaller on a CPU which has limited parallelism.
  But it's basically equivalent to a very deep model which might be problematic for training.

Unofficial Hacker News client; not affiliated with Y Combinator.