‹ BackHN Continuity

Thread

Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM

277 points · 81 comments · volotat

  1. ilaksh · · focus · HN ↗
    If you actually scroll through the transcript he links to, you will see that something that looks like it could be training is happening, but no coherent responses are coming out at any point. At least not that I saw skimming through.

    That might explain why there are no benchmarks of any kind.

    1. synctext · · focus · HN ↗
      Using the term AGI and not including any performance analysis. My AI calls it: "massive marketing overreach". Somebody called this slop in the comments.

      As a professor who published on continual learning I'm leaning towards agreement[1]. It lacks any substance. No relation to related work, no description of algorithm, no ablation study, just hand-waving that we're feeding some data and "Chess is not forgotten".

      This "how-continual-learning-works" markdown text is not an algorithm [2].

      [1] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2301.12530" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2301.12530

      [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;volotat&#x2F;mini-AGI&#x2F;#how-continual-learning-works" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;volotat&#x2F;mini-AGI&#x2F;#how-continual-learning-...

      1. ilaksh · · focus · HN ↗
        Actually I&#x27;m mad that I wasted my time looking at it based on the claims. He implies it is trained and uses the term &quot;AGI&quot; and &quot;continuous learning&quot;. He never finished a single training run or enough that he considers not &quot;undertrained&quot;. It&#x27;s not trained. And actually there is no evidence that it can actually learn anything useful.
        1. synctext · · focus · HN ↗
          Indeed this is wasting HN time.

          &quot;The model reads 524,000 characters of chess&quot;. This is 100KByte of training data in a toy model with rigid parameters and no global learning. Gap with real LLM and trillions of tokens.

          This model really addresses the problem of preserving previously learned knowledge, but by restricting the LR of the trunk it stops acquiring new knowledge. Details: &quot;Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective&quot;

      2. volotat · · focus · HN ↗
        There is no special algorithm, the finding is that slowing down the LR or the trunk, while keeping the LR of the experts is enough to eliminate most of the forgetting in the network. You can see in that experiment where chess data was the only thing the model read for 524K characters, yet it kept almost the same performance (i.e. held-out loss) on all other domains. If you keep LR the same across the whole network the loss in other domains degrades dramatically - this is a clear sign of catastrophic forgetting in action. What I can say for sure is that any traditional network that does pose a sign of catastrophic forgetting would not be able to learn any patterns from a single stream of data.

        There are no benchmarks published as the model is heavily undertrained, but it is learning. And you can see this clearly in the loss and samples even though they are still barely coherent.

        I am not an academic and am not trying to publish a paper about a “major breakthrough” or something like this. I am just a small person who found a cool thing that clearly works and wants to share it with the world. That’s it.

        1. ilaksh · · focus · HN ↗
          You can&#x27;t claim it &quot;works&quot; if it hasn&#x27;t produced any coherent responses and is still early in your first training attempt.
          1. volotat · · focus · HN ↗
            It is a goalpost that is easy to move. By &quot;works&quot; I mean learning from a continuous single (meaning batch-1) stream of data. The fact that it produces full words and full coherent phrases instead of a random stream of characters that would any typical LM produce if trained under the same training regime.
            1. ilaksh · · focus · HN ↗
              I would be okay if you shared it as a potential idea and possibly interesting early result, but the language you are actually using to characterize it is misleading or delusional.

              Please get a model to the point where it seems like it has some natural language understanding and then share again with reasonable characterization.

              1. volotat · · focus · HN ↗
                For sure. As it will pass through the whole corpus I will share the weights, run it through established benchmarks for small models and share all of this as an update. I am also planning on making a Youtube video explaining in detail how it works on a deeper level and the whole reasoning behind why it is built the way it is. But no promises here.
                1. fuzzfactor · · focus · HN ↗
                  Nothing to be ashamed of if you end up pushing back the release date of your feature film :)

                  I had ideas not completely unlike this so long ago, but one big difference can be summed up in one of your parameters.

                  &gt;Directories are walked, binaries are skipped . . . and each file is read from its beginning to its end because a document has an order.

                  For me it was binaries being walked because text and anything approaching a language model was so much further out-of-reach having such limited computer power.

              2. [deleted] · · focus · HN ↗

                [deleted]

              3. bigbadfeline · · focus · HN ↗
                Is &quot;AGI&quot; the language that bothers you? Well, one has to keep his eyes on the prize and I see a bright idea which could lead to AGI, so, why not describe it as such? I also see the inspiration and hard work necessary to move that idea further along, so fingers crossed.
    2. seanhunter · · focus · HN ↗
      To pick a couple of examples:

         --- chess ---
   prompt: &#x27;&lt;g&gt;1700 1-0 1. e4 e5 2. &#x27;
   [raw]  repeated 8-grams 83%
   Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Rxd5 Rxd5 Rxd5 Rxd5. Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 31 Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 Rx
   [adapted]  repeated 8-grams 2%
   Nxd6+ Bxc3+ Nf6 14. Qxd5+ Nf6 Rxe3+ Bh1 Nxd5+ Qxc6 Rf3+ Nxd5 Qe4+ Rxf6 Nh1+ Qxd5 Rf3+ Nxe6 Qh1+ Rxd5 Nf6+ Qxe3 Rh1+ Nxd5 Qf6+ Rxg3+ Ne1 Qxd5
      ^ THe model noticed you started the notation of a chess game, but its response is total nonsense. After &quot;1. e4 e5 2.&quot; you can&#x27;t go Nxd6+. For all kinds of reasons. You haven&#x27;t got your knight out yet. Even if you had, it couldn&#x27;t get to d6. Even if it could, there&#x27;s nothing there it could take. If you did somehow in spite of all that manage to play 2 Nxd6+ the opponent couldn&#x27;t play .... Bxc3+ because they haven&#x27;t got their bish out. Even if they had it couldn&#x27;t get to c3 even if it could there isn&#x27;t anything there to take - you only have a pawn on e4 and a magical knight on d6. Even if somehow in spite of that, you could take on c3 it wouldn&#x27;t be check and EVEN IF SOMEHOW ALL OF THAT WERE TRUE YOU ARE IN CHECK. You can&#x27;t move your bishop you need to do something about the Knight on d6 which has you in check.

      All the rest of it is similarly gibberish. I&#x27;m used to model training garbage but this is in no sense AGI. It&#x27;s beyond nonsense to call it that.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.