‹ BackHN Continuity

Thread

Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM

277 points · 81 comments · volotat

  1. abeppu · · focus · HN ↗
    I have not looked carefully but it seems like this is over-promising on avoiding catastrophic forgetting.

    The "trunk learning rate" is set at 0.1x the learning rate for the experts, so learning on different subjects disproportionately happens in the experts, and the trunk portion is comparatively more stable. But the population of experts can grow and shrink:

    > The pool grows when it is short of capacity and shrinks when parts of it stop being asked for.

    So:

    - doesn't the trunk then _eventually_ still undergo catastrophic forgetting, it just may take much longer?

    - and before that point, catastrophic forgetting happens in stepwise chunks whenever the expert pool shrinks?

    1. volotat · · focus · HN ↗
      I am not sure about eventually, but it learns on the steady paste so far. The main thing to keep in mind is that it learns on the single STREAM of data. Not randomized batched samples. Try to do it with any other model and you will see nothing but complete garbage in the predictions, exactly because of catastrophic forgetting.

      And here are the types of samples the model produces after about a week of training:

          ==============================================================================
    step 191,447   391.3M of 7,879M characters (4.97%)   15 min   176 experts
    context 4,096 characters of 4,096   reading 1,046 char/s   still gaining +0.0412 deep into it
    grad norm 0.98 against a clip of 1   under the clip
    train loss 0.6540   lr 2.28e-04   evidence t -0.15 over 65.7 (effect +0.0660)   rate x0.753
    held-out loss 0.8012 +/-0.0335 nats   1.1559 bits/char   perplexity 2.23   gap +0.1472
      arithmetic 0.645   chat 0.790   chat_hermes 1.153   chess 0.510   code 0.703   reasoning 0.756   stories 0.596   wikipedia 1.256
    repeats 31% of 8-grams, greedy with no guard
    ==============================================================================

    --- stories ---
    prompt: 'Once upon a time, there was a little boy named Tom. One day he '
    [raw]  repeated 8-grams 2%
    was walking in the park with his mom. He saw a big box in the ground. He was very sad and sad.

    "Mom, can I have the box to the box?" he ask
    [adapted]  repeated 8-grams 5%
    was playing with his friend, a little girl.

    "Wow, Tom!" said Tom.

    "What's your name?"

    Tom said, "I'm playing with my friend."

    The little

    --- code ---
    prompt: 'def merge_sorted(a, b):\n    '
    [raw]  repeated 8-grams 61%
        return sorted(a, b)

        def merge_sorted(a, b):
            return sorted(a, b)

        def merge_sorted(a, b):
            return sorted(a, b)

    
    [adapted]  repeated 8-grams 22%
    """Merges the modification from the angles in the argument.
        This merges the angles in the argument.
        """

        def __init__(self, a):


    --- arithmetic ---
    prompt: 'add 4917 + 388 = '
    [raw]  repeated 8-grams 47%
    <think> 7+8+0=5c1 1+8+1=0c1 9+3+1=3c1 4+0+1=5c0 </think> 5305
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    mul 6 * 6 = 36
    a
    [adapted]  repeated 8-grams 3%
    <think> 7+8+0=5c1 1+8+1=0c1 9+3+1=3c1 4+0+1=5c0 </think> 5305
    mul 6287 * 49 = <think> 6287*9=56583 6287*40=251480 56583+251480=307063 </think> 307063
    mul 9428 * 17 = 

    --- chat ---
    prompt: '<user>\nWhat are you?\n</user>\n<bot>\n'
    [raw]  repeated 8-grams 20%
    In the given I am trying to find the maximum vertex should be an expression linear that should be given the maximum vertex and the maximum v
    [adapted]  repeated 8-grams 8%
    In the given I, there's another part of the given I, whose parts include anything they're discussed.
    </bot>
    <user>
    What is the given article by answer

    --- chat_hermes ---
    prompt: '<user>\nA train travels 60 km in 45 minutes. What is its speed in km/h?\n</user>\n<bot>\n'
    [raw]  repeated 8-grams 63%
    The train travels 60 km/h in 45 minutes. The train travels 60 km/h in 45 minutes. The train travels 60 km/h in 45 minutes. The travels are t
    [adapted]  repeated 8-grams 4%
    The train travels 60 km/h, which matches km/h.
    </bot>
    <user>
    Write an example of the following article:

    In the subject of the subject, include an imp

    --- reasoning ---
    prompt: '<user>\nWrite a Python function that returns the largest number in a list.\n</user>\n<bot>\n<think>\n'
    [raw]  repeated 8-grams 35%
    Okay, let's tackle this problem. So, the largest number in a list is a list. The largest number is a list. The largest number is a list. It 
    [adapted]  repeated 8-grams 20%
    Okay, let's try to figure out this problem. So, we'll find the largest number in a list.

    The largest number is always the largest number in

    --- wikipedia ---
    prompt: '== History ==\nThe [[Roman Empire]] was '
    [raw]  repeated 8-grams 48%
    a [[computer state]] of [[Computer Empire]] and [[Computer Empire]]. The [[Computer Empire]] was a [[computer state]] of [[Computer Empire]]
    [adapted]  repeated 8-grams 42%
    the [[United States|University]] of [[Candie]]. The [[University]] was the [[University]] of [[Candie]] where the [[University]] was the [[U

    --- chess ---
    prompt: '<g>1700 1-0 1. e4 e5 2. '
    [raw]  repeated 8-grams 0%   22 legal moves, then Nd3
    Nf3 Nc6 3. Bb5 a6 4. Bxc6 dxc6 5. O-O Bg4 6. h3 Bh5 7. g4 Bg6 8. d3 Be7 9. Nbd2 Nf6 10. Nb3 O-O 11. Nc5 Bxc5 12. d4 Bd6 13. Nd3 Bxf3 14. Qxf
    [adapted]  repeated 8-grams 0%   16 legal moves, then Ba3
    Nf3 d6 3. Bc4 Nf6 4. d3 Be7 5. O-O Nbd7 6. Be3 c6 7. Nbd2 O-O 8. c3 a6 9. Qc2 b5 10. Ba3 Nb6 11. Bxe7 Qxe7 12. Rac1 Bb7 13. Nf1 Rac8 14. Ng3

    --- self-knowledge ---
    prompt: '<user>\nhow do you decide which experts to use?\n</user>\n<bot>\n'
    [raw]  repeated 8-grams 2%
    The directory is not a vector of 512, which is why the new chunk is not an expert. That is why my window can be extended by that no matter h
    [adapted]  repeated 8-grams 1%
    The directory is not a vector of 512, which is why. There is not an expert involve
    </bot>
    <user>
    Can you write change_string? It should change the com
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.