Thread

1. juancn · 1789670322 · focus · HN ↗

   I wonder how (and if) continuous learning models will achieve stability.

   They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run

   1. pixl97 · 1789671312 · focus · HN ↗

      There are two questions about that stability I have.

      One, things like catastrophic forgetting and falling into incoherence.

      Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.

      1. fc417fc802 · 1789699823 · focus · HN ↗

         Aren't such attractors also problems during training? Presumably alignment constraints would need to apply to continuous learning as well.

         1. pixl97 · 1789702561 · focus · HN ↗

            I mean yes, but that's far more difficult than one would expect in a layered system. You may be able to keep a concept aligned, but can you keep the meta concept aligned? Continuous learning means there is continous opportunity for a more powerful system of misalignment to form itself and take control of your alignment. Even worse is hidden layers of this misalignment that can hide itself by not using an interpretable language.

Unofficial Hacker News client; not affiliated with Y Combinator.