Thread

1. miellaby · 1791040378 · focus · HN ↗

   The paper explains absolutely everything as if it was a tutorial &quot;how to made your own modern agentic LLM&quot;. They even tell how they made their dataset. <a href="https:&#x2F;&#x2F;aleph-alpha.com&#x2F;downloads&#x2F;tech-report.pdf" rel="nofollow">https:&#x2F;&#x2F;aleph-alpha.com&#x2F;downloads&#x2F;tech-report.pdf ; It&#x27;s the first time I see this level of openness.

   1. ivo-42 · 1791048754 · focus · HN ↗

      I worked on Kolibri, in particular pre-training data and mid-training. We strive to be as open as possible. Glad you like it.

      1. brcmthrowaway · 1791049247 · focus · HN ↗

         How do you cleanse the data at this scale?

         1. ivo-42 · 1791049791 · focus · HN ↗

            By various forms of deduplication (exact, fuzzy, substring), heuristic filters and distilling quality classifiers that annotate our data. Synthetic rephrases can also be considered a form of cleaning&#x2F;getting more out of existing noisy data.

            We have a lot of details in the tech report if you want to go deeper.

            1. xvfLJfx9 · 1791059217 · focus · HN ↗

               Are there plans to make much larger versions of this model? With 500B-1T params for general purpose knowledge tasks, similar to the current leading proprietary models?

               1. idiotsecant · 1791060311 · focus · HN ↗

                  How exactly would an open project do that?
               2. ivo-42 · 1791068788 · focus · HN ↗

                  Stay tuned, this is just the beginning. Merging with Cohere will help in scaling, too.

Unofficial Hacker News client; not affiliated with Y Combinator.