Thread

1. janalsncm · 1790744462 · focus · HN ↗

   OP, you should not have written this in Rust. It should be in PyTorch, which is by far the most popular. We can’t tell if this architecture is good or whether there is a problem in your implementation.

   You can test the whole thing for free on a GPU with Google Colab. Test both the transformer and your new architecture on a larger dataset. Something that maxes out the GPU for an hour each run.

   Also, the readme mentions keeping the same optimizer schedule which sounds nice at first but they are completely different architectures. The loss is high on the transformer, did you try raising the learning rate on it?

   In general I’m interested in parameter efficient architectures. I don’t think transformers are optimal, and indeed many improvements have been made to vanilla transformers. But if you have an idea for something better you need to show it.

   1. yjftsjthsd-h · 1790749046 · focus · HN ↗

      I dunno, I could probably be convinced to try a new tool purely on the basis of not having to deal with installing pytorch

      1. intoXbox · 1790751463 · focus · HN ↗

         I’m curious, what’s the criticism for PyTorch?

         1. jeroenhd · 1790754213 · focus · HN ↗

            I don't think it's caused by PyTorch on its own, but every AI-related Python project I try out locally manages to depend on a version of PyTorch that isn't in my disk cache yet. Having to download a gigabyte of dependencies for every project gets tiresome.

            The Rust compile cycle will probably generate a gigabyte of files locally as well, but at least they can be `rm`'d out of `target/` once it's done.

            It should be said that for this project that's entirely irrelevant of course, but seeing PyTorch has made me skip over projects on the HN homepage before and probably will again in the future.

Unofficial Hacker News client; not affiliated with Y Combinator.