Thread

1. segmondy · 1791067001 · focus · HN ↗

   I wonder if some of these could carry over for LLM inference. It will be nice to turn more ewaste GPUs into capable processing units for LLM.

   1. saghm · 1791073789 · focus · HN ↗

      I have a Radeon RX 6900 XT (16 GB VRAM, originally released in 2021), and it's possible to run some lightweight models to have okayish performance and quality of output, but nothing I've tried has come anywhere close to the quality even of the models I can use for free from OpenCode Zen or the free tier of Openrouter. If you want to keep everything local on the same card I have, it requires putting up with a model that's noticeably worse in virtually every metric than what you can get for free elsewhere, and the GPUs this article are talking about are three times as old as mine.

      It would be awesome if someone manages to figure out how to get small enough models to fit on older cards to be viable, but I'm not optimistic that it will come without some sort of fundamental architectural innovation rather than incremental improvements, and it's not clear if and when that will happen.

      1. BLKNSLVR · 1791076452 · focus · HN ↗

         This kinda highlights the level of debt the AI companies are in, and will continue to be in, offering anything for free.

         How long is this runway?

         1. saghm · 1791076543 · focus · HN ↗

            I have no clue, and I agree that it does not seem sustainable. Either someone needs to find a magic solution to making it a lot cheaper, or a lot of companies are going to need a lot of money from somewhere that isn't clear.

Unofficial Hacker News client; not affiliated with Y Combinator.