Thread

1. boutell · 1790944763 · focus · HN ↗

   I head a much, much smaller open source project. Since the November Singularity we've been seeing at least six responsibly reported security advisories a month. However, this last month we had 22 unique security advisories. Our project has been built with adherence to the OWASP Top Ten Guidelines and other best practices from the beginning. But software is hard and AI is thorough.

   Each month, we fix them all in our monthly maintenance release and disclose at that time. We fight AI fire with fire, and hand-review, of course.

   So far, we can keep up. One hopes this is possible at the scale of the Linux project, which assuredly has more humans and more AI to throw at the problem. But team size does not scale linearly with interested audience, and potential bugs do scale with codebase size (and other extremely important factors, like code quality, at which the Linux team is assuredly much better than we are).

   ("November Singularity" is a cheeky reference to the arrival of Opus 4.5 and "good enough" coding models and harnesses generally.)

   1. Roark66 · 1790958524 · focus · HN ↗

      I wonder if August of this year will be remembered like that too. The very first "opus like" local AI model came out this August (Qwen 3.8 Flash Next). I've been running it locally since for real programming and I consider it pretty much the same as opus 4.6 in coding ability (it lacks a bit in the factual knowledge area). It even exceeds opus on some tasks.

      I've tried every hyped "open source" model before and all including latest models bigger than 1T parameters are pretty much toys.

      This is the first one that isn't. It can't be overstated how huge of a deal that is. No more reliance on Anthropic.

      Running this model to do real work is still not cheap. I run it on a pc with 6 rtx3090s and 192GB of RAM (and I use 90gb of that ram for KV cache). The model is entirely on gpus. It runs at 55tok/s 1500 prefill for one user at a time, and about 35tok/s 650 prefill per user for 5 simultaneous users. It doesn't seem like much until one realises you manage your own kv cache. You ca leave your sessions in cache for as long as you want. You can save them and restore 200k sessions a week later in a dozen seconds.

      What many people don't realise is that usage of those models skews extremely heavily towards input processing. My claude code usage is about 1.3B tokens input per week and only about 8M output. On claude code I get 80% cache. At home it's more like 95%.

      If this progress keeps up, and we get a fable quality model in a year in under 200B to run at home... Those "frontier labs" will be renting all of their gpus per hour not to go bankrupt.

      1. QwenGlazer9000 · 1790965281 · focus · HN ↗

         What Quants do you run?
      2. roosterIllusi0n · 1790972245 · focus · HN ↗

         I am using qwen3.8-27B-UD-Q3_K_XL on a 5080 16gb card with 16gb of system ram at around 43-45tok/s with 65k context. I am using that model to set up containers on a proxmox server with two b70s and 96gb ram. The Q3 model implemented 10 different chat models, 2 image models, and 7 web based harnesses so I can compare. It can rewrite any part to improve it.

         qwen3.8 is more than capable for software dev. The gemma models were terrible and fell apart during compaction. I think as long as the llm can test the results, you don't need anything being offered by a "frontier" model. The cloud AI is going to be used by people unable to run their own and they will eventually get squeezed on price.

         The one tip I could give is compact before starting new steps or any action in the plan that is different than what was previously worked on. You want to manage what is in the context and don't want unnecessary work history details filling it up. You can always ask the llm to list the current plan, then compact after and do it more than once until you get the compaction <30%. This will be fixable by the harness that can choose better times to compact.

         You don't want to start a new phase and have it compact a few minute after starting. This happening over and over again seems to potentially cause issues for long running sessions. Compaction slop that screws up what is in the context.

         I can compare what I use at home vs paid models like astra at work and the difference is mostly meaningless.

         1. vlovich123 · 1791048660 · focus · HN ↗

            What harness do you use?
      3. lukan · 1791013115 · focus · HN ↗

         "Those "frontier labs" will be renting all of their gpus per hour not to go bankrupt."

         And those who go bankrupt, can sell their GPU's cheap, so I can indeed finally have my own fable.

Unofficial Hacker News client; not affiliated with Y Combinator.