Thread

1. Jackson__ · 1791138774 · focus · HN ↗

   I've just tested Strata on a simple 50 image vision benchmark. The task is to output the exact coordinates of a requested object. The result via Strata had a median error distance of 154.8 pixels, avg of 168.8. Running the exact same GGUF and vision adapter weights on llama.cpp gives me a median error of 46.5, avg 81.4.

   To put that into perspective, here are some more numbers from other models via llama.cpp:

   Median/Average

   Qwen 3.5 9B BF16: 46.5 / 193.3

   Qwen 3.6 35B Q4 K XL: 38.4 / 76.4

   Qwen 3.5 122B Q3 K M: 32.9 / 68.6

   The difference in vision performance is as large as the jump from a 9B model to a 35B model. All tests were performed at temp=0.

   I have done no further testing, as these results line up perfectly with my expectations.

   1. larodi · 1791178923 · focus · HN ↗

      To be done right, such vision benchmark should consider the peculiarities of the vision tower, and was it quantized and optimized. So a lot of success/loss of quality may not be due to the transformer.

      Second of all, the "find coordinates" of something is super difficult task of any model, so you tried to test a small quantized buddy with a tri-star challenge. Not sure what expectations were set.

      disclaimer: myself do large volume VLM work daily, including in production, for more than 1.5 years now.

      1. Jackson__ · 1791185378 · focus · HN ↗

         >To be done right, such vision benchmark should consider the peculiarities of the vision tower

         As per my previous comment, I used the _exact same weights_ for the comparison. The vision tower is always kept as an unquantized BF16 file for GGUF, as I believe is the default.

         >Second of all, the "find coordinates" of something is super difficult task of any model

         It is in fact not (anymore), and most recent VLMs I tested have been standardized to point and bbox in a relative 0..1000 coordinate system with decent enough accuracy.

         And finally, none of this excuses that Strata performs so much worse with the same weights. Which does kinda leave me confused as to what the point of this reply is.

         1. larodi · 1791191891 · focus · HN ↗

            > Which does kinda leave me confused as to what the point of this reply is.

            to point out that 1)measuring VLM is not straight-forward, and needs considering what was actually ablated.

            and that 2) perhaps such metric is the most challenging one for a brutally quantized and otherwise lobotomized model.... which otherwise performs well in other tasks (which I also doubt - for the record!).

            i can subscribe to the idea that the Strata performs worse because it stripped the model of important quality, but not entirely to the fact it was really properly evaluated.

            the way such models/attempts degrade can_be/is insightful on its own.

Unofficial Hacker News client; not affiliated with Y Combinator.