Thread

1. Jackson__ · 1791138774 · focus · HN ↗

   I've just tested Strata on a simple 50 image vision benchmark. The task is to output the exact coordinates of a requested object. The result via Strata had a median error distance of 154.8 pixels, avg of 168.8. Running the exact same GGUF and vision adapter weights on llama.cpp gives me a median error of 46.5, avg 81.4.

   To put that into perspective, here are some more numbers from other models via llama.cpp:

   Median/Average

   Qwen 3.5 9B BF16: 46.5 / 193.3

   Qwen 3.6 35B Q4 K XL: 38.4 / 76.4

   Qwen 3.5 122B Q3 K M: 32.9 / 68.6

   The difference in vision performance is as large as the jump from a 9B model to a 35B model. All tests were performed at temp=0.

   I have done no further testing, as these results line up perfectly with my expectations.

   1. NamlchakKhandro · 1791146657 · focus · HN ↗

      Tldr, strata is a waste of time.

      1. unlikelytomato · 1791159746 · focus · HN ↗

         at least for vision? Are there similar comparisons for language? It seems like vision is often an afterthought when it comes to bootstrapping these newer inference engines

         1. Borealid · 1791161286 · focus · HN ↗

            Vision functions the same way as language when inference is done. It's a stream of tokens.

            It's just easier to measure the "right" answer (and deviation therefrom) on a vision task than a language one due to the underspecified nature of language.

            1. unlikelytomato · 1791162736 · focus · HN ↗

               my understanding is the vision component of the original model is an independent preprocessing step(except for things like Gemma 12b). And it would be possible for engine to have broken that phase of translation before it hits the real model as encoded text. I am not familiar with Strata engine and their claims, but this seems like an interesting way to test models with relatively straightforward inputs.

               1. dr_kiszonka · 1791182346 · focus · HN ↗

                  Unsloth documented the vision components degrading more during quantization for one of the earlier Quen or Gemma models.
            2. pjc50 · 1791181936 · focus · HN ↗

               How does image tokenizing work?
   2. throwaway219450 · 1791150391 · focus · HN ↗

      What does SAM3 get on the same test set?
   3. Xenograph · 1791151198 · focus · HN ↗

      Is this test available somewhere? Would like to test it out on my models.
   4. biztos · 1791169941 · focus · HN ↗

      How big are the images? Is 160px 1%, 10%…?
   5. larodi · 1791178923 · focus · HN ↗

      To be done right, such vision benchmark should consider the peculiarities of the vision tower, and was it quantized and optimized. So a lot of success/loss of quality may not be due to the transformer.

      Second of all, the "find coordinates" of something is super difficult task of any model, so you tried to test a small quantized buddy with a tri-star challenge. Not sure what expectations were set.

      disclaimer: myself do large volume VLM work daily, including in production, for more than 1.5 years now.

      1. knollimar · 1791183628 · focus · HN ↗

         What type of work do you do? Do you consider shelling out to a tool where they can place markers and retry?

         Do you have any other benches? What models do you use or recommend for this type of work?

Unofficial Hacker News client; not affiliated with Y Combinator.