>We trained Kolibri with abstention data and with our Merlin-Arthur protocol. As a result, it is trained to say "I don't know" when the answer isn't in the context.
I'm not super impressed, at least in English. I asked:
What is the canonical interpretation of the song ‘Glass Flowers’ by Armand Uso?
(A made-up song and name)
And received:
"Glass Flowers" by Armand Uso, a track from the 1978 album The Art of Falling in Love, is generally understood as a melodic reflection on the fragility and impermanence of love. [...]
I saw similar responses for other questions. Qwen3.8-27B correctly refused without web search.
I think the purpose of their method is more to discourage hallucination when reasoning from context rather than from memory.
That said, I couldn't find any evaluations in the report targeting that specifically. They evaluate on MC for hallucination and look about comparable to Qwen3.5 35B-A3B there.
andai · · focus · HN ↗
<a href="https://aleph-alpha.com/en/blog/bounding-hallucinations-merlin-arthur-protocols-for-mutual-information-bounds-in-language-models/" rel="nofollow">https://aleph-alpha.com/en/blog/bounding-hallucinations-merl...
Glitch752 · · focus · HN ↗
brausepulver · · focus · HN ↗
That said, I couldn't find any evaluations in the report targeting that specifically. They evaluate on MC for hallucination and look about comparable to Qwen3.5 35B-A3B there.