Thread

1. pjankiewicz · 1790156068 · focus · HN ↗

   What I'm missing here is also type guarantees. I don't think you can do it without token level logic which forces the model to output the tokens from a predefined pool of tokens. A logic like this given some JSON schema is not that difficult to implement. If the LLM must output JSON schema compatible value then you can also add that it doesn't "hallucinate". Which is funny too because just guaranteeing the type does not mean the model does not hallucinate but this is another story.

   1. ex-aws-dude · 1790204532 · focus · HN ↗

      The type is guaranteed because you only read the probability from your set of choice tokens

      You don't actually use the "next token" that the model chooses

      That's what the author is doing in this part

        token_ids = [model.tokenize(text=label.encode(), add_bos=False)[0] for label in labels]
        choice_logits = numpy.asarray([logits[token_id] for token_id in token_ids])
        logprobs = choice_logits - numpy.logaddexp.reduce(choice_logits)
        probabilities = numpy.exp(logprobs)
      

      This works because the model is always producing probabilities for all tokens

Unofficial Hacker News client; not affiliated with Y Combinator.