I'm not exactly following through with the claim, can someone explain how the built-in classification would not necessitate more tokens used, or be much different from turning on reasoning? Not that I don't see the difference, I just doing see how OpenAI would do it well.
It's hard to say without knowing their architecture, but I'd guess something like block attention. You can process the prompt separately from the classifications into a latent space and then do some kind of late interaction with the encodings from the classifications.
There are plenty of other ways to do zero shot classification that would result in more "token usage" (really just having to reprocess everything for each class), but the pricing and the way they describe it narrows it down somewhat.
tolugenius · · focus · HN ↗
deepsquirrelnet · · focus · HN ↗
There are plenty of other ways to do zero shot classification that would result in more "token usage" (really just having to reprocess everything for each class), but the pricing and the way they describe it narrows it down somewhat.