You can turn any sufficiently smart LLM into yes/no decision model or equivalent. I already have an existing workflow with a two paragraph detailed prompt, that sends pages of stuff to an LLM and asks it to return only 7 JSON objects. Several of those objects are binary "yes or no" choices of like, whether the content contains certain things.
You can even do it with small not particularly hard to host local LLMs like a variant of Qwen 3.6 35B A3B or 3.8 27B.
But does it always 100% of the time sticks to the schema? We have a prompt that is explicitly instructed to return a single html tag with the response inside it and it sometimes hallucinates
yes, it does, with appropriate tuning/testing of the llm's parameters (temperature, top p, top k, using the right model, and the right prompt). You have to give it a rigid and very specific prompt to only answer in the JSON form. Also test it with LLMs that will handle being given a very low temperature to be very 'literal'. You're not asking for creative writing.
I should also add that the source comes from one of about 400 possible places and in a variety of messed up formats, it's the raw feed from a news scraper...
Cannot fail? It's so smart it literally can't ever parse incoming content incorrectly and provide the wrong answer to a question like "Is at least 25% of the text content in this document written in German? (yes/no)"
You are talking about the content of the response, while krial is talking about deviations in the structure of the response. The point is the number of possible states an output can take, so the next step can ingest it and it can be part of a robust pipeline.
I think the truth is sort of halfway between yall.
Hallucinations in tool call results _do still exist_, but basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
That all goes out the window when a string field has a “hidden” schema in that only particular strings are valid, but that restriction isn’t in the JSON schema. I have had failures when I want a field to be specifically formatted Markdown or something.
We’ll probably see something that handles this better in the future like jsonnet or Cue or dhall that has some execution capabilities so you can write a custom validator beyond what JSON schema supports.
> and re-prompt the LLM until it emits something with a valid schema.
Well, that's essential what happens, but on the "backend" side still, so it's significantly more efficient. Also, model providers can play with e.g. how likely are they to accept a valid/invalid character, so for example the first token may only be [, { or ".
As for what's not in the schema, that's an orthogonal question.
> basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
Basically no one is doing that since 2024, have you been living under the rock? Read about constrained decoding.
> I have had failures when I want a field to be specifically formatted Markdown or something.
And Jev has an advantage here because it can’t generate anything?
walrus01 · · focus · HN ↗
You can even do it with small not particularly hard to host local LLMs like a variant of Qwen 3.6 35B A3B or 3.8 27B.
opiotrek · · focus · HN ↗
walrus01 · · focus · HN ↗
I should also add that the source comes from one of about 400 possible places and in a variety of messed up formats, it's the raw feed from a news scraper...
erkl · · focus · HN ↗
walrus01 · · focus · HN ↗
krial · · focus · HN ↗
walrus01 · · focus · HN ↗
david_draco · · focus · HN ↗
gf000 · · focus · HN ↗
<a href="https://developers.openai.com/api/docs/guides/structured-outputs?api-mode=responses" rel="nofollow">https://developers.openai.com/api/docs/guides/structured-out...
[deleted] · · focus · HN ↗
[deleted]
everforward · · focus · HN ↗
Hallucinations in tool call results _do still exist_, but basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
That all goes out the window when a string field has a “hidden” schema in that only particular strings are valid, but that restriction isn’t in the JSON schema. I have had failures when I want a field to be specifically formatted Markdown or something.
We’ll probably see something that handles this better in the future like jsonnet or Cue or dhall that has some execution capabilities so you can write a custom validator beyond what JSON schema supports.
gf000 · · focus · HN ↗
Well, that's essential what happens, but on the "backend" side still, so it's significantly more efficient. Also, model providers can play with e.g. how likely are they to accept a valid/invalid character, so for example the first token may only be [, { or ".
As for what's not in the schema, that's an orthogonal question.
hbrn · · focus · HN ↗
Basically no one is doing that since 2024, have you been living under the rock? Read about constrained decoding.
> I have had failures when I want a field to be specifically formatted Markdown or something.
And Jev has an advantage here because it can’t generate anything?
StevenWaterman · · focus · HN ↗