But does it always 100% of the time sticks to the schema? We have a prompt that is explicitly instructed to return a single html tag with the response inside it and it sometimes hallucinates
I think the truth is sort of halfway between yall.
Hallucinations in tool call results _do still exist_, but basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
That all goes out the window when a string field has a “hidden” schema in that only particular strings are valid, but that restriction isn’t in the JSON schema. I have had failures when I want a field to be specifically formatted Markdown or something.
We’ll probably see something that handles this better in the future like jsonnet or Cue or dhall that has some execution capabilities so you can write a custom validator beyond what JSON schema supports.
> basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
Basically no one is doing that since 2024, have you been living under the rock? Read about constrained decoding.
> I have had failures when I want a field to be specifically formatted Markdown or something.
And Jev has an advantage here because it can’t generate anything?
> and re-prompt the LLM until it emits something with a valid schema.
Well, that's essential what happens, but on the "backend" side still, so it's significantly more efficient. Also, model providers can play with e.g. how likely are they to accept a valid/invalid character, so for example the first token may only be [, { or ".
As for what's not in the schema, that's an orthogonal question.
yes, it does, with appropriate tuning/testing of the llm's parameters (temperature, top p, top k, using the right model, and the right prompt). You have to give it a rigid and very specific prompt to only answer in the JSON form. Also test it with LLMs that will handle being given a very low temperature to be very 'literal'. You're not asking for creative writing.
I should also add that the source comes from one of about 400 possible places and in a variety of messed up formats, it's the raw feed from a news scraper...
Cannot fail? It's so smart it literally can't ever parse incoming content incorrectly and provide the wrong answer to a question like "Is at least 25% of the text content in this document written in German? (yes/no)"
You are talking about the content of the response, while krial is talking about deviations in the structure of the response. The point is the number of possible states an output can take, so the next step can ingest it and it can be part of a robust pipeline.
I feel like it would be unfair to make them include the SVG decoder, the artical uses TypeScript and I believe that is a feature most runtimes (browsers) provide. However they do have to include the SVG files themselves. The Canadian flag from Wikimedia comes in at 699 bytes and France at 262 bytes. So I'm fairly confident that even with compression they'll be over 5kB
Right but that is part of this long rolling episode that started with the WP Engine thing, which is really when he began to act in an unhinged way.
That is really when the sudden shift in his public manner happened. It has been going on that long. He should have stepped back as either Automattic CEO or Wordpress BFDL at least a year ago IMO; the ACF/SCF debacle settled it.
WP and the community around it would have been better for it if he had.
WP was the community's, apparently, not Matt's. Unless it suited him, in which case it was his.
WP was open source, and WP Engine followed the license. What crookery?
Oh, they misrepresented themselves as being WP affiliated, like the claimed "independence" of WP.com (Matt's for profit entity, not the open source app)? Why was one okay not the other?
Let's not discuss that Matt, and Automattic, while pushing press releases saying that WP was forever an unencumbered open source app for the community, that very same day issued WP.com an exclusive, free, unlimited, irrevocable commercial license... didn't tell anyone about that, it came out in the lawsuit.
Or that the WP community board was effectively Matt and a friend, and a largely absent third friend.
Crooks, stealing? WP Engine contributed to the OSS app with thousands of developer hours a year - not as many as Matt "felt" they should. And only them, not other WP hosters.
Crooks, stealing? WP Engine donated six digits a year to sponsor and support WP community events, including the one where Matt went on stage and spent 45 minutes slandering them - after disinviting them, and refusing to return their sponsorship or donation. He made WP Engine effectively pay to give him a stage to talk shit about them.
Crooks, stealing? Like Matt taking software packages that WP Engine wrote, doing a search/replace on Copyright, license, authorship, and replacing their entries (with all their accumulated community reviews, ratings, followership and traction) with his wholesale-copied variants?
Matt was unhinged. I think "right to be" is "debatable, at best".
reply