we build a product called Veracity focused on citation and claim checking. legal is a major focus for us, also scientific and technical publishing, where we detect hallucinated citations in submitted works and all that jazz
planning to release something soon focused on the LLM harness layer
we've been working on the problem for years. gradually getting traction
all completely closed source atm, though we often revisit that discussion
i would contest a sibling comment - the solution is not RAG, at least not entirely. 1) because it doesn't necessarily involve 'generation' (e.g our grounding model doesn't depend on gen ai) and 2) RAG is just enriching the context. even if you can get the retrieval right (not a given) it doesn't guarentee that the LLM will faithfully or correctly use that context, or that all its claims are supported by sources. RAG is not the same as traceability and verification
I'll add they also seem to make 'bad' judgement calls around which layer to implement something, often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.
They also duplicate code lots rather than finding and reusing existing.
Both of these patterns lead to degradation and time wasting in complex code bases. Part of the behavioural adaptation I've made in using these tools is heading these off as much as possible at prompt-time
My prompts are often verbose to several paragraphs
This is true, but we also don’t know the long term architectural tradeoffs that are right for an LLM-maintained codebase.
For human-maintained code you manage duplication and locality of changes in the codebase because you want concepts to fit inside a human’s working memory. Keeping method bodies short is so they fit in one code window on the screen.
LLMs’ working memory works in very different ways; they see codebases not as windows of code but as blips of grep results. It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
> It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
One thing they are definitely not doing is making decisions based on how easily and quickly they will be able to maintain those decisions in the future, or considering that duplicating code rather than creating abstractions will be easier for them to keep track of because of their "very different working memories."
They're just doing a bad job. It's not some sophisticated work avoidance/efficiency in defiance of feeble human abilities to maintain what they have written. 1) LLMs are not lazy, and 2) if writing maintainable code is part of the remit, they're consistently failing. You can't convince me simultaneously that LLMs can handle everything, and also that they can't figure out how to factor their code in a way that would be a pleasure to review.
It simply doesn't make sense. If they're so good, they can make it easy to read and well-factored. If they can't make it easy to read and well-factored, they ain't so good. Saying that they're so good that the code has to be this bad is a very weird rationalization. One that apparently involves speculating about their internal states in a strange way that sounds a lot like the projection of fantasies.
I’m just saying we don’t yet know what works best long term for LLM managed code. Not suggesting that what LLMs produce today is optimized for that, and ‘human-readable’ is for sure a good default place to start.
It also differes between human-maintained codebases, too.
For some greenfield project I'm developing, 99% of the time, fixing architecture and going as far upstream as possible is the right choice.
For the codebase I'm working on professionally, doing the hacky thing is the right choice most of the time, because there's enough people changing it frequently enough, with little coordination, that architecture is fluid, and hacks only survive if they work well enough (otherwise someone will come around and fix it).
That was true pre-LLM, too, and is simply a function of scale. And ironically, it's not laziness but pragmatism - at this scale, you just can't assume code you worked on last month will look today like anything you remember. Code review process + extensive linting and testing and AI-assisted analysis filters out most of the junior-style mistakes and flag ugly hacks, so you get a moment to pause and double-check, but if it's good enough, you merge it in and go do another useful thing.
This is how you end up with a big ball of mud [1]. If the argument is that LLMs are just as capable as humans at building big balls of mud, then I'm not sure it's much of an endorsement of their use.
I've thought about that and I agree. One thing I'm wondering though is, are we really ready to go all-in on making a codebase that is effectively unmaintainable by humans?
They're very tuned towards 'solve the problem now with the resources you have'. They can implement a hacky downstream fix and then there is a fix. A proper upstream fix takes time and probably requires access that they don't have, and it might not be accepted.
This has become such a problem in scholarly publishing that we have a business that provides citation checking https://groundedai.company/ that we've been buidling for a couple of years now
So far we basically just provide a very rule-based approach and try not use LLMs as much as possible. So we extract and parse the citations using various ML and rule-based approaches, and carry out a bunch of predetermined queries and do various fuzzy matching approaches on the metadata components, and have a bunch of rules around risk levels of things we should have found/matched based on what type of source it is, which venue we should have found it in, etc.
So there are absolutely a bunch of tasks that could be evaled/benchmarked, but "hallucination rate" isn't particularly applicable/interesting as a metric of how good the tool is
that said, we do use various LLMs (mostly local, fine-tuned, small, for things like NER/parsing/metadata comparison, etc.). and they can and do hallucinate, but we have very hard constraints on the validation, so any extraction results that don't match 1:1 back to the input text are discarded for example. so again, rather than hallucination risk we prefer hard constraints
We collaborated with Nature here to study the extent of fake/frankenstein citations in scholarly literature (from top 5 publishers - Springer, Elsevier, Wiley, Sage, Taylor & Francis)
We're estimating hundreds of thousands of papers affected in 2025 with hallucinated citation issues
As part of the work we analysed 20k papers generated with ChatGPT API to figure out which citation errors are characteristic of gen AI use and use that classify the errors we saw in the wild.
The world's gone mad, publishing is in a nuts state, the training data is poisoned!
yeah this makes sense. we run a citation verification service and provide publishers with data of hey this citation could be fake etc. but we don't currently capture any "action" or "measured result" so i guess that's what we need to expand to next
I work on Veracity https://groundedai.company/veracity/ which does citation checking for academic publishers. I see stuff like this all the time in paper submissions. Publishers are inundated
That's basically what we're doing with app.studyrecon.ai.
What we've found is that vector similarity is often not the final solution. It is still only a crude proxy for the true goal of 'informativeness' or 'usefulness' with relation to the user goal/query. Works okay, but we're definitely seeing a need for more rigorous LLM-postprocessing to enrich the results set.
we build a product called Veracity focused on citation and claim checking. legal is a major focus for us, also scientific and technical publishing, where we detect hallucinated citations in submitted works and all that jazz
planning to release something soon focused on the LLM harness layer
we've been working on the problem for years. gradually getting traction
all completely closed source atm, though we often revisit that discussion
i would contest a sibling comment - the solution is not RAG, at least not entirely. 1) because it doesn't necessarily involve 'generation' (e.g our grounding model doesn't depend on gen ai) and 2) RAG is just enriching the context. even if you can get the retrieval right (not a given) it doesn't guarentee that the LLM will faithfully or correctly use that context, or that all its claims are supported by sources. RAG is not the same as traceability and verification
reply