I've also seen similar recently at my place of work with LLM linting. Lots of hard to spot bugs in old very critical code have been caught. I remember seeing similar kinds of impacts back when linting or sanitizers were introduced. They're good for making it easy to encode more complex rules/areas-of-focus etc. that may have not been meaningfully possible with the more traditional tooling. Commercial LLMs in their current incarnations make for very expensive, but very effective intelligent linters.
On the other hand, my experience so far has been that they are terrible code-reviewers, to the point almost everyone ignores any kind of generic LLM review entirely (correctly - most of it is noise).
I feel the industry obsession with full-automation causes them to go down the wrong rabbit-holes on full-automation vs augmentation. In my view would be significantly more effective if the LLM review process in forge-tooling was designed with a human reviewer in the loop. The goal imo should be drastically improving the review efficacy & throughput of the human element, who needs to be acquiring confidence in & socializing the change in any sane org anyway. For example iteratively & interactively rubber-ducking out analysis + remediation, or interactively providing focusing guidance on the patch, instead of as agentic batch or harness processes as is cool nowadays. Sometimes a custom REPL is just better for an expert system. These things matter more as more and more code is LLM generated.
On the other hand, my experience so far has been that they are terrible code-reviewers, to the point almost everyone ignores any kind of generic LLM review entirely (correctly - most of it is noise).
I feel the industry obsession with full-automation causes them to go down the wrong rabbit-holes on full-automation vs augmentation. In my view would be significantly more effective if the LLM review process in forge-tooling was designed with a human reviewer in the loop. The goal imo should be drastically improving the review efficacy & throughput of the human element, who needs to be acquiring confidence in & socializing the change in any sane org anyway. For example iteratively & interactively rubber-ducking out analysis + remediation, or interactively providing focusing guidance on the patch, instead of as agentic batch or harness processes as is cool nowadays. Sometimes a custom REPL is just better for an expert system. These things matter more as more and more code is LLM generated.