If you don’t trust the code to write a decent comment, why trust it write good code?
Of course, ensuring compilation or other checks can verify some code, which it can’t do for comments. But comments still serve the same purpose as human comments.
If I understand correctly, it’s not that the LLM can’t write a good comment, it’s that you want to be able to interpret and understand the generated code without comments - and in that process end up writing comments yourself.
I don’t get that argument. Most of the time by the end of the session the comments from the agent encode tricky details that I told the agent to write down so it stops making “simplifying” assumptions. Thus comments at the end of a couple days of agent-only coding, when I start to actually read and edit the prose, contain the details which aren’t possible to know from reading the local code. It may help that my last couple sessions before I start reading the code myself are variations on telling the agent to self-review and improve the comments in specific ways, so the comments left are only those the agent thought remain meaningful at clarifying unexpected interactions between the local code and other code that needs to be referenced to understand it.
The actual code output has improved a lot over the past year. I’ve found it matches existing patterns better, and the code is succinct so I can easily tweak it if I don’t like the way the agent wrote it.
The problem with comments is that LLMs tend to copy their verbose chat output format and insert session/prompt specific details. It makes me think that LLMs aren’t constrained in their comment output the same way they are with their code output
Add "Don't add any code comments anywhere" to your system prompt.
If the model doesn't follow this, you want to start using a better model ASAP, because SOTA models for the last year or so, been able to following this without an issue.
Alright, what you'd put instead of "Don't add any code comments anywhere"?
I agree with the general guidance, but it's a general one and not applicable for everything. Some things cannot be expressed in a "do" way rather than "don't".
I don't think changing "don't" to "do not" is what the paper authors had in mind
I use things like "Your code should be self-documenting, so as to require as few comments as possible. Add comments to explain "why" or give important context not apparent from the code itself, but keep them to necessary comments only"
But that's a much laxer rule. I don't think you can truly express "no comments, ever" without a "don't" rule.
Your proposal is still a "Prohibition-type constraint" that the paper you linked earlier say "isn't good".
Some of these constraints we want simply aren't possible without adding "do not" somewhere in the line, even if you prefix/suffix it with other stuff, as you noticed yourself :)