I would like to amend this comment and say I was far too reductive and quick to judgement, I didn't appreciate the other improvements they have made, and after a few days of use I like it a lot.
I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
Hi, most memory tools get evaluated on benchmarks like LoCoMo, LongMemEval and BEAM, plus how many tokens they use
We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md
What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.
Curious what you're using today, and what result would convince you?
I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
I have a project I am working on where I wanted to have nice docs, and so I have some slop docs sitting there as placeholders until I get around to writing them, awful claudish stuff
I've actually spent some effort trying to make them halfway decent placeholders by putting together a skill based on diataxis framework (https://diataxis.fr/) that at least got everything into the right location
just took a pass at rewriting them w/ argon and am very pleased, big improvement
Yes I have just started dabbling in some harness evals where the task is translating the RealWorld “mother of all demos” app, maybe one of the most translated apps of all time, and they cannot reliably 100% it
I don’t write the same program over and over again so I am always confused by how exactly this industrialization metaphor is supposed to map on to software development
Think of it this way, the Persian rug isn't the same pattern every time either. The base job is the same, but the implementation requires planning and the like. A rug making machine is good at creating the same thing over and over again, but not very good in being flexible.
Software... a lot of software is basically the same thing (database, api, frontend), the only difference is what data and business rules. But the act of writing code isn't very different. It's the what code to write where software engineers come in.
A better analogy is that software dev is more like making the rug making machine. If you were doing that, it would be useful to be intimately familiar with precisely how rugs are made, and you might want to make a bit of one by hand in order to get that experience. Then when you design the machine, you use gears, 4 bar linkages, and other mechanisms that you are already familiar with because you carefully studied those concepts earlier, likely with some hands-on exercise. Then you draw a design, simulate it, and send it to a machine shop and can possibly never touch the machine itself (although I think in practice these kinds of bespoke first-gen machines need debugging after manufacturing).
I.e. in order to get to the point where you didn't have to manually touch this thing, you likely did have to learn deeply about its underlying concepts by manually doing things.
Operating machines doesn't always have the same phenomenon. E.g. experience with a hand-powered drill is unnecessary to use an electric drill.
I didn't mean it literally in the sense of "the machine can produce 1000 copies of the same program!" (which indeed would not need an LLM to accomplish...). I meant industrial-era automation in the wider sense of "a machine and a human operator can do in one week the work previously done by 10 people in two months".
To put it another way: a generalisation of OP's question is "is it worth spending a lot of effort learning to do X manually, when a machine can do a lower-quality version of X massively faster and cheaper than I can ever do?"
In the article, X = coding. I wanted to check if the arguments given for "yes, and" hold for different values of X and the first that came to my mind was rugs; probably unfortunate that I know nothing of that topic...
I don't think that the answers for one value of X always apply completely to other values of X (i.e. whatever happened to hand-woven rugs might well be very different than what happens with coding) but I found it an interesting thought experiment, because I can fully see a master artisan of any type of X giving a very close version of the article's arguments. Maybe finding when they were eventually right and when they were not for past scenarios can teach me something now. Maybe...
would be fun to revisit and see what v2 would look like today, I have a feeling I wouldn't be manually recording voice lines this time around ha
https://github.com/pathouse/kitchen_assistant
reply