great writer, starts forgetting instructions around 80k context in my experience

markdown specs are certainly the funniest way to claim to distribute software

I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself

so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive

who out there is running evals of giving agents these kinds of tools and how are you doing it?

Hi, most memory tools get evaluated on benchmarks like LoCoMo, LongMemEval and BEAM, plus how many tokens they use

We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md

What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.

Curious what you're using today, and what result would convince you?

I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general

I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another

not super pleased with it

the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for

one thing you could do is run it against FrontierHarness eval and see how it stacks up?

Yeah learn how to do even the basics of science, please. Control groups!! Confounders!!

Letting another LLM decide quality probably isn't the only meterstick you want, either.

I have a project I am working on where I wanted to have nice docs, and so I have some slop docs sitting there as placeholders until I get around to writing them, awful claudish stuff

I've actually spent some effort trying to make them halfway decent placeholders by putting together a skill based on diataxis framework (https://diataxis.fr/) that at least got everything into the right location

just took a pass at rewriting them w/ argon and am very pleased, big improvement

I’m guessing they’re referring to jamal khashoggi

Yes I have just started dabbling in some harness evals where the task is translating the RealWorld “mother of all demos” app, maybe one of the most translated apps of all time, and they cannot reliably 100% it

I don’t write the same program over and over again so I am always confused by how exactly this industrialization metaphor is supposed to map on to software development

Think of it this way, the Persian rug isn't the same pattern every time either. The base job is the same, but the implementation requires planning and the like. A rug making machine is good at creating the same thing over and over again, but not very good in being flexible.

Software... a lot of software is basically the same thing (database, api, frontend), the only difference is what data and business rules. But the act of writing code isn't very different. It's the what code to write where software engineers come in.

A better analogy is that software dev is more like making the rug making machine. If you were doing that, it would be useful to be intimately familiar with precisely how rugs are made, and you might want to make a bit of one by hand in order to get that experience. Then when you design the machine, you use gears, 4 bar linkages, and other mechanisms that you are already familiar with because you carefully studied those concepts earlier, likely with some hands-on exercise. Then you draw a design, simulate it, and send it to a machine shop and can possibly never touch the machine itself (although I think in practice these kinds of bespoke first-gen machines need debugging after manufacturing).

I.e. in order to get to the point where you didn't have to manually touch this thing, you likely did have to learn deeply about its underlying concepts by manually doing things.

Operating machines doesn't always have the same phenomenon. E.g. experience with a hand-powered drill is unnecessary to use an electric drill.

Most of the software which was the same thing has been abstracted already in CRM, WordPress, content builders, website generators...

What's left is a lot of very different software which have very few things in common.

> the only difference is what data and business rules

The "only" difference? Wow.

Database, API and Frontend are tools and generic raw material that need to be whittled down into the final shape.

for all the CRUD apps out there, which are not the full set of software programs

I didn't mean it literally in the sense of "the machine can produce 1000 copies of the same program!" (which indeed would not need an LLM to accomplish...). I meant industrial-era automation in the wider sense of "a machine and a human operator can do in one week the work previously done by 10 people in two months".

To put it another way: a generalisation of OP's question is "is it worth spending a lot of effort learning to do X manually, when a machine can do a lower-quality version of X massively faster and cheaper than I can ever do?" In the article, X = coding. I wanted to check if the arguments given for "yes, and" hold for different values of X and the first that came to my mind was rugs; probably unfortunate that I know nothing of that topic...

I don't think that the answers for one value of X always apply completely to other values of X (i.e. whatever happened to hand-woven rugs might well be very different than what happens with coding) but I found it an interesting thought experiment, because I can fully see a master artisan of any type of X giving a very close version of the article's arguments. Maybe finding when they were eventually right and when they were not for past scenarios can teach me something now. Maybe...

I appreciated this post, had never heard of codemode and I think the name is bad but it makes perfect sense

I have wanted something like this ever since I hooked up my first database MCP

Originally I had thought to set up a python runtime with like a standard data science toolkit and the db mcps available as functions, and maybe I still will but JS has been working fine for now

I’ve been slow blinking at every cat I meet for years, I swear by it haha

My mum calls them cat kisses, and we do it with other cats too, lol!