HN seems to have had a stream of agent harness benchmarks floating past. And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test? Because right now nobody seems to bother testing mine! (https://juggler.studio)
I know Juggler's very new, but there's so much churn going on in this area that it's hard to know where I should be pushing it. It's hard to guess whether juggler's strengths would played well with a particular test like this, or made it look bad, all feedback about the kind of parameters people are interested in is useful to know when I'm deciding what to optimise.
> And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
> For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session...
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
It's not a VM; it's containerized with a FUSE-based custom overlay, and the agent never touches a real mount (with two exceptions: passthrough read-only mounts for things like /bin, and ephemeral tmpfs mount at /tmp).
What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
There are a comical amount of harness design features in the early 1.x release. A full LSP integration, find symbols, batch edits, etc. Terminal bench isn't everything but I spent a couple of weeks testing different combinations to no statistical effect greater than a bare loop. It was like running uphill against what the underlying LLM wanted to do.
> What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
Same. But there's a trick to it, in a way: vibe code it first, test it out yourself, and then decide whether it should be reverted or kept and deslopped/properly shipped.
> I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
For me personally it's a game-changing workflow, because it means I don't have to sandbox (e.g. with a Docker) my agents while still being able to run full YOLO. Essentially have my cake and eat it too. If the agent does something stupid I can just rewind. And I don't need to spend as much effort continuously reviewing what it does, because I can just do it right at the end on the `/apply` screen.
(But this only applies if the filesystem is actually fully virtualized, as opposed to the git-based checkpointing that so many harnesses have, which to me is pretty much useless because it doesn't actually give me any guarantees. That is: the agent running `rm -Rf /*` should be a non-event that doesn't do anything to my filesystem, and can only nuke the agent's view of the filesystem.)
Of course, this could just be me, and other people may not really care. But then, is your objective to make the best possible harness for you and people like you, or for everyone? In my case it's the former.
> And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
It seems we need a place where harness creators can pay for that work. There are more people talking about their own harness in this post than talking about the post.
Because juggler has a nested-thread architecture, I had a great ah-ha! moment when I realised that with sub-threads, compaction becomes almost free: you just take the list of items in the conversation, move them all into a new sub-thread, then allow that sub-thread to summarise itself (this summary is what the parent thread sees). Hey-presto, your parent thread is now compacted, and the sub-thread contains all the old messages, so you can browse them if you need to, or just undo the whole operation if you change you mind. This struck me as super-elegant, and I was chuffed that it works.
However.. What I find myself actually doing now is just using the "new conversation" tool to hand over to a new conversation - I just tell the LLM "create a new conversation, and tell it everything it needs to continue this task". It gets you to the same place, but generally I'm not interested in the history, and having a fresh conversation with a new name usually feels neater.
Oh, I'm fully expecting people to ask for all kinds of weird custom providers, and they're easy to add. For launch I just focused on getting the provider API solid, and implementing the mainstream ones. Feel free to open an issue on GH if you want to request something that it doesn't do
ACP hadn't crossed my radar before some people mentioned it here, but it's an easy add to make juggler an ACP client - I'll do that in the next couple of days
FYI I've got a plan for this now, I'll make it fallback to using node/bun as its javascript engine if GTK isn't available. So watch out for the next release if this is a showstopper for you..
Where I've found editing to be the most useful is e.g. when I've had a long task that got a bit rambling and digressed, and then I go away and need to resume it later. Leaving it means the provider will have lost the cache for it, but rather than spending a lot of tokens to /compact or just to continue the entire thread, you can go through and delete all the useless bits, maybe just leaving enough content in there for the LLM to figure out how to continue, and then get back on track quite cheaply
Originally I allowed the LLM to decide whether a subthread inherits the parent context, but had to back out of that one. I found quite expensively that if the parent context includes an instruction to do a task in a subthread, it could fly off into an infinite recursion of subthreads all burning tokens to delegate the task to a subthread!
So I've just made it so they don't inherit! If we find a situation where inheriting is useful and safe, it'd be easy to add again.
I know Juggler's very new, but there's so much churn going on in this area that it's hard to know where I should be pushing it. It's hard to guess whether juggler's strengths would played well with a particular test like this, or made it look bad, all feedback about the kind of parameters people are interested in is useful to know when I'm deciding what to optimise.
reply