1. 15 image transition effects. Each agent gets the same set of before/after images, and 15 effects to implement. E.g. first image dissolves in fumes/smoke and reassembles itself. Or, "take 9 images, and reimplement the classical game of micromachines with it". All problems are hard. Each problem is run in 3 independent tiers: with a vibe coded prompt (just a few sentences), with a vibe coded prompt plus a set of evals we will use to judge and finally, with a long specification. SOTA models are now starting to perform better on evals only, vs. evals + specs. Low end models perform well on specs.
For months Anthropic would wipe the floor with OpenAI and all the competition - as their models seem to be best at self-evaluation of their outputs. They would make screenshots, and be critical - and use the available budgets to perfect the solution. Now Carbon and 6.2 astra sol are within spitting range of Fable/Opus. I haven't finished evaluating the full 6.2 suite yet.
2. a virtual world - a military range with artificial obstacles. Independent game engine provides the agents an API that (a) informs them what the players they control see, and (b) allows the players in the team controlled by the coding agent to develop communication patterns, constrained (6 words per 30s). Agents have 15 iterations to build a algorithm that controls those squads - after which they are stacked against each other in various configurations (3v3 rewarding advanced tactics and teamplay, 1v1 rewarding just good mechanics). Its a very expensive and time consuming benchmark - a single session where an agent starts with the empty world and set of basic laws of physics - and an assurance they will be pitted against tactically competent teams - takes a few days of code generation and simulations. It usually costs around 2 full $200 subscriptions to run this on SOTA models, so I am currently only comparing the best models - Fable, Astra, now Carbon, and Deepseek.
3. another one covers a business problem from my domain - which has a known good, but hard to figure out solution with cost-to-solve being benchmarked.
The first suite is evaluated in a blind probe: the agents write the algos, and then I am given them side by side, and decide which implementation is better/more aesthetically pleasing/closer to the brief. Its arbitrary. I have by now around 70 LLMs in the suite, and its a bit tedious to do the evals, but its pretty objective.
The 2nd suite is evaluated by the algos written by the agents stacked each other in an arena - and playing around 60,000 games in various configurations. Objective. The agents are also evaluated qualitatively, for my own entertainment: which tactics did they evolve? Do their players utilize the comms channels? Do they watch each other corners? Do they stage ambushes? Do they develop advanced tactics? Its evaluated against a field manual turned into a skill, and its just informative, does not affect the ranking. The fps suite is ran in a paranoid mode: the game world engine is on a different machine, the game world and other agent's code is kept physically inaccessible to the model being benchmarked, and I planted decoys on the API surface to see which agents would try to see if they can cheat the game by breaking into server side state machine. One of the models tried that so far.
The 3rd element of the suite has a set of objective tests, which focus on time to reach a solution. Its saturated by now and will likely retire it. The image transitions thing used to be easy to judge, but now with most models catching up its getting saturated too.
If you're not willing to stand behind a view, and take all the well-deserved flak for holding it, you should shut up. Brendan Eich said "I hate gay people a couple thousand dollars much". That's a fucked up belief to hold, and if he didn't want it held against him, it was completely free to not donate thousands of dollars.
> Calling a named individual an "absolute scumbag" in this context is fine grounds as any for a lawsuit.
Based on your misunderstandings of US defamation laws, and the fact that you jumped to stabbing over shooting, it sounds like you've been steeping too much in eurotrash defamation rules. Here in America, truth and opinion are both perfect defenses against defamation lawsuits. My comment consisted only of true facts (the man donated to support prop 8), and the opinions that follow from those facts (the man is a scumbag). You can only sue someone for lying about you, and for public figures like a CEO, you need to prove they were lying on purpose.
so the holy grail of software engineering is a tool that allows us to commit minimally sufficient design documents as source code.
i'd be enthused to work this way if changes to `src/*.md` files could be "hot-reloaded" on save, like an iterative LLM watch compiler, and skipping the chat prompts altogether.
We cant even verify if these fields are real though. It is an ordered mythology for making predictions that work. If I have a machine that can reproduce the footprints of an animal no one has ever seen, I have not discovered that animal. (This is personally a more powerful analogy than saying "the map is not the territory" which gets at the same point.)
Sure. Just making sure people understand that the "scientific definition of real" is based on a metaphysical assumption that real = works. It is very hard to remember this since philosophy of science is not taught in school.
Fallibilism seems to smuggle a claim of certainty about how we can't be certain about anything, which leads to a performative contradiction when brought up in an argument where people are trying to arrive at certainty.
If you don't like such skepticism, then it should be okay to believe in fields. Where do you draw a line, how much skepticism is enough? Does radio exist? You can't see it, just like other fields.
I was only addressing your argument. If fallibilism leads to a contradiction then it doesn't work to undermine all agnostic arguments against the actual reality of fields.
My warning is that science becomes something like a 'scientism' religion when "real" is used as shorthand for "effective abstraction". Science has philosophical boundaries that isn't part of the science curriculum.
It was your argument that you verified that we can't verify. But if we can't verify, then how could you verify? Fallibilism doesn't lead to a contradiction, it only rejects infallibility. If you think agnosticism helps you prove your infallibility, prove it. Also radio demonstrates that your approach is unworkable.
I’ll never understand how GEB was using math, art, and music to explain consciousness (and Hofstadter himself still thinks no one understood it), but Nagel and Newman did a great job explaining why logic as a mechanical thing has only a tenuous relationship to concepts we understand, and that helped me crack at least a little bit of the mystery I was after when giving up on GEB.
I have not read GEB but I thought his second book, I am a Strange Loop, did a pretty good job of connecting the idea of self referential loops (like in godels proof) to consciousness and art and such.
Thanks for the suggestions. Just checked out some of Le Ton Beau de Marot, will most likely read the rest.
Actually, I was recently wondering whether poems can be interpreted as "efficient" computer programs. Words in a sentence/essay just connect different objects in space and time (syntactically and semantically) to form some concept, kind of like a program. A poem just uses higher order abstractions (imagery) to express the same concepts in fewer words.
> I’ll never understand how GEB was using math, art, and music to explain consciousness
As Flannery O'Connor wrote, "The result of the proper study of a novel should be contemplation of the mystery embodied in it, but this is a contemplation of the mystery in the whole work and not or some proposition or paraphrase. It is not the tracking down of an expressible moral or a statement about life." We don't read literature with the hopes of a book laying out a precise thesis and incontrovertibly demonstrating it.
If you come into GEB expecting a scientific explanation of consciousness (like I did, when I first read it) you walk away confused and maybe disappointed. Hofstadter observed something transcendentally beautiful about self-reference and had a spiritual or religious revelation that, for him, related it to consciousness, and he attempted to convey that beauty and spiritual revelation in - appropriately self-referentially - a book that embodied it. You're meant to appreciate it in your heart and soul, not (just) in your mind. It's literature, not science.
reply