I am starting to wonder if I am doing something wrong: I ignore evals and instead I just try new models or new harnesses (or tweak my own harnesses) by solving problems I want to solve in any case; I just use new tools and form my own subjective opinions of them.
When I say evals I mean the evals you write that verify that your use cases are upheld. Think of it like a regression test for different behaviours/user stories.
The idea would be that if you already know what you want from an autonomous system, you don't need to verify manually every time and instead just run these tests to see if there's any regression of any kind. Generally I recommend structure output and evals that are just a plain assertion, if possible. Cheaper, faster, deterministic assertions.
