If you work with multiple model providers in one harness, it has probably become clear to you that the benchmarks we commonly look to do not necessarily reflect real-world experience. This article is about running evals yourself, based on the way you work, to determine which models give you the best bang for buck.
The method comes from Mike Taylor, head of evals at Every. He set it out in his Classroom session at Thesis, Every’s conference, after the keynote. His two examples follow, each marked as his. How I would use the method is my own judgement.
Start with the work. A unit of work you can recognise: a deck, a brief, a review. Each has a standard an accepted result must meet. My test of a model is whether its output clears that standard on that task.
Start with a correction you already made
Take the last output you fixed. That correction already holds your standard. Write it as a yes-or-no question and it becomes a check.
Taylor’s example is a deck. He asked an AI for a deck to teach a team Compound Engineering, Every’s method for coding with AI agents. It reused an illustration from his brand guide. His correction: always make a custom SVG, a scalable vector graphic, because the format keeps diagrams crisp. As a check: did the deck draw its own diagram? Any judge can answer that of any future deck.
Collect the corrections from a task you repeat and you hold a personal benchmark. Run each new model against the same checks, as I do when I compare routes in Model Selection, and the result answers my question directly. Taylor ran his deck task across models. GPT 6.1 Sol failed, reusing the brand-guide image. Grok 4.6, which he had expected little from, passed. It drew its own. A leaderboard place would have told him neither result in the terms he cared about.
Talk the standard aloud
You do not need to write a rubric. You need to state what you liked and what you did not, while the failure is still in front of you.
Taylor puts the division of labour plainly: “AI is really good at taking that unstructured transcript and turning it into evals.” Dictate the correction. I work the same way. Voice captures the standard at the moment it is clearest, before I tidy it into something vaguer. His team found voice mode works best for this. So have I.
A check must survive the next task
A check can teach the wrong lesson. Read each one as you would read a draft, and cut any that only fits the example that produced it.
Taylor told a model that its ASCII-art cat “isn’t recognizable as a cat.” The model wrote a fair check. It also gave itself a rule: “the cat sits facing front.” Asked later for a cat curled up asleep, it drew a front-facing face anyway. The output passed the check. It still failed the request.
He calls that overfitting “the number one problem in AI right now.” My rule follows from it: a check must hold for a reasonable variation of the task, or it goes. Keep the set small, keep it tied to work you do every week, and keep retesting. As Taylor puts it, “Reality is the ultimate eval.”
Steal it
Paste this into any chat AI the next time you correct an output for a task you repeat:
I’m going to paste an AI output for a task I do often: [task, e.g. “weekly client update”]. Then I’ll talk through what I like and don’t like about it. Turn each of my comments into a yes/no check that a judge could apply to future outputs of this same task. Keep every check specific and testable. Flag any check that would be wrong for a reasonable variation of the task (overfitting). Save the checks as a list titled “[task] checks.” Next time I paste an output for this task, from a new model or a changed prompt, grade it against every check and tell me which ones failed.
You will need any chat AI. Voice dictation helps.
Go deeper in Taylor’s own guide to building a personal AI benchmark, and in his full session, “From Vibe Checks to Evals”, in the Thesis recordings (Classroom, 0:33:43).
Source: Mike Taylor, head of evals at Every, Classroom session at Thesis, following the keynote. Method, examples and prompt are Taylor’s; wording here is mine.