It’s easy to lose the plot in the overwhelming proliferation of LLM and agentic benchmarks. In the last few months, I’ve found my sense of wonder diminish as I watch the numbers tick steadily up and the pelicans improve, but I don’t think this complacency is entirely appropriate. Real-world economic impact is probably the most important yardstick in the end, but it’s hard to translate a 10% boost in ARC-AGI-7 Verified Final-Bench Pro into GDP, or jobs, or layoffs, or the human misery of a person who finds themselves suddenly stripped of purpose.
Recently, I have had a different benchmark in mind when I try out a new model, especially the cheaper ones:
Who would I trust more to carry out everyday office tasks: this model, or Derek, the median guy I went to high school with?
(I have someone in mind. His name is not Derek.)
Derek is, by 2020 standards, fine. I like Derek! He’s hardworking enough, competent, and decently pleasant to talk to — not the sort of guy you’d want to run your billion-dollar startup, but perfectly capable of writing an email or putting together a slide deck. Derek’s in a better place than a lot of us.
I am increasingly convinced that Derek would get absolutely destroyed by the frontier LLMs available today on any given short-to-medium-duration task that takes place entirely on a computer. I don’t think this was true a year ago, but it is true now. Go back and forth via email to schedule a meeting? Any recent model with a suitable harness will be more responsive and accurate than Derek will. Data analysis? Derek doesn’t know what a Ridge regression is. Slide deck? Spreadsheet? Memo? The LLM will deliver a more polished and more sophisticated product in a fraction of the time for a fraction of the price, as long as all of the relevant information is available to the model (an important caveat!).
So where does Derek still win? Reliability, for one. Derek is probably less prone to making stuff up when he doesn’t know the answer, or at least I think I’d be better able to tell when Derek is lying than when Claude is. Derek is also currently better at navigating the poorly documented or word-of-mouth SOPs that define a lot of the work in most organizations. There’s also something to be said for accountability: you can yell at a person if something goes wrong. But ultimately, all of Derek’s advantages here seem like they can be addressed largely by organizational and business process changes. New companies will be built from the ground up to mitigate these problems. Ultimately, these are fingers in the dike rather than durable barriers that will keep Derek indispensable forever.
DerekBench worries me. If millions of Dereks—real people who may need to reskill and adapt rapidly—are left behind or rapidly rendered unnecessary, we will all suffer. It’s the China shock to American manufacturing all over again, except almost all of us work in a steel mill, myself included.
I still believe, perhaps naively, that I have enough expertise in one or two specific areas to be useful for at least a little while longer. I’m also fortunate to have stumbled into a field, drug discovery, that cannot simply be brute forced with infinite compute; there is still no calculation that can completely replace rat PK, or in vivo toxicology, or a clinical trial. Those barriers will fall too, slowly, though they won’t be felled by LLMs alone.
But the moment I am forced to leave those protected coves of inquiry? I’ll be Derek’d so fast I won’t know what hit me.