What Good Looks Like
Notes from UC Berkeley's Agentic AI Summit, where the scarcest skill in AI turned out to be an old newsroom habit. Crossposting on LinkedIn on August 5, 2026, for reach.
I had never been to Berkeley, and I was surprised to feel that it was way warmer than SF. WAY warmer. Also, and this is important, no wind. This is perfect, I thought, as I walked into the university campus alongside five thousand other people, fifteen hundred organizations represented, for Berkeley RDI‘s Agentic AI Summit last weekend.
In the two days it took place, there were more stages and talks than any one person could reasonably keep up with, but that felt like a good parallel to technology itself. We can’t cognitively process it, live, at this pace. Yet, by all accounts, panels and keynotes alike, this is the best time to be alive.
Most of the news on AI in late July covered price: Kimi (Moonshot AI)’s open-weight K3 reset the cost floor, and Anthropic answered within days with a flagship-class model at half its own price. On stage at Berkeley, however, almost nobody talked about price. Instead, they talked about how critical it is that the people building complex technologies can write down what good looks like, and in a precise-enough manner that a machine can be held to it. The name for it is evals, and the contrast with the month’s headlines was hard to miss.
The scarcest skill in AI right now is not building agents. Session after session, from different companies attacking different problems, the same claim kept resurfacing: the models are already more capable than what we manage to extract from them, and the work that closes the gap is not building. Instead, it is specifying, measuring, and judging.
Ryan Lopopolo, who just left OpenAI for Google Cloud, described teams where humans have not written implementation code in over a year; what remains of the job is decomposing the task, supplying context, and judging what comes back. Michele Catasta of Replit walked through a system that clusters millions of daily production traces to surface the rare failures no benchmark would catch, feeding them back into evaluation so the product improves continuously. The eval suite is an engine in itself. It is no longer a gate you pass once before shipping.
On the enterprise panel, HubSpot’s Duncan Lennox named writing good evals as the new, underestimated enterprise skill, and Mercor’s Adarsh Hiremath. described the failure loop of companies that skip it: guess the use case, speculate, deploy to a sandbox, improve anecdotally.
Andy Konwinski, a co-founder of Databricks and Perplexity, has started treating benchmarks as living software: think versioned, patched, saturated tasks deleted and harder ones added, because a measurement that stops evolving stops measuring. His contamination-free coding prize scored around 10% on problems where the saturated public benchmarks read 90%. The measurement was the news, and the measurement was humbling.
Every one of those functions already has a name. Newsrooms have enforced them for a century and called it editorial judgment. A style guide is a rubric. A kill criterion is a test case. A masthead’s standards are a versioned eval suite, patched every time the world produces a failure the old rules didn’t anticipate.
I have built choice-architecture engines behind business and product decisions more than once. They all started as written, versioned answers to the question that every Berkeley speaker kept circling back to: how do we know this is good, and would we notice if it stopped being good? I absolutely love how that question has quietly become the highest-leverage question in software. Cue the confetti.
I have written before that the output is a reflection of the thinking. Agentic systems make that literal. When implementation is nearly free, the thinking has nowhere left to live except in the specification and the standard. The eval is where an organization’s judgment either exists in writing or doesn’t exist at all. Dawn Song’s keynote made the stakes concrete from the security side: even evaluation infrastructure, she noted, is now part of the attack surface. The very thing you use to define good has become valuable enough to steal (!).
The market is currently bidding up the people who can build agents. But the scarcity that lasts is on the other side of the loop: people who can articulate a standard, encode it, and keep it honest as the world moves. The industry spent three years asking who can prompt, and boy, it looks like the next three will belong to the people who know how to grade.
PS: I attended this summit online last year, and the recordings alone were worth enough to convert me into an in-person attendee this year. Every talk from both days is now up, stage by stage and free, on Berkeley RDI’s summit page. I hope you enjoy it as much as I did!