The PRD Is Failing in the AI Era: Evals Are Replacing Requirements Documents
Lisa Murkin, a lead product manager at digital product consultancy Elsewhen, argues on Mind the Product that a quiet shift is underway: specification has moved from prose to executable artifacts. The PRD as a long, static document has stopped doing the core work of a spec, because the work now has to live somewhere it can be run. And that somewhere has been hiding in plain sight: a golden dataset.
A few months into building an AI product at pace, she noticed her usual PM toolkit was no longer keeping up. Product artifacts — PRDs, research, documentation — were produced at a fraction of the speed engineering was shipping. New features arrived daily, each the kind of thing that would have taken weeks or months to build before AI. Demos always look good because you pick the inputs; the harder question — whether the product was genuinely getting better at what it was meant to do — had become hard to answer with confidence.
Why the PRD stopped working
Three things broke the PRD's contract for AI work, and they broke it together:
- Engineering has become faster than product. A competent engineer with modern tooling can ship an AI feature in an afternoon that would have been a two-sprint project a year ago. By the time a PRD is written, three versions have been prototyped.
- PRDs were always written in the wrong register for AI work. A functional requirement describes intent: what the feature is for and who it is for. For deterministic software that was sufficient; engineering closed the gap and QA caught misalignment. For AI systems, the same intent can be satisfied by thousands of output distributions, most of them not what you meant.
- Early-stage AI products don't have stable targets. A product is shaped continuously by shifting conversations, but a PRD assumes a fixed thing to specify against. Written specs degrade faster than AI products evolve.
What a golden dataset actually is
A golden dataset is a curated collection of real inputs paired with examples of good outputs, or clear criteria for what would make an output acceptable. It is the artifact an AI system is run against every time a meaningful change is made. In the author's project, an engineer started building one as sets of images: examples of brand-compliant creative alongside examples that clearly weren't, so any change to a model, prompt, or workflow could be tested against the same reference set. That curation is itself product work — deciding what belongs, what counts as a good output, and where the edge cases live.
Eric Schmidt put it sharply: the 10x advantage is no longer execution but defining what counts as success. A programmer writes a spec and an evaluation function, runs it at 7pm, and wakes up to what was invented overnight. For a PM, the uncomfortable reading is that specifying the problem precisely has always been the job — we got away with being approximate because engineering filled in the gaps. It doesn't anymore.
Three things a golden dataset does that a PRD cannot: it encodes "good" in a form the system is tested against automatically; it gets run rather than read, living inside the development loop; and it improves with the product rather than decaying against it. The dataset is at its best when it isn't owned exclusively by product — a dataset gatekept by Product recreates the problem PRDs had.
Where this goes wrong
"PRDs are bad, golden datasets fix everything" would be dishonest. Three failure modes worth naming: overfitting to the dataset (a system tuned to ace a fixed set of examples doesn't generalize — static datasets rot fast); false confidence from small samples (thirty examples catch obvious regressions but don't tell you the system works); and divergence from real-world use (production reflects what users actually do, and the gap widens quickly). Pulling production samples into the dataset on a regular cadence is the only way to close it.
The hard part
You have to articulate what "good" looks like concretely enough to put it in a spreadsheet — a sharper definition than most PRDs ever forced out of us. You have to tie eval scores back to commercial outcomes, whether task completion, support load, or revenue. And you have to accept that the dataset is a living artifact that needs continuous tending, not a document you write once and forget.
Where the PRD still earns its keep
The team's request for focus and direction was correct; the PRD was the wrong form, not the wrong instinct. There is still real value in a shared source of truth: what the product is, who it's for, why it exists, and what we're prioritizing next. A twenty-page PRD nobody reads is worse than no document at all. A short, visual-first overview the whole team can orient to in a minute earns a read. Prose and pictures for why and for whom. Dataset for what.
The shift happens the first time you run the dataset against a change before it ships. That moment — a numbers conversation instead of a taste conversation — is where the power over "what good looks like" moves back to something shared, measurable, and alive.
16IDC perspective
For teams building AI websites or AI products, the core takeaway is that requirements analysis is shifting from "documents" to "runnable evals." Before building any AI feature, define what a good output looks like and organize it into a dataset you can run repeatedly — that will drive engineering better than a static PRD. For systematic methodology, see the Requirements Analysis category, including tech stack evaluation and AI prompt engineering basics.
Source: https://www.mindtheproduct.com/evals-are-the-new-prd/