How to measure a legal automation pilot
By Asım Serdar Yılmaz · Published · Last updated · 2 min read
A pilot without metrics is a demo: it looks impressive in the room, everyone nods, and three months later no one can say whether it worked. I have run enough of these in a large in-house team to be fairly sure the plan matters more than the tool.
Baseline the workflow before the model touches it. Time the cycle end to end, count the touches, meaning how many people handle a task and how often it changes hands, and add up the external spend, outside counsel hours and per-matter fees. Those three are the minimum baseline, and the quality metrics belong next to them rather than in an appendix someone reads later: accuracy, rework, error severity.
Then define the error types each task can produce. I sort them into three: critical errors that change a legal outcome, material errors that need real rework, and formal errors that are cosmetic. Set a tolerance for each class, zero for the critical one, and write the numbers down before the pilot starts rather than after you have seen the output.
The number to watch is reviewer time. A model answers in seconds, and the hour a lawyer then spends checking and fixing that answer is the part that costs money. If review erases the saving, the pilot failed however fast the model ran.
Decide the kill-or-scale threshold in advance, because a threshold set once the results are in is only a description of the results. A widely discussed 2025 report from MIT NANDA, The GenAI Divide, found that 95 percent of enterprise generative AI pilots showed no measurable impact on profit and loss. In its 2025 report, Thomson Reuters found only 20 percent of the legal professionals surveyed are measuring the return on AI. Read those two findings together and the position is an uncomfortable one: if you are not measuring, you cannot know which side of it you are on.
Baseline, tolerances, reviewer time: all of it should land in one place, a pilot decision record carrying the owner’s name, the date, the kill-or-scale threshold and the evidence behind the call. When the scale decision comes, argue it from that record rather than from the nodding in the room. It answers the question once; a live workflow asks it again on every change.