Every vendor pitch for document extraction leads with an accuracy figure. Ninety-five per cent. Ninety-eight on clean scans. The number is almost always real and almost always useless, because it answers a question nobody in an operation is actually asking.
The operational question is not how often is it right. It is which fields does a human still have to look at, and how long does that take. Those are different questions, and a system can score brilliantly on the first while being unusable on the second.
Why an accuracy percentage hides the thing that matters
Consider a delivery challan with forty fields, extracted at 95% accuracy. That sounds strong. It means roughly two wrong fields on every document.
Now the operational question: which two? If the system cannot say, then a human has to check all forty to find them — and you have automated nothing. You have added a step that produces a document somebody must now proofread in full, which is slower than the typing it replaced, and considerably more boring, which means it will be done worse.
The failure is not accuracy. It is the absence of calibration: the model’s own sense of which answers it is sure about.
A system that is 95% accurate and cannot tell you which 5% is wrong forces a full manual review of every document.
A system that is 92% accurate and flags exactly the fields it is unsure about lets a human check four fields out of forty. It is less accurate and dramatically more useful.
Per-field confidence, and a threshold somebody owns
The design that works is unglamorous. For each field the extractor returns a value and a confidence score. Anything below a threshold is flagged for review before the document can progress. The operator sees a form where thirty-six fields are filled and quiet, and four are amber and waiting.
On the freight system we built for a Kolkata broker, challans are extracted field by field, and anything scoring under 95% is flagged amber for operator review before the trip can be dispatched. Across a sample of thirty challans, per-field accuracy ran between 96 and 99 per cent — and around nine fields out of forty were flagged for review on an average document.
Read those two numbers together, because the relationship is the whole point. Accuracy is high, yet nearly a quarter of fields get flagged. The system is deliberately pessimistic. It would rather ask about a field it got right than let through one it got wrong, because in that operation a wrong vehicle number on a challan is a dispute with a customer three weeks later, and thirty seconds of operator attention is much cheaper than that.
Setting the threshold is a business decision
This is the part that gets handed to engineers and should not be. The threshold encodes an exchange rate between two costs:
- The cost of a review — seconds of operator attention, multiplied by document volume.
- The cost of an error that escapes — a wrong figure on an invoice, a rejected e-way bill, a dispute where your record does not match the customer’s.
Those costs differ enormously by field. A misread remark column costs nothing. A misread quantity or vehicle number costs a lot. So the threshold should differ by field too — high for the fields that touch money and compliance, relaxed for the fields that are effectively notes. Set one global threshold and you will either drown operators in flags or let expensive errors through, depending on which way you erred.
Four things that decide whether this works in production
1. The review interface is the product
Most of the engineering value here is not in the model. It is in the screen where a human resolves nine amber fields in twenty seconds: the source image beside the form, the disputed region highlighted, keyboard-only progression from flag to flag. Get this wrong and a well-calibrated extractor still fails, because the humans route around it.
2. Corrections have to be captured as data
When an operator fixes a field, that correction is training signal and a metric. It tells you which fields are chronically wrong, which document formats are worst, and whether last month’s change helped. If corrections are only written into the final record and not logged as corrections, you have thrown away the only feedback the system generates.
3. Format drift is the real maintenance cost
Counterparties redesign their documents without telling anyone. A new challan layout arrives and accuracy on that customer quietly collapses. Track flag rates per document format, not in aggregate — aggregate accuracy will barely move while one customer’s documents become unusable.
4. Extraction must not be the system of record
The extracted values populate a form. The human confirms or corrects. The confirmed record is what the business runs on, and it carries a flag saying which fields were machine-read and which were touched by a person. Six months later, when a figure is disputed, that distinction is the first thing you will want.
What to ask a vendor
If you are evaluating document extraction, the accuracy number is not the interesting part of the conversation. These are:
- Do you return per-field confidence, or one score per document?
- Can the threshold be set per field, by us, without a deployment?
- What does the review screen look like, and how many keystrokes to clear a flagged document?
- Are operator corrections stored as corrections, and can we export them?
- Can you report flag rate segmented by document format?
A vendor who leads with accuracy and cannot answer these has built a demo. A vendor who answers all five has built something that survives contact with an operation — which is, in the end, the only test that has ever mattered.