Short answer: Test AI document extraction on a fixed sample of your own document types, with the correct answers written down before the test runs. Report field-level correctness, whole-document correctness, how many documents needed review, what stayed unresolved and how long people still spent on them. One headline accuracy percentage, without its sample and error definition, tells you almost nothing.
Status of our own results: we publish this method before our pilot results. Measured results will be added with their sample size, date and limits once an evaluation is complete. Until then, nothing on this page is a performance claim.
Why one accuracy number misleads
"98% accurate" can mean 98% of characters, of fields, or of documents. It can be measured on clean digital PDFs that never reach your accounts team, or on the same examples the system was tuned on. A document with one wrong total is a wrong document, even if every other field is right. Ask what was counted, on which documents, and how an error was defined.
Build the test set first
Collect a sample that looks like a real month of work, not the easiest cases:
- Clean digital PDFs, as generated by supplier systems.
- Scans, including skewed, low-contrast and stamped pages.
- Phone photos, because that is how many delivery notes arrive.
- Mixed Arabic and English, including Arabic supplier names with English line items and mixed number formats.
- Missing or conflicting fields, such as an invoice without a purchase-order number or totals that don't add up.
- Several layouts per document type, since each supplier formats differently.
Write down the expected value for every field before testing: supplier, invoice number, dates, line items, VAT and totals. If two people disagree on the right answer, settle that first; the tool cannot be more consistent than the definition.
Keep two separate sets. A development set is used to tune prompts, rules and checks. A held-out evaluation set is touched only for the final measurement. Results on examples the team has already tuned against are optimistic by design.
For every run, record the model and version, configuration, date and how the sample was selected. A small first sample is fine, as long as it is described as a pilot sample and not a universal benchmark.
Report these numbers separately
| Measure | What it tells you |
|---|---|
| Field-level correctness | Share of individual fields extracted correctly, by field type |
| Whole-document correctness | Share of documents with every required field correct |
| Review rate | Share of documents the system sent to a person |
| Unresolved exceptions | Documents that still could not be completed after review |
| Human handling time, mean and median | Time people still spend per document, including review |
| Failures | Documents rejected, misread or posted incorrectly, with the reason |
Report mean handling time as well as the median. The median shows the typical document; the mean shows the workload. If 90 documents take two minutes and ten exceptions take thirty, the median is two minutes but the average is 4.8. Plan staffing and savings on the average.
Compare against the same manual task on a comparable sample. "Faster than before" only means something next to how long the manual process actually took.
What a results table should look like
When we publish pilot results, they will follow this structure, with every cell filled from the held-out set:
| Sample | Field-level | Whole-document | Review rate | Mean human time | |
|---|---|---|---|---|---|
| Clean PDFs | n = | ||||
| Scans and photos | n = | ||||
| Mixed Arabic and English | n = | ||||
| All documents | n = |
An empty table is more honest than a confident number without its denominator.
Questions to ask any vendor
- Which documents were in the test, and were they yours or theirs?
- Was the test set kept separate from the examples used for tuning?
- Is accuracy per field or per whole document, and how was an error defined?
- What share of documents went to review, and how long did review take?
- What happens when a field is missing: is it guessed, flagged or rejected?
- What happens when the target system rejects the update?
- Which model, version and settings produced the result, and when?
Estimate the value with your own numbers
Use the worksheet below with the averages from your own test. It shows hours released and capacity value after operating costs, and it will show a negative result if the assisted process is slower. That is a useful answer too.
Value worksheet: one document workflow
A scenario based on your own assumptions, per month, in AED. Released capacity is time your team gets back; it only becomes a cash saving where spending actually falls or the time is used productively.
Enter all five values to see the estimate. Blank fields are not treated as zero.
Formula. Hours released = (manual minutes − average assisted human minutes) × documents per month ÷ 60. Capacity value = hours released × loaded hourly cost. After operating costs = capacity value − monthly running cost.
- Use the average assisted time across all documents. A few long exceptions can double the real average while the typical case looks quick.
- Machine processing time only counts as human time if a person has to stay occupied while it runs.
- Setup and implementation costs are not included. Add them separately when comparing options.
- This is not a Digital Cubic estimate or benchmark. Measure both times on a comparable sample before relying on the result.
