Canon: a custom document classification benchmark

Canon wanted to assess custom document classification on its own data and compare it with an off-the-shelf service. The work focused on classification: assigning a document to a category.

DNA Solutions built and evaluated an open-source classification model using text features and a linear SVM. Training and inference were packaged in Python. The custom classifier is distinct from the OCR, API and hosting services that a complete document-processing system may use.

• 94.73% accuracy for the custom classifier versus 84.21% for Azure AI Document Intelligence on one dataset.
• 87.5% versus 85.71% on a second dataset.
• 100% for both approaches on the third dataset.

The custom classifier scored higher on two of the three tested datasets and tied on the third. The highlighted 94.7% is a rounded result for one dataset, rather than an average across all three.

These measurements describe the tested documents and configuration. They do not establish accuracy on new document populations or a guaranteed production result. A deployment decision needs evaluation on representative documents, error analysis, integration tests and an operational review.

An open-source classification model gives control over that component. It does not make the complete pipeline independent of third-party OCR or cloud infrastructure. Hosting, data transfers, retention, access controls and applicable compliance obligations need to be assessed separately for the planned deployment.