Scan or text, in a millisecond

Before a document pipeline can read a PDF, it has to know whether the file has text in it or is just pictures of pages that need OCR. We built a small classifier that answers without rendering a single page, in about a millisecond, and tested it against the open-source tool that inspired it.

The question

Every document pipeline hits the same fork. A digital PDF already has its text, and extracting it is cheap and exact. A scanned one is only images, and needs OCR, which is slow and sometimes wrong. Many files are both: a typed contract with a scanned signature page. The fork has to be decided before anything else happens, so it needs to be fast.

pdf-inspector, a Rust library from Firecrawl, does this check and quotes about 10 to 50 ms. We set ourselves under 5 ms. Can a deliberately minimal classifier make that decision in under 5 ms, as accurately, and still do it inside a web page?

How it works

It reads the file’s structure and never looks at the pixels. One pass indexes every object in the file, including those packed inside compressed object streams, and walks the page tree.

Each page’s drawing instructions then run through a tiny interpreter, about a hundred lines of Rust that track three things: characters of text drawn, whether that text is invisible (how OCR layers hide themselves), and how much of the page images cover. It follows the transformation matrix so it knows each image’s real size, and it looks inside nested forms and inline images too.

Every page gets a verdict: text, scanned with a text layer, image only, or blank. A document needs OCR if any page is image only, and the answer says which pages.

The whole thing is about 800 lines of Rust with three dependencies: inflate (miniz_oxide), and MD5 and AES for files encrypted without a password. The WebAssembly build is 116 KB, 59 KB compressed.

How we tested it

The real files are the 375 source PDFs of ContractNLI (Koreeda and Manning, 2021; CC BY 4.0), public NDAs from company sites and filings. The ground truth is a slow, full parse of every page with PyMuPDF, recording its text and how much image covers it. The files are 370 digital, 2 scans with an OCR layer, and 3 mixed, each an executed contract with a scanned signature page. 18 are encrypted with an empty password, to restrict printing or editing.

Real digital contracts can’t test the other side much, so we built 150 known cases from 60 randomly chosen ones. 60 became pure scans (half stored as JPEG, like most scanners), 60 were made mixed by scanning the last page, and 30 were scanned and then run through Tesseract OCR, giving an image with an invisible text layer that doesn’t need OCR again. An independent labeller agreed with the construction on all 150.

To keep the race fair, our classifier and pdf-inspector 1.24.0 (its routing call, with default sampling of 8 pages) ran in the same Rust process on the same bytes, 9 times each, and the median counts.

Results

The test laptop’s speed drifted by about two times between runs, so compare numbers within a run. In the head-to-head, both tools ran under identical conditions, and our classifier was faster on 488 of the 525 files, typically 2.2 times. In two back-to-back runs of the classifier alone, the median was 0.67 ms native and 0.87 ms in WebAssembly, with the 95th percentile at 1.9 ms and 2.4 ms. These come from a rerun on 27 September, after four parser fixes ported from our text-extraction work; routing was 525 of 525 before and after. The first run, on 25 September, had the classifier faster on 523 files, typically 2.3 times, and PyMuPDF’s timings come from that run. That puts WebAssembly at about 1.3 times native, comfortably under 5 ms either way.

The 34 disagreements

Every one of pdf-inspector’s 34 misses sends a file to OCR that doesn’t need it. It never missed a file that did.

32 of them are scans that already have a text layer: the 30 Tesseract files and the 2 real ones. pdf-inspector calls them scanned. That’s a difference in definition more than a bug. If you distrust an existing OCR layer, running OCR again is the right call, and if you trust it, it’s wasted work. The other 2 are digital contracts it called scanned, and both have extractable text on every page.

What didn’t work

The first perfect score was luck. The first version had no decryption and returned “unknown” for the 18 encrypted files, and its default of “no OCR” happened to be right for all of them. Counting unknown as undecided gave the honest score: 507 right, 18 undecided. pdf-inspector decrypted them all from the start.

Decryption took two tries. RC4 worked first time, but AES failed on every file, because the bytes read from each stream included the line break before its end marker, so the data wasn’t a whole number of 16-byte blocks. Decrypting whole blocks only fixed it.

One file needed a text encoding we’d left out. Its content was ASCII85-encoded as well as compressed, and supporting that took 25 more lines.

Limits

The real set has very few files that need OCR (3), so the scanned side rests mostly on constructed files, which are cleaner than real scans, without skew, noise or tiled strips. It doesn’t handle AES-256 encryption or files that need a password; those come back “can’t tell”. And it counts characters roughly. The thresholds (40 characters; images over 30% or 80% of the page) mirror the ground-truth labeller’s, which suits this test but may need tuning elsewhere.

Credits

• pdf-inspector by Firecrawl (MIT), for the idea and the comparison.
• ContractNLI (Koreeda and Manning, 2021), CC BY 4.0, for the real PDFs.
• PyMuPDF for the ground truth, Tesseract for the OCR’d scans, and miniz_oxide and the RustCrypto crates inside the classifier.