What is this file, really?

A document pipeline starts by trusting a filename, and filenames can be wrong. We built a small check that reads a few kilobytes of the file’s structure and says what it really is, and whether it’s locked, renamed or cut off. Then we tested it against three well-known tools on files it had never seen.

The question

Before a pipeline can read an upload, it has to route it: PDF to one parser, spreadsheet to another, old binary Word files to a third. Most systems route on the extension. The costly failures are the ones the extension hides: a password-protected .docx that no parser can open, a spreadsheet saved as .pdf, a download that stopped halfway.

Checking the first few bytes, the "magic number", is a solved problem, and it isn’t enough. Every modern Office file starts the same way as any zip, and every old Office file, Outlook message and locked document starts the same way as each other. So: can a check with no model and no dependencies tell these apart, correctly, in microseconds? And does it beat Magika, Google’s deep-learning detector, and the long-standing tools?

How it works

The first 8 KB settle most formats: PDF (allowing a little junk before the header, as readers do), images, archives, Rich Text, and the text formats.

For zip files it reads the table of contents and never the contents. A zip lists its entries at the end. word/document.xml means Word, xl/ Excel, ppt/ PowerPoint; a stored mimetype entry first means OpenDocument or EPUB. Nothing is decompressed.

Old Office files need the directory. .doc, .xls, .ppt, .msg and password-protected modern Office files all use one container format, a small file system inside the file. The check walks its directory and reads the names of the top-level streams: WordDocument, Workbook, PowerPoint Document, Outlook’s __substg1.0_ streams, or EncryptedPackage for a locked file.

On top of the type it sets three flags: encrypted, extension doesn’t match, and truncated (a PDF with no end marker, a zip with no table of contents, a container whose directory runs past the end of the file). It’s about 600 lines of Rust with no dependencies, and the WebAssembly build is 39 KB, 18 KB compressed.

How we tested it

The truth comes from parsers. A file’s true type is whatever a real parser can open it as: PyMuPDF for PDF, python-docx, openpyxl and python-pptx for modern Office, olefile for the old container, Pillow for images, and the standard gzip, JSON and XML readers. A file no parser opens isn’t scored.

We tuned it on 976 files from govdocs1 (thread 001, crawled from US government sites), plus 270 files we built to cover what it lacks: fresh Office files, 30 password-protected ones, OpenDocument, email, 20 renamed files and 15 cut in half.

Then the rules were frozen, checked by a hash of the source, and run once on files nobody had looked at: govdocs1 thread 002 (990 files) and the test data of Apache POI (1,529 files), the Java library for Office formats. POI’s collection is hard on purpose, with real files from bug reports, odd variants and fuzzer-generated wrecks.

Every tool got the same files. Magika 1.0.3, libmagic (the engine behind the Unix file command, 5.45) and file-type 22.1.1 each ran in their own batch mode, and each tool’s answers were mapped to the same 30 labels by a fixed table.

Results

• 1,914 of 1,917: held-out files identified correctly
• 47 of 47: password-protected Office files; Magika found none

On ordinary files, trusting the extension is right 97.8% of the time, about as often as Magika. The tools differ mostly on the few costly cases in the next section, and the totals don’t show that.

The costly cases

There were 47 password-protected Office files (30 built, 17 in POI). This check found all 47 and libmagic 46. Magika found none: it called them Windows thumbnail caches (25), old Word files (16), old Excel files (2) or unknown (4). file-type calls them a generic container, and the extension calls them ordinary Word, Excel and PowerPoint files, which they aren’t until someone types the password.

file-type sees only the shared container in old Office and Outlook files, so it misses 885 held-out .doc, .xls, .ppt and .msg files. That’s most of its gap.

All 29 renamed files were flagged (20 built, 9 found in the held-out set), with no false alarms. All 15 built cut-off files were flagged too, and still named by what they were meant to be. In the held-out set it flagged 35 as cut off: 31 are POI’s fuzzer and crash tests, and we checked the other 4 by hand. They’re all really broken.

Three old PDFs start with junk before the header, two of them wrapped in a classic Mac file header. file-type missed all three, Magika two and libmagic one.

Speed and size

These cross languages and runtimes, so only the orders of magnitude are worth comparing. This laptop also drifts about two times between runs. The WebAssembly build gave the same answer as the native one on all 3,765 files. The check typically asks for 8 to 9 KB of a file, and at most 182 KB (overlapping reads counted twice).

Limits

Text formats are the weak spot. Whether a file is CSV or plain text, or HTML or plain text, has no parser to settle it. On the easy files we built it got 60 of 60 (libmagic 54, Magika 52), but on real ones we can only report where the tools agree. Magika is built for this, and for source code, which is outside these 30 labels. It knows over 200 types, and this compares only the 30 here.

The files aren’t today’s uploads either: govdocs1 is mostly from the 2000s, and POI leans towards odd files.

The check names a file’s type; it doesn’t validate the file. Files built to fool a detector weren’t tested, and a file that opens in one Python library isn’t proof that every pipeline can process it.

What didn’t work

Three rules were wrong at first, all found on the tuning set before the freeze. A locked .docx was flagged as renamed, but the name is right, since Office opens it after asking for the password. Cut-off spreadsheets came back as plain zips, because the file listing their parts is written last. And an old SGML document with a <title> tag was called HTML.

Our first comparison was broken. The file list had Windows line endings, so libmagic couldn’t open a single file and scored 12. We caught it by looking at libmagic’s raw output before believing the table.

The truth needed a fix too. The labeller called any zip the Python libraries refused a plain zip, which is wrong for macro-enabled files, templates and binary workbooks. Those became unscored, and we fixed this before looking at any held-out results.

Three held-out misses stay in. Two are fuzzer-made files it called PowerPoint and Excel, where the parser found no Office stream, and one is a .txt that’s really a binary weather-data format, which it called text.

Credits

• govdocs1 from Digital Corpora (Garfinkel et al., 2009), public domain, and Apache POI’s test data (Apache-2.0), for the files. None of them is served here. The four samples in the demo are made-up files, generated for this page.
• Magika by Google, libmagic by the file project, and file-type by Sindre Sorhus, for the comparison.
• PyMuPDF, olefile, python-docx, openpyxl, python-pptx and Pillow for the ground truth, msoffcrypto-tool for the locked files, and wasm-bindgen for the browser build.