A small model that reads NDAs

Can a small model read a non-disclosure agreement and say, for 17 standard clauses, whether the contract includes each one, rules it out or never mentions it, and quote the sentences that show it? We trained one on public contracts and tested it once on a sealed set it had never seen. It scored 84.5%.

The question

Reading an NDA is mostly checking the same things every time. Can the other side share it with advisers? Must it be returned at the end? Does anything survive termination? We wanted to know whether a model small enough to run on a laptop, with nothing sent to a cloud service, could do that checking reliably, and show its working.

How it works

The data is ContractNLI (Koreeda and Manning, 2021; CC BY 4.0): 607 real NDAs from public sources, each annotated for 17 fixed clause hypotheses. We used the published split of 423 for training, 61 for development and 123 for test.

For each hypothesis (“Receiving Party may share some Confidential Information with some third-parties”, for example) the answer is Entailment (the contract says so), Contradiction (it says the opposite) or NotMentioned.

The model has two heads on one DeBERTa-v3-base backbone. One gives the three-way verdict and the other picks which sentences support it, so every answer comes with its evidence. Long contracts are packed into 512-token windows. Contradiction is only 11% of the labels, so the loss gives it more weight; nothing is oversampled.

It’s small: 184.4 million parameters, 703 MB on disk. Training ran for a few hours on one rented GPU, a few dollars in all, and inference runs on a CPU.

The sealed test

The 123 test contracts were never touched while the model was being built. Every choice was made on the development set. At the end, all four candidate checkpoints were run on the test set together, in one pass, and all four are reported below, so the winner can’t have been picked after seeing the results. The one adopted, seed 9, was already the front-runner on the development set.

84.5%: verdicts correct, 2,091 clause checks
0.741: F1 for the quoted evidence
0.788: macro F1 across the three answers
31%: the same checks by an off-the-shelf NLI model, untuned

The comparison is a general-purpose NLI model (MiniLM) used as it comes, with no training on contracts, on a 340-row development sample. The test score was also slightly higher than the development score (0.836 accuracy, 0.763 macro F1), so choosing a checkpoint on the development set didn’t overfit it.

Candidate	Accuracy	Macro F1	Evidence F1
Seed 2	0.828	0.769	0.670
Seed 7	0.797	0.721	0.559
Seed 8	0.817	0.771	0.725
Seed 9 (adopted)	0.845	0.788	0.741

By answer and by clause

Answer	Precision	Recall	F1
Entailment	0.824	0.934	0.876
NotMentioned	0.924	0.810	0.863
Contradiction	0.650	0.600	0.624

Contradiction, the rare class, is the weakest, and it’s also the one that matters most in practice: it’s where a contract says the opposite of what you’d expect. Clause by clause, accuracy runs from 100% for “no reverse engineering” to 57% for “may keep some confidential information after returning it”.

Clause	Accuracy	95% interval
No reverse engineering	1.000	0.970 to 1.000
No solicitation	0.967	0.919 to 0.987
May develop similar information	0.959	0.908 to 0.983
May acquire similar information	0.943	0.887 to 0.972
Notice on compelled disclosure	0.943	0.887 to 0.972
No licensing	0.919	0.857 to 0.955
Confidentiality of the agreement itself	0.902	0.837 to 0.943
Limited use	0.894	0.828 to 0.937
Sharing with employees	0.854	0.781 to 0.905
Includes verbally conveyed information	0.837	0.762 to 0.892
Survival of obligations	0.837	0.762 to 0.892
Return of confidential information	0.805	0.726 to 0.865
Only technical information	0.797	0.717 to 0.858
May make copies	0.748	0.665 to 0.816
Must be explicitly identified	0.724	0.639 to 0.795
Sharing with third parties	0.667	0.579 to 0.744
May keep some after return or destruction	0.569	0.481 to 0.653

What didn’t work

Training wasn’t stable at first. Before the random seed was fixed, runs sometimes collapsed into predicting one answer for everything. With seeding, three of four runs succeeded and the fourth still collapsed, so the model is sensitive to its starting point.

One clause found a shortcut. An early model answered Contradiction for every one of the 61 development contracts on “may keep some after return”, whatever they said. It had learned that clause’s most common training answer, which happens to be the rare answer in development and test. That early model scored 20% on this clause in development. The adopted one scores 57% in the sealed test, and it’s still the weakest clause.

Shrinking it failed, twice. Standard 8-bit dynamic quantisation cut accuracy on a development sample from 83% to 49%. Quantising only the backbone made it worse: 32%, against 82% for the full model on the same sample. It also saved less than it promises. The model only got 1.5 times smaller, because the 128,000-word vocabulary table (98 million of the 184 million parameters) isn’t quantised at all. Our guess is DeBERTa’s disentangled attention, which sums several separate projections so small errors compound, but we didn’t verify that. The model ships at full size.

We also had the size wrong. For most of the project we described it as 86 million parameters, a figure that leaves out the vocabulary table. Measured from the checkpoint, it’s 184.4 million.

Limits

The contracts are public ones, from documents found by web search and from securities filings, and all in English. Private deal paperwork may look different. It checks 17 fixed clauses and can’t answer arbitrary questions about a contract. The local demo splits sentences with a simple rule, which is rougher than the dataset’s own annotation. And its output isn’t legal advice.