# A small model that reads NDAs · Insights · Dscvry

> A 184M-parameter model that reads an NDA, rules on 17 standard clauses and quotes its evidence. Tested once on a sealed set: 84.5% accuracy.

Source: https://dscvryai.com/insights/nda-clause-model/

1. [Home](https://dscvryai.com/)
2. [Insights](https://dscvryai.com/#insights)
3. Research

Research · 4 September 2026

# A small model that reads NDAs

It rules on 17 standard clauses and quotes its evidence: 84.5% on a sealed test set, running on an ordinary CPU.

By **Sharad Bapat**, founder

84.5% of 2,091 clause checks correct on a sealed test set

- **Result** 84.5% of 2,091 clause checks correct on a sealed test set
- **Evidence** 0.741 F1 for the quoted evidence
- **Size** 184.4 million parameters, 703 MB on disk
- **Built with** DeBERTa-v3-base, PyTorch, one rented GPU
- **Status** Concluded; runs locally on a CPU

Can a small model read a non-disclosure agreement and say, for 17 standard clauses, whether the contract includes each one, rules it out or never mentions it, and quote the sentences that show it? We trained one on public contracts and tested it once on a sealed set it had never seen. It scored 84.5%.

## The question

Reading an NDA is mostly checking the same things every time. Can the other side share it with advisers? Must it be returned at the end? Does anything survive termination? We wanted to know whether a model small enough to run on a laptop, with nothing sent to a cloud service, could do that checking reliably, and show its working.

## How it works

The data is [ContractNLI](https://stanfordnlp.github.io/contract-nli/) (Koreeda and Manning, 2021; CC BY 4.0): 607 real NDAs from public sources, each annotated for 17 fixed clause hypotheses. We used the published split of 423 for training, 61 for development and 123 for test.

For each hypothesis (“Receiving Party may share some Confidential Information with some third-parties”, for example) the answer is Entailment (the contract says so), Contradiction (it says the opposite) or NotMentioned.

The model has two heads on one DeBERTa-v3-base backbone. One gives the three-way verdict and the other picks which sentences support it, so every answer comes with its evidence. Long contracts are packed into 512-token windows. Contradiction is only 11% of the labels, so the loss gives it more weight; nothing is oversampled.

It’s small: 184.4 million parameters, 703 MB on disk. Training ran for a few hours on one rented GPU, a few dollars in all, and inference runs on a CPU.

## The sealed test

The 123 test contracts were never touched while the model was being built. Every choice was made on the development set. At the end, all four candidate checkpoints were run on the test set together, in one pass, and all four are reported below, so the winner can’t have been picked after seeing the results. The one adopted, seed 9, was already the front-runner on the development set.

- **84.5%**: verdicts correct, 2,091 clause checks
- **0.741**: F1 for the quoted evidence
- **0.788**: macro F1 across the three answers
- **31%**: the same checks by an off-the-shelf NLI model, untuned

The comparison is a general-purpose NLI model (MiniLM) used as it comes, with no training on contracts, on a 340-row development sample. The test score was also slightly higher than the development score (0.836 accuracy, 0.763 macro F1), so choosing a checkpoint on the development set didn’t overfit it.

| Candidate | Accuracy | Macro F1 | Evidence F1 |
|---|---|---|---|
| Seed 2 | 0.828 | 0.769 | 0.670 |
| Seed 7 | 0.797 | 0.721 | 0.559 |
| Seed 8 | 0.817 | 0.771 | 0.725 |
| Seed 9 (adopted) | 0.845 | 0.788 | 0.741 |

## By answer and by clause

| Answer | Precision | Recall | F1 |
|---|---|---|---|
| Entailment | 0.824 | 0.934 | 0.876 |
| NotMentioned | 0.924 | 0.810 | 0.863 |
| Contradiction | 0.650 | 0.600 | 0.624 |

Contradiction, the rare class, is the weakest, and it’s also the one that matters most in practice: it’s where a contract says the opposite of what you’d expect. Clause by clause, accuracy runs from 100% for “no reverse engineering” to 57% for “may keep some confidential information after returning it”.

| Clause | Accuracy | 95% interval |
|---|---|---|
| No reverse engineering | 1.000 | 0.970 to 1.000 |
| No solicitation | 0.967 | 0.919 to 0.987 |
| May develop similar information | 0.959 | 0.908 to 0.983 |
| May acquire similar information | 0.943 | 0.887 to 0.972 |
| Notice on compelled disclosure | 0.943 | 0.887 to 0.972 |
| No licensing | 0.919 | 0.857 to 0.955 |
| Confidentiality of the agreement itself | 0.902 | 0.837 to 0.943 |
| Limited use | 0.894 | 0.828 to 0.937 |
| Sharing with employees | 0.854 | 0.781 to 0.905 |
| Includes verbally conveyed information | 0.837 | 0.762 to 0.892 |
| Survival of obligations | 0.837 | 0.762 to 0.892 |
| Return of confidential information | 0.805 | 0.726 to 0.865 |
| Only technical information | 0.797 | 0.717 to 0.858 |
| May make copies | 0.748 | 0.665 to 0.816 |
| Must be explicitly identified | 0.724 | 0.639 to 0.795 |
| Sharing with third parties | 0.667 | 0.579 to 0.744 |
| May keep some after return or destruction | 0.569 | 0.481 to 0.653 |

## What didn’t work

Training wasn’t stable at first. Before the random seed was fixed, runs sometimes collapsed into predicting one answer for everything. With seeding, three of four runs succeeded and the fourth still collapsed, so the model is sensitive to its starting point.

One clause found a shortcut. An early model answered Contradiction for every one of the 61 development contracts on “may keep some after return”, whatever they said. It had learned that clause’s most common training answer, which happens to be the rare answer in development and test. That early model scored 20% on this clause in development. The adopted one scores 57% in the sealed test, and it’s still the weakest clause.

Shrinking it failed, twice. Standard 8-bit dynamic quantisation cut accuracy on a development sample from 83% to 49%. Quantising only the backbone made it worse: 32%, against 82% for the full model on the same sample. It also saved less than it promises. The model only got 1.5 times smaller, because the 128,000-word vocabulary table (98 million of the 184 million parameters) isn’t quantised at all. Our guess is DeBERTa’s disentangled attention, which sums several separate projections so small errors compound, but we didn’t verify that. The model ships at full size.

We also had the size wrong. For most of the project we described it as 86 million parameters, a figure that leaves out the vocabulary table. Measured from the checkpoint, it’s 184.4 million.

## Limits

The contracts are public ones, from documents found by web search and from securities filings, and all in English. Private deal paperwork may look different. It checks 17 fixed clauses and can’t answer arbitrary questions about a contract. The local demo splits sentences with a simple rule, which is rougher than the dataset’s own annotation. And its output isn’t legal advice.

## Bring us the work where wrong is expensive.

Start with the operational problem. We’ll work out the data, models and agents it needs, and where it should run.

[Talk to us →](https://dscvryai.com/contact/?topic=A%20small%20model%20that%20reads%20NDAs)

## More insights

- [Research OCR only what needs it Find the parts of a page that exist only as pixels. We read 5.5% of the page area instead of 98.8%, with fewer word errors.](https://dscvryai.com/insights/what-needs-ocr/)
- [Research Every word, straight from the PDF No model needed: 99.55% of 1.7 million held-out reference words found, each with its box.](https://dscvryai.com/insights/where-is-the-text/)
- [Research Scan or text, in a millisecond A 116 KB classifier that tells whether a PDF needs OCR without rendering it. It got all 525 test files right.](https://dscvryai.com/insights/scan-or-text/)