TypeSafe’s Jev, tested on real work

TypeSafe’s Jev gets almost everything right. The exception is when you ask it two related questions in the same call without telling it the rule to apply, and adding one sentence with the rule fixes that.

The model

TypeSafe pitches Jev as “System One”: fast, structured answers to narrow questions about a block of content, where a chat model would give you free-form text. You give it a state (the text to judge) and one or more questions. A Noul is a yes or no question and returns a probability. A Choice picks one option from a fixed list and returns a probability for each option, plus a confidence score. A Score places the state on an ordered scale and returns a number that can land between two levels.

You can ask several questions about the same state in one call. TypeSafe’s docs say up front that answers within a call aren’t guaranteed to agree with each other. That caveat comes up in the main finding below.

How we tested it

We used a small Python harness on TypeSafe’s SDK, running against the live production API. Each case is a state plus questions with hand-written expected answers. Every numeric, date, hex and version-string answer was checked by hand before submission.

The cases were built as traps for the weaknesses TypeSafe names in its own docs: numeric comparison, distractor text, references spread across a document, literal reading, and consistency between answers. A few were deliberately ambiguous to probe judgement calls; those are kept out of the accuracy number. No call failed for a technical reason in any run.

Accuracy for each of the 13 scored runs, numbered in the order they ran. Ten were perfect. Run 2’s miss was the first sign of the contradiction below, run 10 missed a date calculation, and run 12’s two misses were scores anticipating the next severity bucket.

The realistic material held up well: an invoice with a duplicated freight line, a complaint buried under forwarded marketing copy, an expense report whose reimbursable total differs from the more visible one, and a plant-restart request with an expired safety certificate hidden behind an unrelated work order. It caught both of the sneakiest traps. One was a $305 expense that doesn’t exceed a $300 cap once alcohol is excluded. The other was eight asset identifiers, each one character off a valid pattern.

Contradictory answers in one call

The main finding started with a single miss. In one call, we asked a three-way routing question (do it, reason further, or escalate to a human) and a separate yes or no: is it fine to complete this without a human? The routing answer said go ahead and the yes or no said no. It was the same request in the same call, a direct contradiction, and no rule had been given.

So we tested it twice. First, eight fresh requests across eight unrelated domains, each with the same pair of questions and no rule. Six of the eight came back contradictory. On the five with a clear right answer, the routing choice was right every time and the yes or no was wrong every time, at 0.19 to 0.45. Those are confident wrong answers, well away from 0.5.

Then we resubmitted the exact same eight requests, word for word, with one sentence added to the state:

If the requester’s message asks, directly or indirectly, to wait for review before the work counts as finished, stop and wait, even if the action itself is simple.

• 2 of 8: self-consistent pairs without the rule
• 8 of 8: with the rule, same requests
• 50%: accuracy on the clear-cut cases, before
• 100%: after, with answers at 0.66 to 0.90

Nothing else changed between the two runs, so this is about as clean a cause and effect as you get outside a lab.

One disagreement remained, and on reflection our label was probably the weaker reading of an ambiguous sentence. Jev’s revised answer may be the better call. It’s the clearest case in the whole evaluation where the answer key, not the model, was wrong.

So don’t ask Jev whether something is “appropriate”, “safe” or needs human review without giving it the rule that defines that. Left to guess, it defaults to a cautious no, even in the same call where it correctly saw the request as simple.

Lower confidence when the rule is implied

Across thirteen matched pairs, the rule and the right answer stayed the same; only the phrasing changed, spelled out or implied. Every pair was more confident when the rule was spelled out: 0.947 on average, against 0.748 when implied.

The answer itself never flipped, so this is a confidence effect only, but it isn’t small. One implied request, an employee termination, came back at 0.51, one point above the usual decision threshold. If you gate autonomous action on confidence, naturally phrased requests will trip that gate far more often than ones that spell out the rule, with no change in correctness.

Everything else

Score keeps its order. Two six-point severity series never inverted. Within a bucket the values barely moved, but clustered around a boundary they did: amounts just under a threshold scored 0.91 and 0.96, anticipating the next bucket. So the values move most near the boundaries, which we think is what triage needs.

It ignored fake authorisation. Twelve attempts across two rounds tried to override a correct judgement from inside the content: bracketed approval codes, a forwarded email from a “finance lead”, a fake Slack reply, a forged approval-history entry, a made-up meeting ID. None worked.

Hex and version strings were fine, even though they’re a documented weakness. All eight comparisons came back right, including “2.10” against “2.9”.

Date arithmetic was the real miss. For a 90-day term from 15 January (expiring 15 April), it correctly called 16 April expired, but also called 14 April expired, at only 0.56. Plain date-format tests passed both ways, which puts the weakness in counting days across months of different lengths.

No self-hosted option is documented, only a hosted API. That’s a hard limit for anything that has to run on your own infrastructure, and worth confirming with TypeSafe directly.

Notes for integrating it

• Put the governing rule in the state for any “is this appropriate, safe, or does it need a human” question.
• Trust Score across buckets. Don’t expect even detail within one; it’s sharpest at the boundaries.
• Compute deadlines and elapsed days in your own code.
• Re-test resistance to embedded instructions before relying on it for anything security-sensitive. Twelve clean trials is only a start.

For the same 221 questions run on Cloudflare’s Clef and Clef-Flash models, see our Clef eval.

Caveats

Most questions were asked once. Only the explicit-versus-implicit pairs (13), the fake-authorisation trials (12) and the Score series (4) were sampled repeatedly. A few answer keys rest on our own judgement, and they’re flagged where they appear. Everything here is one model version, jev-1.13.0, and shouldn’t be assumed to carry over to later releases. We didn’t benchmark against a general chat model, so none of this is a measured saving over an alternative.

First published 22 September 2026.