Where Jev fits among classification models
Introduction
TypeSafe’s Jev has generated significant discussion on social media since its release. Some categorize it as just a classifier, while others view it as a replacement for LLMs. This post situates Jev among existing classification models.
The primary target of Jev is automation for System 1 tasks, a term from Daniel Kahneman’s distinction between fast, intuitive thinking (System 1) and slow, reasoning-heavy thinking (System 2). While LLMs are fine-tuned for human preference, they are not designed for fast, type-safe, structured decisions with calibrated confidence values that software can consume directly.
TypeSafe expects AI automation to be dominated by 99% machine-to-machine interactions as compared to 1% human interaction. Therefore they design a model that behave predictably inside software rather than a model that can interact directly with humans.
Key characteristics of Jev
- Jev seems to work well for a large number of system 1 classification tasks in zero shot setting.
- Jev is fast becasue it is non-autoregressive (it does not generate output token by token) and can be parallelized, making it 40x-200x faster for comparable levels of frontier intelligence.
- It is also cheaper because output tokens, typically the most expensive part of an LLM call, does not cost anything for Jev which makes it 444.6x cheaper than comparable frontier models.
- It is type-safe and structured because it cannot hallucinate a value that is not provided in the input schema (this does not mean the output is correct).
- Jev is calibrated and consistent confidence values are possible due to their secret method Reinforcement Learning for Calibrated Decisions (RLCD). Consistent here means similar inputs yield similar outputs, so an outcome assigned a 70% probability should occur roughly 70% of the time. Having said that, claims of confidence scores need to be benchmarked as intial results show that it overestimates the confidence.
Jev supports three kinds of questions: Choice selects an option from a list, Score evaluates a state on a scale, and Noul determines whether a statement is true (yes/no probability). Choice and Score also return probabilities and confidence values.
Primary use cases
Some of the tasks for which Jev could be used:
LLM as a Judge: Jev evaluates agent outputs or traces against specific questions and returns structured, typed results instead of generated reasoning. It is a faster, lower-cost alternative for high-volume evaluations.
LLM routing: Jev classifies incoming requests into tiers, letting a router select the appropriate model for each request. In LiteLLM’s benchmark, Jev provided routing decisions significantly faster and at a much lower cost than the tested LLM classifiers.
Tool Calling: Before an agent executes a proposed tool call, Jev can evaluate whether it should be allowed. This adds a fast decision layer that blocks risky or inappropriate tool use without relying on a full LLM call.
For additional community resources on Jev, see:
Classification Models Landscape
We will not discuss the machine learning based classification here. Instead we will focus on encoder only models, LLMs (decoder only) and new opensource Jev like models.
Encoder-only language models
Standard supervised fine-tuning (BERT)
Modern encoder-only architectures began with BERT. Pretraining on a large corpus followed by task-specific fine-tuning allowed models to adapt to new domains with relatively small labeled datasets. To create a custom classifier, developers only needed a dataset of labeled samples to fine-tune the classification head rather than retraining all layers of the model.
Few-shot classification (SetFit)
To reduce the burden of acquiring large labeled datasets, few-shot classifiers emerged. SetFit, for example, applies contrastive learning to Sentence Transformer models by pulling similar samples closer together while pushing dissimilar ones apart. This approach achieves performance competitive with models like RoBERTa while requiring as few as eight labeled samples per class instead of thousands.
Zero-shot classification via NLI
To eliminate labeled training data requirements entirely, zero-shot classifiers reframed classification as a Natural Language Inference (NLI) task. An NLI model determines whether a text premise logically entails or contradicts a target hypothesis (such as checking if a premise entails “This text is about sports”). However, because inference must be run separately for every target class, processing costs scale linearly with the number of candidate categories.
Single-pass zero-shot classifiers (GLiNER2 & GLiformer)
GLiNER addressed these efficiency limitations in zero-shot Named Entity Recognition, and GLiNER2 adapted this design to general text classification. By prefixing input text with a task prompt and candidate labels ([P] task ([L] label 1 [L] label 2 ...) [SEP] text), GLiNER2 generates contextual embeddings for all label tokens simultaneously. An MLP projects these embeddings into logits in a single forward pass, delivering fast CPU inference with constant latency regardless of label count. GLiformer extends the same single-pass design.
LLMs
LLMs are autoregressive decoder-only models. The hard part with LLMs is making small decisions consistently across many requests and confidence scores are not accurate. LLMs can handle diverse tasks, including zero-shot text classification. However, because autoregressive language models generate text sequentially, they are inherently designed for generation rather than classification. Structured outputs attempt to constrain LLMs to a schema, but schema generation remains token-by-token, making it slow and prone to errors. Alternative approaches, such as SGLang’s /v1/score endpoint, allow labels to be specified in the prompt to output scores at the first token generation step. Nonetheless, LLM scores are generally not calibrated for seamless system integration, as LLMs are not instruction-tuned for that objective.
Sytem One / Decision models
Some of the open source models that have emerged after the realase of Jev that are tring to replicate the results produced by closed source Jev.
GLiNER2.5-Decide is an open-weight encoder model specifically engineered for schema-defined, structured decision-making. It processes text and typed questions in a single forward pass, using joint constrained decoding to enforce inter-question rules and eliminate contradictory outputs. Alongside structured choices, it returns probability distributions, confidence scores and constraint-feasibility metadata. It is designed to solve system 1 tasks.
Laya is perhaps the closest open-source alternative to Jev with calibrated confidence. It works well when the number of classes is low, but underperforms relative to Jev as class count increases. Furthermore, Laya struggles in zero-shot settings (base models score around 0.35 on the typed-decisions benchmark against a 0.318 random baseline), which is the primary appeal of Jev. Calibration for Laya also typically needs to be fine-tuned for specific tasks.
Jev vs. Encoder-only LMs vs. LLMs
The Jev announcement compares the model with LLMs in terms of speed, cost, and capability on System 1 tasks. If Jev’s zero-shot performance is equivalent to LLMs on System 1 tasks, then this comparison is reasonable. However, LLMs also excel at System 2 tasks, which Jev does not tackle, placing Jev closer to the domain of zero-shot encoder-only models. Therefore, it is worthwhile to compare Jev with zero-shot encoder-only classification models.
While extensive studies do not yet exist, initial results suggest that Jev performs better than NLI-based classifiers in a zero-shot setting. It performs worse than fine-tuned classifiers when a labeled dataset is available, which is expected: LLMs also generally do not outperform fine-tuned models in terms of accuracy, cost, and latency when domain-specific data is present.
One experiment compare Jev and Opus 5 on Banking77 dataset. Opus scores better on accuracy by 3.3% difference while costing more ($0.34 vs 7.44). One potential option is to use LLMs and Jev together: when Jev’s confidence score falls below a set threshold (e.g., 0.9), the system can fall back to an LLM. Both Jev and LLMs perform worse than fine-tuned classifiers.
Conclusion
Jev, like the other decision models in this post, is built for one job: decisions that software can consume directly, at high volume and low cost. That is where it beats LLMs and where it loses to fine-tuned classifiers when labeled data exists. The results come from initial comparisons, and they need confirmation on real workloads. Anyway, Jev has renewed interest in specialized classifiers for system 1 tasks rather than using LLMs as hammer for all the tasks.
Related articles
Leave a comment