Beyond the News
Sign in
Artificial IntelligenceAI ModelsAI AgentsOpen ModelsInference & APIs Active dossier

GLiNER2.5-Decide puts structured AI decisions on a 340M CPU model

Fastino's Apache-2.0 GLiNER2.5-Decide turns schema-defined routing, triage and guardrail decisions into a 340M open-weight model that can run locally on CPU and jointly enforce constraints across related answers.

Reviewed and published by Lars Jansen Updated 5 min read
Comment
First published

Key details

  1. GLiNER2.5-Decide has 340M parameters and is based on an encoder architecture rather than an autoregressive text generator.
  2. The model accepts text plus typed questions, permitted answer sets, descriptions, examples and cross-question constraints.
  3. Outputs include selected answers, probabilities, confidence scores and constraint-feasibility metadata.
  4. The model is released under Apache 2.0 and can run locally on CPU, GPU or air-gapped infrastructure.
  5. Fastino reports 167.3 ms p50 on a 48-vCPU Intel Xeon Platinum 8581C for a short 64-token workload.
  6. Fastino reports a 60.1% average on its internal 17-dataset Fast Decisions suite and first place on 9 datasets.
  7. The Jev comparison uses JevK5, an open reproduction, not TypeSafe's production Jev.

What builders should take away

  1. Use this class of model when the application already knows the allowed choices and needs a bounded decision rather than generated prose.
  2. Consider CPU deployment for low-volume routing, triage or guardrail workloads where adding a GPU or external model API would be disproportionate.
  3. Encode cross-field business rules in the schema when contradictory independent classifications would create unsafe or invalid downstream states.
  4. Benchmark on your own labels and traffic before replacing a general LLM or existing classifier; Fastino's headline accuracy and latency numbers are vendor-produced.
  5. Keep an escalation path for low-confidence cases and for requests whose correct answer is not represented in the declared schema.

What changed

Fastino released GLiNER2.5-Decide, a 340M-parameter open-weight encoder specialised for schema-defined decisions. Applications supply text plus typed questions and permitted answers; the model returns selected answers, probability distributions, confidence scores and feasibility metadata. It can jointly decode related questions under explicit implications, exclusions, cardinality limits and ordinal bounds. The checkpoint is Apache 2.0, runs on CPU or GPU, supports air-gapped deployment, and is available through the GLiNER2 tooling.

Why it matters

The recent System One model wave has mostly been framed around specialised decision APIs or GPU-hosted models. GLiNER2.5-Decide pushes the same software-facing pattern down to a 340M encoder that Fastino reports at 167.3 ms p50 on a 48-vCPU Xeon for short inputs. That makes bounded model routing, ticket triage, guardrails and tool selection plausible without a generative model or dedicated GPU. The joint constraint decoder is also operationally useful: downstream software can require related decisions to be mutually consistent instead of repairing contradictory classifier outputs after the fact.

It is a classifier, not a tiny chatbot

GLiNER2.5-Decide does not generate prose or reason through open-ended questions. A caller declares the legal answers in a schema and the encoder scores them. That narrower contract is the point: software receives bounded values, probabilities and confidence rather than text that must be parsed.

Constraints are part of decoding

Schemas can connect decisions with implications, exclusions, cardinality limits and ordinal bounds. Fastino's example shows why this matters: independently decoded safety and harm labels can contradict one another, while joint decoding searches for the highest-scoring assignment that satisfies the declared rules.

CPU deployment changes the economics

Fastino reports 167.3 ms p50 for a short two-head classification workload on a 48-vCPU Intel Xeon Platinum 8581C, with GPU latency around 38–47 ms on several tested accelerators. Those are vendor measurements on a specific schema and input size, but a 340M encoder is a very different deployment proposition from an 8B decision model or a frontier API.

The benchmark is useful but internal

Fastino reports a 60.1% average across its unseen 17-dataset Fast Decisions suite, ahead of an open JevK5 reproduction at 57.5% and SemIf at 56.4%. The suite is internally generated, JevK5 is not TypeSafe's production Jev, and these results therefore should not be treated as a neutral ranking of the decision-model market.

What to watch next

  • Independent comparisons with production Jev, CLM-8B and conventional fine-tuned classifiers on the same real workloads.
  • Calibration quality of the returned probabilities and confidence scores outside Fastino's benchmark suite.
  • Whether agent frameworks adopt the schema and constraint interface for routing and tool selection.
  • Real CPU throughput and cost under longer documents, larger label sets and concurrent production traffic.

Still unclear

  • The main accuracy benchmark is internally generated and reported by Fastino.
  • The published JevK5 baseline is an open reproduction rather than TypeSafe's production Jev.
  • Latency varies with input length, schema size, hardware and batching; the headline CPU figure is for a short batch-1 workload.
  • A bounded decision model cannot invent an action that the application failed to include in the allowed schema.

Sources

Direct reading behind this dossier.

2 sources

Discussion

Discussion is reader-contributed. Comments are not part of the BTN dossier or its editorial evidence.

0 visible comments

Join the discussion

Keep comments useful and relevant. Reader contributions may be moderated and are not BTN editorial evidence.

Sign in to comment
Continue reading
Artificial Intelligence

CLM-8B makes System One decision models open and locally deployable

CLM-8B targets the same narrow decision layer as Jev, but with open weights, local deployment and a contrastive architecture that separates state and action representations. The headline speed and coding results are researcher-produced and need careful interpretation.

Artificial Intelligence

Jev becomes Vercel AI Gateway’s fastest-adopted model in its first 24 hours

Jev’s launch claims were interesting; Vercel’s usage data is more useful. Nearly 13% of paid AI Gateway teams tried the typed decision model in its first day, while Jev also rose to a material share of gateway requests. That does not establish retention or production success, but it is unusually fast developer uptake for a model designed to make bounded software decisions rather than generate prose.