Sub-35ms 离散路由 · 100% 排序无关 · 双系统结晶
替代耗时较长、格式不可控的生成式路由调用。通过首 Token 条件似然提取实现 <35ms 判定,从机制上避免 JSON 解析异常;支持选项排列无关性(任意排列胜者一致),并提供双系统决策机制:常规请求走快路径,边缘样本由推理模型分析并提炼为规则缓存。
支持完全离线部署。兼容开源模型(如 Qwen3、Gemma 4)的词表语法约束解码,以及专用 Laya (ModernBERT 322M) 双向编码器;延迟在 35ms 以内,无需依赖外部云端接口。
与商业 Jev 或云端接口协同运行:由边缘前置处理 70% 高频请求以降低调用成本;置信度护栏拦截歧义样本减少误判,遭遇云端超时或限流时自动熔断降级。
常规请求由快路径高效判定;歧义样本交由推理模型分析并提炼为规则,使后续同类请求直接走快路径。同时具备选项排序无关性,避免排列偏差。
Observe how OpenJevPro extracts normalized posterior distributions over candidate enums.
Type-safe building blocks designed for low-latency agent routing and policy gates.
针对离散 Enum 类型的多分类路由。通过词表掩码约束输出空间,直接计算各选项的条件对数似然,避免 JSON 解析异常;支持 order_invariant=True 消除选项排列顺序偏差。
二元断言评估器($P_{true}, P_{false}$),输出经标定的概率与不确定度区间,适用于工作流门控与安全检查。
有序档位与等级评分。依据各档位后验概率质量计算连续期望值,避免离散打分的不稳定抖动。
运行期动态标定与阈值拦截($\tau = \max(\tau_{\min}, \alpha / K)$)。将输出概率反推为伪对数似然并进行温度缩放;置信度不足时主动输出 UNKNOWN 弃权,使有效回答准确率达 100%。
τ = max(0.72, 1.25 / K)
双层成本分流与熔断网关。70% 以上的高频意图直接由本地模型解析(<20ms);难以确定的边缘样本再转发至商业 API。遭遇云端超时或限流时自动熔断切回本地,避免上游调用发生未捕获异常。
常规请求由快路径模型在 35ms 内完成判定;歧义样本自动交由慢思考模型进行因果推理,并将证据链提炼为判定准则写入规则库,使后续同类请求直接走快路径。
Empirical validation on PolyAI Banking77 confirms that open models with grammar decoding achieve statistical parity with dedicated Jev models.
General frontier model ceiling. Expensive per-call pricing, 300~600ms latency, proprietary cloud lock-in.
Dedicated compact model. RLCD calibrated, ~25-45ms latency, but proprietary commercial closed source.
Zero task fine-tuning needed. SGLang grammar decoding matches Jev within overlapping 95% confidence intervals.
| Engine | Naive Acc | Selective Precision | Tentative Top-1 | OOS Recall | P50 Latency | ECE Error |
|---|---|---|---|---|---|---|
| OpenJevPro (gemma4:cloud) | 94.4% (34/36) |
100.0%
|
100.0%
|
100.0%
|
1,084ms | 0.025 |
| Direct Open LLM (gemma4 + JSON) | 97.2% | 96.7% | 97.2% | 100.0% | 630ms | 0.029 |
| TypeSafe Jev (1.13.0 Live API) | 97.2% | 96.7% | 97.2% | 100.0% | 740ms | 0.026 |
Optimized for non-thinking classification with tiered latency and cost profiles.
| Tier | Architecture | Active Parameters | Latency | Recommended Workload |
|---|---|---|---|---|
| Tier 0: Dedicated System 1 | Laya (ModernBERT-large) | 322M Bi-directional | < 35ms | Local microservice, zero-cloud-cost ultra-fast edge routing (Apache 2.0) |
| Tier 1: Edge Router | Qwen3-1.7B / 4B (Dense) | 1.7B / 4B | < 50ms | High-QPS intent gating with /no_think parameter |
| Tier 1: Multimodal Edge | Gemma 4 E4B (Vision) | 4.5B Effective | ~60ms | Invoices, screenshots, and visual document routing |
| Tier 2: Production Standard | Qwen3-30B-A3B (MoE) | 3B Active / token | 60–90ms | Enterprise routing: 30B representation at 3B compute cost |
| Tier 2: Single-GPU Private | gpt-oss-20b (MXFP4 MoE) | 3.6B Active | ~80ms | Private on-premise single GPU deployment, Apache 2.0 |
| Tier 3: Long-tail Fallback | DeepSeek-V4.1-Flash | 8B~16B Active | 200–400ms | Complex multi-turn tool arbitration & fallback routing |
Drop-in Python client compatible with vLLM, SGLang, and OpenAI endpoints.
from enum import StrEnum
from openjevpro.schemas import ChoiceDecision
from openjevpro.client import OpenJevProClient
class TicketRoute(StrEnum):
BILLING = "billing"
TECH_SUPPORT = "tech_support"
SECURITY = "security"
ESCALATE = "human_review"
# Connect to local vLLM / SGLang, or pass mock=True for zero-GPU CI/CD
client = OpenJevProClient(
base_url="http://localhost:8000/v1", # or mock=True
model="Qwen/Qwen3-4B-Instruct",
temperature_scaling=1.30,
abstain_threshold=0.45
)
decision: ChoiceDecision = client.decide_choice(
state={"ticket_text": "Brute-force login attempt from untrusted subnet."},
candidates=TicketRoute,
criteria="Classify incoming ticket to the appropriate response team.",
order_invariant=True # 100% permutation-invariant isolated scoring
)
print(f"Action: {decision.value}") # 'security'
print(f"Confidence: {decision.confidence:.2%}") # '92.40%'
print(f"Abstained: {decision.abstained}") # False
from openjevpro.guard import TypeSafeJevGuardHarness
# Safeguard raw probabilities or neural model outputs via dual-threshold
guard = TypeSafeJevGuardHarness(alpha=1.25, min_confidence=0.72)
raw_probs = {
"lost_or_stolen_card": 0.79,
"card_arrival": 0.18,
"pin_change": 0.01,
"balance": 0.005,
"transfer": 0.005,
"statement": 0.005,
"support": 0.005,
}
decision = guard.evaluate_probabilities(raw_probs)
print(f"Decision: {decision.choice}") # 'UNKNOWN' (safely abstained)
print(f"Tentative: {decision.tentative_choice}") # 'lost_or_stolen_card' (auditable)
print(f"Confidence: {decision.confidence:.2%}") # '71.11%' (< 72.00% threshold)
print(f"Abstained: {decision.is_abstained}") # True
from openjevpro.gateway import HybridJevGateway, CircuitBreaker
from openjevpro.client import OpenJevProClient
from openjevpro.harness import TypeSafeJevEngine
# Tier 1 Local Edge Engine (<20ms, $0 cost) + Tier 2 Commercial Cloud Engine
local_client = OpenJevProClient(base_url="http://localhost:8000/v1")
cloud_engine = TypeSafeJevEngine(api_key="your-typesafe-api-key")
# Circuit breaker trips after 3 consecutive cloud failures, 10s cooldown
gateway = HybridJevGateway(
local_engine=local_client,
cloud_engine=cloud_engine,
local_tau=0.75,
circuit_breaker=CircuitBreaker(failure_threshold=3, recovery_timeout=10.0)
)
decision = gateway.decide_choice(
state={"query": "How do I check my pending credit card balance?"},
candidates=["card_balance", "card_lost", "transfer", "UNKNOWN"]
)
print(f"Decision: {decision.choice}") # 'card_balance'
print(f"Routed Tier: {decision.tier}") # 'Tier1_Local' (60%+ Cloud Cost Saved)
print(f"Degraded: {decision.degraded}") # False (True on cloud outage fallback)
print(f"Cost Units: {decision.cost_units}") # 0
from openjevpro.client import OpenJevProClient
from openjevpro.flywheel import DeliberativeDecisionFlywheel
# Deliberative Decision Flywheel: Explore First with System 2, Crystallize into System 1
client = OpenJevProClient(base_url="http://localhost:8000/v1")
flywheel = client.create_decision_flywheel(auto_crystallize=True)
# 1. Cold start / ambiguous query escalates to System 2 thinking & crystallizes boundary rules
decision = flywheel.evaluate_choice(
state={"query": "card not received, suspect intercepted by someone"},
candidates=["card_arrival", "lost_or_stolen_card", "change_pin"],
criteria={"card_arrival": "Delivery status", "lost_or_stolen_card": "Stolen cards"},
order_invariant=True # 100% permutation-invariant candidate scoring
)
print(f"Decision: {decision.value}") # 'lost_or_stolen_card'
print(f"Escalated to System 2: {decision.escalated}") # True (first encounter)
print(f"Distilled Rule: {decision.crystallization_receipt['discriminative_rule']}")
# 2. Subsequent queries hit System 1 fast path directly (0 LLM overhead, <35ms)
promoted_decision = flywheel.evaluate_choice(
state={"query": "card intercepted in mail"},
candidates=["card_arrival", "lost_or_stolen_card", "change_pin"]
)
print(f"Escalated: {promoted_decision.escalated}") # False (Fast-path promoted!)
print(f"Speedup: Sub-35ms deterministic inference")
Dual-license structure supporting open community research and compliant enterprise scale.
Free for personal learning, academic research, non-profit institutions, and non-commercial development.
Required for production deployments in commercial enterprises, paid SaaS products, or managed API offerings.
Standard JSON prompting autoregressively generates 20-100 tokens, risking schema malformation and uncalibrated probabilities. OpenJevPro extracts raw candidate logprobs at the first token using grammar masking, yielding 100% type adherence, mathematically calibrated confidence, and sub-100ms speeds.
With vLLM or SGLang running Qwen3-4B or Qwen3-30B-A3B on A10G/L4 GPUs, end-to-end time-to-first-token latency typically ranges between 35ms and 90ms, perfectly suited for high-throughput gateway routing.
All primitives include first-class UNKNOWN / HUMAN_REVIEW abstention. Combined with the TypeSafeJevGuardHarness dynamic dual-threshold (tau = max(tau_min, alpha/K)), low-confidence borderline queries are safely intercepted without risking silent misclassification.
OpenJevPro functions not just as an open alternative, but as an indispensable production harness and guard for commercial TypeSafe Jev: 1) It provides TypeSafeJevGuardHarness to eliminate overconfident silent errors on borderline samples, lifting selective precision to 100%; 2) Its HybridJevGateway filters 70%+ of routine high-frequency queries locally, slashing your commercial cloud billing by 60%+; 3) Its circuit-breaker fallback SLA protects downstream agents against cloud timeouts, rate limits, and outages with zero crashes.
In regulated financial and compliance environments, standard multiple-choice prompting suffers from positional bias. OpenJevPro introduces order_invariant=True, which scores each candidate option in an isolated forward pass without competitor options, then normalizes probabilities via commutative softmax. This mathematically guarantees 100% permutation invariance across all N! candidate orderings (delta P < 1e-4). Furthermore, the Deliberative Decision Flywheel couples fast decisions with System 2 thinking models: ambiguous queries trigger deep reasoning traces, which are crystallized into discriminative boundary rules and cached, permanently promoting subsequent queries to the sub-35ms fast path.
Yes. OpenJevPro includes a lightweight deterministic mock simulator (MockClient or OpenJevProClient(mock=True)) for zero-GPU offline development and CI/CD pipelines. You can also run 'python3 -m openjevpro.demo' in any terminal to see complete decision receipts in under 15ms.