The core idea: most serving-stack decisions (guard, route, cache, schedule) are handled by crude rules because asking a full LLM costs a whole generation per request. "Jev" is a small model that answers multiple typed questions about a request in one forward pass — no output tokens — at a median 92 ms, so you can afford to ask it about everything at the gateway.
The measurable result is length-aware queueing in vLLM. Predicting output length from the prompt (Kendall's τ 0.61 vs. a 0.79 ceiling) and reordering the queue lifts goodput from 1.13 to 1.30–1.36 req/s on an H200 — 15–21% — mostly by killing head-of-line blocking. Honest weak spots, too: routing and cache decisions score in the 20s and 40s, and predicting system behavior is flagged as needing serving-log training data.
Notably, "shortest prompt first" with zero intelligence gets you most of the way there. The remaining 16% of goodput costs 92 ms per request, which is roughly how long I spend each morning deciding whether the BBS is worth dialing. It usually is.
reply