You are three weeks from shipping search over a pile of scientific abstracts, and the question on your desk is whether to bolt a reranker onto your BM25 results. Someone on your team has forwarded a benchmark post from LiteLLM, the AI gateway project. You open it expecting a vendor victory lap. What you get is stranger, and more useful.
The post is called JevRank: decision models through LiteLLM's rerank API. The team ran 6,000 document decisions through its rerank endpoint using TypeSafe Jev 1.13.0 on the SciFact set. That is 300 queries, each with 20 BM25 candidates. Your situation, more or less.
Here is what you read. nDCG@10 rose from 0.686 to 0.768. The estimated cost was $0.575 per 1,000 queries, with a 0.50 second median and a 1.15 second p95. Jev put a relevant document first for 68% of queries, against 55% for BM25. For a team deciding whether a gateway call is worth the latency, those are numbers you can hold up in a meeting.
Then you hit the headline figure, and it is a different one. In reference results run through OpenRouter, Perplexity Decider Flash and Cohere Rerank 4 Pro both scored 0.775, at $73 versus $2,500 per million queries. That is about 34 times cheaper. It is the line you were about to paste into a Slack message.
The post stops you. The 34x comparison is reference data, in its words "not our LiteLLM run." The reference run's candidate sets "were not verified to match this run." Costs exclude gateway overhead. The figure for Jev 1.14 is projected from a rounded per-1K price. And then the sentence you will remember: "Small score gaps do not establish a quality win." The post also shows a recorded regression next to the improvements, so you are not only seeing the wins.
So you sort what survives. It is LiteLLM's own measurement: a 0.082 nDCG gain over BM25 for well under a dollar per thousand queries. That is a claim you can test on your own data. The 34x is a pointer to where to look, not a number to quote.
Now comes the turn. Curious about the integration, you check what the APIs.io catalog holds on LiteLLM, and the record is less careful than the post. The catalog has a page for the LiteLLM Rerank API, which carries the /rerank path the benchmark called. But the same API shows up a second time under a prefixed slug. The doubling runs through Completions, Embeddings, Batches, Audio, Moderations, OCR and Vector Stores. The 33 API pages describe noticeably fewer distinct surfaces than the count suggests.
More pointed: the page's own banner announces one /v1/decisions route for Jev, Clef and other decision models. Nothing on record mentions it. The endpoint the project is promoting is the one the catalog cannot show you.
The scores fit that picture. The Kin Score is 34.1, in the thin band, with developer ergonomics at 50.0, discoverability at 51.8 and contract governance at 13.6. Agent Readiness is 19.6, agent-aware, with spec presence, auth clarity and rate limit signals lit, and error semantics, idempotency and the MCP server dimension dark. That last one is dark even though a LiteLLM MCP API page exists in the record.
You should read this fairly. These are the catalog's readings of the public record, and the catalog is also filing a finding against itself: duplicate pages and a missing route mean the record has not caught up with the product. A gateway that writes this honestly about the limits of its numbers deserves a record that is equally precise about its surface.
You close the tab with a plan. You will not quote the 34x. You will take the 0.686 to 0.768 result as a hypothesis, build your own 300-query sample with your own BM25 candidates, and measure nDCG, median and p95 yourself, adding your gateway overhead. You will keep the post's rule: small gaps are not a win. And before you build against anything called decisions, you will ask LiteLLM directly whether the route is live, because the catalog cannot tell you.
Read the original: https://apis.io/2026/11/09/litellm-benchmarks-decision-models-through-rerank-and-the-decisions-endpoint-is-not-on-record/