What a Tokvera report shows

Efficiency Score

D60.8 of 100 points; 55 of 60 requests succeeded.

Provisional scoring. The weights and thresholds are provisional: they have not yet been calibrated against real pilot scans and may change before scoring 1.0.0.

• Capped at D: more than 5% of requests failed
• Capped at C: time to first token collapses under load

Findings, ranked

1. [fail] Time to first token collapses under load

   Evidence: p99 201 ms at concurrency 16 against 11 ms at concurrency 1

   Why it matters: The tail grows much faster than the load: requests are queueing behind each other, so the slowest users see a multiple of the idle latency at the concurrency your traffic reaches.

   Fix: Find the saturation point in the level table and keep traffic below it: raise the server's batch and concurrent-sequence limits if memory allows, add replicas behind the endpoint, or put an admission limit in front so excess requests fail fast instead of queueing.

2. [fail] Requests are failing

   Evidence: 5 of 60 requests failed (HTTP 429 ×5)

   Why it matters: Every failed request is either a user-visible error or a retry that costs latency and money. The status breakdown in the evidence says whether it is rate limiting, timeouts or server errors.

   Fix: For 429s, raise the rate limit or add client-side retry with exponential backoff and jitter. For timeouts, compare the request timeout with the measured request duration. For 5xx, read the server logs at the times of the scan.

3. [fail] Many prompts repeat exactly

   Evidence: 16 of 40 prompts in the demo sample repeat an earlier one exactly

   Why it matters: If this sample is what reaches the model, every repeat is a paid call whose answer was already computed once.

   Fix: Put an exact-match response cache (keyed on the normalized prompt and model parameters, with a TTL) in front of the model for the repeating routes, and measure the hit rate before considering a semantic cache.

4. [warn] No Prometheus metrics at /metrics

   Evidence: no Prometheus exposition answered at /metrics on the target's origin

   Why it matters: Queue depth, batch size and cache usage are what explain a latency problem. Without them, a slowdown can be seen but not diagnosed.

   Fix: Enable the server's metrics endpoint (llama.cpp needs `--metrics`; vLLM and TGI serve /metrics by default) and scrape it. If the endpoint sits behind a gateway, expose the metrics there.

Latency by concurrency

Cost: not computed: the rate sheet has no rate for self_hosted/tokvera-sim-demo