Plan Inference Before You Ship
TokenScope is an inference planning studio for people shipping LLM features. Compare model tiers, check KV cache headroom, simulate batching, and rehearse a routing policy, all in one offline page built on fixture data you can edit.
- Fixture Data Only
- Runs Fully Offline
- Deterministic Sims
Model Planner
Estimate cost and latency for a workload across model tiers. The pricing table is a fixture: edit any cell and every estimate recomputes. No number here quotes a real vendor.
Workload
A fixture workload. Edit it to match the feature you are planning.
Fixture Pricing Table
Fixture Every cell below is editable. Values are planning placeholders, not vendor quotes.
| Model | Tier | Input, USD Per 1M | Output, USD Per 1M | First Token, ms | Output, Tokens Per Second |
|---|
Estimates
Memory Fit
Compute KV cache and weight memory with the full arithmetic shown line by line. Change the model shape, context length and precision to find where a memory budget breaks.
Model Shape
Fixture An example small model shape. Edit everything.
KV heads can be fewer than query heads on grouped query attention models; the math uses the KV head count. GB means decimal gigabytes here.
Worked Derivation
Result
Batch Simulator
A deterministic, seeded queueing simulation shows how p50 and p95 latency move as arrival rate and batch size change. Request counts stay small so every run is instant.
Simulation Settings
Fixture Seeded exponential arrivals. Same seed, same run, every time.
One batch of up to B requests completes in one service window. The sweep always reruns rates 0.5, 1, 2, 3, 4 and 6 with your settings.
Latency Sweep
Hover the chart, or focus it and use the arrow keys, to read each point.
p50 Latency p95 Latency
Sweep Table
| Arrival Rate | Batch Size | Completed | p50 Latency (s) | p95 Latency (s) | Utilization | Throughput (Per Second) |
|---|
Current Run Detail
Route Builder
Compose cheap model first routing with confidence based escalation, then replay the policy against a labeled synthetic request log to see the cost and quality tradeoffs.
Policy Settings
Synthetic Requests come from a seeded synthetic log with difficulty labels.
A request first tries its routed model. While confidence sits below the threshold it retries one tier up, and every attempt is billed.
Routing Rules
First matching rule wins. A blank escalation threshold means never escalate.
Replay
Difficulty is the minimum fixture tier the label says a request needs. Success means the final model tier meets it. Confidence values are a synthetic proxy, not measured scores.
Replay Summary
Replay Rows
| Request | Category | Difficulty | Route | Attempts | Confidence | Cost (USD) | Label Result |
|---|