Know where your VRAM goes.

InferGrid plans open weight LLM serving on NVIDIA GPUs. Move a slider and watch weights, KV cache and free HBM re-flow on every card, with time to first token and inter token latency from the roofline.

Open the workbench Read the physics

Runs entirely in your browser with no key and no server. Every number comes from a datasheet or a published model config, and the arithmetic is shown.

Loading the first scenario.

Capacity workbench

Pick a model, a GPU layout and a load. The diagram, the verdict and both charts update as you move any control.

Workload presets

Scenario

Engine settings and calibration

Efficiency scales the datasheet peaks to what kernels usually reach. The defaults are assumptions. Fit them to a real endpoint with the probe.

Memory per GPU

Roofline

Hover the chart to read the ceiling at any arithmetic intensity. Dots show where prefill and decode land after communication and launch overhead.

First token latency against burst size

Same model and layout on four GPUs. Dashed lines mark a GPU where the layout does not fit in memory. H100 and H200 share Tensor Core rates, so compute bound prefill is identical on both, and H200 pulls ahead only when decode or KV cache is the limit.

All numbers

Show the arithmetic

Route a workload across pools

The router places each request on the cheapest replica that meets its latency targets, has enough KV blocks and is capable enough for the task. The same requests are then replayed through three simpler policies.

Cluster pools

Rates default to Nebius list prices read on 2026-10-25 (H100 $4.50, H200 $5.40, L40S from $1.55 per GPU hour). Nebius does not list A100, so that rate is yours to enter. Check current prices before budgeting.

Workload

Result

Pool load

Every decision

Deployment drawer

Copy launch files built from the same engine. Parallelism, memory utilization, max length, max sequences and quantization flags come straight from the layout above.

    Endpoint probe

    Optional. Point InferGrid at a Nebius AI Studio key or a local vLLM server and compare a streamed completion with the roofline for the scenario above. Offline replay needs nothing.

    Held in this tab only and sent only to the base URL above. It is never stored, logged or placed in a link.

    Browsers call the endpoint directly, so it must allow cross origin requests. Nebius answered a keyless preflight from another origin with an open allow-origin header on 2026-10-25, and vLLM allows all origins by default. Some browsers block plain http endpoints from an https page.

    How the numbers are made

    Everything on this page is an analytical model. Nothing here was measured on a cluster, and no benchmark result is claimed. The model uses published datasheet bandwidth and FLOPS, published model configs, and a few labelled assumptions.

    Memory

    KV bytes per token = 2 x L x n_kv x d_head x bytes
per GPU = 2 x ceil(L / PP) x ceil(n_kv / TP) x d_head x bytes
weights = non-embedding params x bytes + embeddings x 2
budget = HBM x gpu-memory-utilization
KV pool = budget - weights/(TP x PP) - activations - runtime
sequences = floor(pool blocks / blocks per sequence)

    Embeddings and the output head stay at 16 bit when the rest is quantized. Blocks are 16 tokens, as in vLLM. Parameter counts are derived from layers, hidden size, heads and FFN width, then checked against the published totals in the test suite.

    Latency

    decode  = max(bytes / (BW x eff), FLOPs / (peak x eff))
          + 2 x L x all-reduce + launch overhead
prefill = max(weight reads + KV writes, 2 x params x tokens
          + 2 x L x n_q x d_head x prompt^2) + the same overheads
all-reduce = latency + 2 (TP - 1) / TP x bytes / link

    Decode reads every weight once per step, so it is bandwidth bound until the batch is large. Prefill is compute bound. L40S has no NVLink, so tensor parallel pays PCIe all-reduce costs. A100 has no FP8 Tensor Cores, so FP8 only saves memory there.

    GPU specifications

    Throughput is dense Tensor Core throughput. The H100 and H200 pages print only the figures with structured sparsity (1,979 TFLOPS FP16 and 3,958 TFLOPS FP8), so the engine uses half of those. Sources are the NVIDIA datacenter pages for H100, H200, L40S and A100.

    Model architectures

    Shapes come from each model's config.json on Hugging Face (ungated mirrors where the original is gated) and, for Nemotron-4-340B, Table 1 of the technical report.

    What it does not model

    • Mixture of experts routing, speculative decoding, prefix caching and disaggregated prefill.
    • Gemma-2 sliding window layers, which would reduce KV below what is shown.
    • Allocator fragmentation, CUDA graph capture sizes beyond a flat estimate, and NCCL topology effects beyond a single latency and bandwidth per link.
    • Queueing in the workbench. The router adds a simple per replica prefill queue and a batch snapshot at admission.
    • Task quality. The router capability score is a size proxy (log of parameter count, 400B equals 1.0). Replace it with your own eval results.