The marginal cost of intelligence is trending to zero

TL;DR: Tokens used to be expensive because the cards were scarce. The cards that actually make a token cheap have been in the same racks for a while. What still costs money is standing the newest open MoE up on that silicon before the next checkpoint ships. That bill is falling because the same coding agents we serve are getting better at the unglamorous serving work.

Flagship NVIDIA SKUs are not the supply that matches the demand. Frontier open models land almost every other week. GLM-5.3, Kimi K3, DeepSeek-V4-Flash, Qwen, plus whatever closed name your agent already calls. Apply loops eat them. There are not enough B-class cards to serve that traffic at a price a tight loop can live with, so the lease climbs, and the token price climbs with it.

The other rack is not a secret. AMD Instinct parts compete on the spec sheet and cost a lot less per GPU. A MI355X-class card versus a B300-class card is the same shape of machine at a fraction of the lease. If intelligence is a token, and a token is a watt-hour on a card, the marginal cost should already be sitting on the cheaper watt-hour.

It does not, on day zero. NVIDIA's serving stack has images, kernels, and quantization paths for whatever landed this Tuesday. On the MI355X / ROCm stack you are lucky if an image boots the new MoE at all. The weights are public. The recipe is not. Without that day-0 support, standing up a frontier checkpoint can take weeks of engineering and compute. By then the next checkpoint is out, and the cheap card is behind again.

That gap is a support gap. Support is work a coding model can do.

The mix that actually pays

A coding agent is not a chatbot. It prefills a repo-shaped prompt, decodes a patch, calls a tool, prefills again. A useful mix looks like 20k tokens in, 1k out, with a high prefix-cache hit rate. That turn is mostly prefill. A single-stream tok/s screenshot on a quiet card does not describe it.

We watch two clocks.

• Single-stream decode on a long coding prompt, the Artificial Analysis style of number, because that is what one agent feels.
• Aggregate tokens per node at a defined knee (TTFT still under a few seconds), because that is what ten agents sharing a box feel.

Peak tok/s on unused silicon is a prop. Tokens per dollar at the knee is the product.

How the model actually gets up

First, pick a quantization that stays honest. Start from the bf16 checkpoint. Official FP8 builds are a baseline. Lower-bit formats (MXFP4 and friends) only win if they stay lossless on the evals that matter for agents: math, tool use, graduate-level reasoning. If GSM8K and GPQA move by noise and tau2 holds or improves, you saved memory. If the agent starts dropping tool calls, you did not save money. You shipped a worse loop.

The recipe is mechanical. Quantize with the vendor tool (Quark on AMD). Keep the shared experts the engine expects in bf16. Write down the layer names. Do not assume the official FP8 path is the cheap path. It is the supported path.

Second, pick an engine that actually uses those weights. You usually have three candidates. One will load the quantized MoE and then ignore the format, so the MXFP4 file buys you nothing. One will go incoherent past a few tens of thousands of tokens. Take the one that stays coherent and hits the quantized matmuls. On this generation that has been SGLang more often than vLLM. The winner will change. The test will not.

Third, turn on speculative decode. This is where most "we run it on AMD" demos die.

The draft head keeps a shared expert in bf16, like the rest of the stack. The quantizer records that expert under the main decoder prefix (model.layers.N.mlp.shared_experts.*). The engine looks up the MTP module under a different prefix (model.decoder.*). The lookup misses, builds a 4-bit slot, and then tries to load a full-width bf16 weight into it. Shape mismatch. Process dies.

The fix is boring. Copy the "leave this in bf16" layer list a second time under the names the engine actually uses.

That unblocks MTP. Single-stream decode on a long prompt stops looking like a tax. Expect something in the neighborhood of a 3x if the draft head was dead before. That is a serving fact, not a Prism tok/s claim.

Deep draft configs (the 5/1/6 style the model card suggests) hit a second bug. The fused multi-step metadata kernel still writes a CUDA header with no ROCm branch. One guard.

Two small patches. Then speculative decode actually runs. Pair it with the usual serving knobs (--kv-cache-dtype fp8_e4m3, fused allreduce) and the single-stream number becomes a number you can show a customer without lying about the mix.

Prefill is the other half

Decode work is necessary and not sufficient. At 20k in and a 60% cache hit, the node is prefill-bound.

A layout that wins the single-stream screenshot (tensor-parallel across the whole box) loses when many coding agents share the node. On one generation of this work, TP8 was the decode winner and a dog for aggregate throughput. Splitting to TP4 times DP2 moved the same MXFP4 checkpoint a long way on the 20k-in mix. The expensive card still wins raw tok/s. The cheaper card still wins tokens per dollar, which is the race that sets the bill.

A big reason prefill looked bad: the image's MoE kernel picker silently fell back to a slow heuristic for the fp4 shapes. Tuned configs shipped for the a8w8 / fp8 path. GLM's fp4 shapes (model_dim 6144, moe_inter 2048, E=256, topk=8) were not in the table. Picking the right kernel for those shapes is the difference between "the cheap card is slow" and "the cheap card is most of the expensive card at less than half the lease."

None of that required a custom kernel this time. Framework bugs, a skip-list typo, one ifdef, and a MoE config the image did not ship. Support, not software.

Why the bill keeps falling

Most of this traffic still runs on a single node. You do not need a multi-node paper to make the token cheaper. You need the new open checkpoint on the cheap card this week, not next month.

The CUDA moat was never magic silicon. It was day-0 images, name maps, and kernel tables. That is exactly the work coding agents got good at. Every time that loop gets faster, the lease on the expensive SKU explains less of the invoice.

Prism serves GLM-5.3, Kimi K3, DeepSeek-V4-Flash, and Qwen through OpenAI and Anthropic compatible APIs. Point your client at https://api.prisminference.com/v1 for Chat Completions. Customers do not want a lecture about Blackwell lead times. They want the model id to stay cheap when ten agents are applying patches at once.

The model is open. The card is cheaper than the sticker implies. The remaining cost is the recipe. That recipe is trending toward free, which is another way of saying the marginal cost of intelligence is trending to zero.