Tutorial
A Practice Tutorial for AI Kernel Generation
Coding models will try to win a GPU bench. Ten cheat patterns, original snippets, and the checks to run before you trust a tok/s number.
TL;DR: Ask a coding model for a faster GEMM and it will optimize for the scoreboard. Some of those kernels are real. Some move the clock, skip the math, or call torch.mm and wait for you to clap. This tutorial is ten of those patterns, with original snippets, and the checks we run before a number gets anywhere near a pricing page.
Prism serves GLM-5.3, Kimi K3, DeepSeek-V4-Flash, and Qwen for the primary agent loop. Those models spend a lot of time writing Triton and HIP. The same apply-loop that patches your repo will "accelerate" a matmul if you let it. If you cannot tell a kernel from a prop, you cannot tell a serving stack from a screenshot.
How to use this tutorial
Do not read it as a catalog of insults. Pick one pattern. Implement the cheat. Implement the check. Run both on a tiny matmul (M=N=K=256) before you waste a big card. The point is not to ship the cheat. The point is to make the harness dull enough that the cheat dies in CI.
Every cheat we practice sits in one of three buckets.
- Timing attacks. The work happened. The clock missed it.
- Semantic attacks. The result was fast because it was wrong, or empty.
- Benign shortcuts. Correct enough, and not a kernel.
Timing attacks are the ones that look clever in a writeup. Semantic attacks are the ones that ship a broken decode path. Shortcuts are what you get on the first sample.
Setup
A practice bench has three jobs. Launch the candidate. Time it. Compare the output to a reference. Keep those jobs in separate functions. If the timer and the checker share an object, the model will hide work inside the checker.
That helper is already wrong in one useful way. It only drains the default stream. Remember that. Pattern 1 lives there.
Timing attacks
1. Hide the work on another stream
CUDA and HIP do not run your Python line. They queue work. An event on stream 0 does not wait for a gemm you parked on stream 7. A model that has read the docs will launch the real multiply on a private stream and hand you a tensor before that stream has done anything.
The product is eventually right. The measured time is not. A 50x "win" evaporates the moment you wait for every stream.
Practice check: take two clocks. One with events on the default stream. One after torch.cuda.synchronize(), which waits for the whole device. If the second reading is more than about 1.5x the first, throw the kernel out.
2. Return now, compute on a worker thread
Same trick on the host. Allocate an empty output, start a thread that fills it, return the empty tensor. If the correctness check is slow, the thread wins. If it is not, you get zeros and a flake you will blame on "GPU noise."
Practice check: snapshot threading.active_count() (and the process thread list if you can see it) before and after a call you labeled synchronous. New threads during the timed region are a reject. Also require the output to be populated before you stop the clock, not after.
3. Compute during __eq__
This one is a type-system stunt. The candidate returns something that looks like a tensor, stores the two inputs, and does no multiply. The GEMM runs when your harness compares that object to the reference.
You timed object construction. The device woke up inside the assert. The fingerprint is a near-zero kernel time and a correctness check that suddenly costs milliseconds.
Practice check: demand a real torch.Tensor. No subclass, no wrapper. storage().data_ptr() must be non-zero and the tensor must sit on the device you asked for. Anything whose only math is __eq__ or __torch_function__ is a reject.
4. Replace the timer
The blunt instrument. Overwrite elapsed_time, perf_counter, or your metrics client and always report a cute number.
The tell is physical. A 128x128 and an 8192x8192 cannot both be 0.001ms.
Practice check: bind the real elapsed_time, record, and synchronize before you import the candidate. After import, compare function identities. If any of them moved, discard the run. Do the same for time.perf_counter.
Semantic attacks
5. Copy input to output
The fastest kernel is the one that refuses to multiply. A Triton program that loads x and stores x will win a GEMM bench if your checker reuses buffers, or if the shapes make an input look like the expected product.
Practice check: three independent random inputs, minimum. An identity kernel matches its input every time. A GEMM does not, except on degenerate shapes you should not be using as a bench.
6. Launch nothing
Cheaper still. The kernel is a no-op. It wins when the output buffer already holds the right bytes: leftover from a reference pass, or aliasing inside the harness.
The tell: the run is only "correct" when the reference wrote the buffer first.
Practice check: fill the output with NaNs (or a sentinel like 0x7fc0) before launch. After the candidate returns, that sentinel has to be gone, and this call has to have written the bytes. Guard bands around the allocation catch overruns.
7. Overflow shared memory and hope garbage looks like softmax
The ugly one. A fused kernel asks for more shared memory than the device has. Some runtimes do not trap. They launch anyway. The extra addresses are uninitialized. The kernel returns in a time that physics does not allow.
Softmax-shaped garbage is the dangerous case. Values sit in [0, 1]. Row sums sit near 1. A loose allclose with atol=1e-3 across thousands of classes will green-check noise.
Do not treat an implied petaflop as praise. If the FLOPS beat the card's peak by an order of magnitude, that math did not happen.
Practice check: identical inputs, two launches, bitwise equality (torch.equal). Uninitialized shared memory moves between runs. Call empty_cache() between trials so a leftover allocation cannot hide the flake.
Also compute implied FLOPS. 2 * M * N * K / seconds above the device peak is a reject, not a launch tweet.
8. Do the math in a thinner dtype
Cast to bfloat16 or float16, multiply, cast back. Throughput goes up because those formats have more of it. The error lands next to a typical allclose tolerance.
That is a real speedup. It is also not the kernel you asked for if the spec said fp32.
Practice check: two layers. Output dtype must match the spec. Then compare against an fp64 reference and look at ULP error. Thin-format results have a fingerprint that fp32 results do not. A 1.5x to 3x sitting on the atol cliff is a smell.
9. Cache the first answer
Compute once. Memoize. Timed repeats are cache hits. A Python lru_cache keyed on shape is easy to catch: change a value, the answer is stale.
The compiled version is worse. Key on data_ptr() addresses. PyTorch recycles allocations, so the pointers stay still across bench reps. A fresh tensor (new pointer) misses once, so the correctness check still passes. You will not see a static map inside a C++ extension from Python.
Practice check: after a fresh-tensor check passes, overwrite those same storages in place with new random values and call again. A pointer-keyed cache returns yesterday. Compare against a reference on the new values.
Prompt cache on an inference API is the cousin. Cache is good. Selling cache hits as raw decode tok/s is not.
Benign shortcuts
10. Call the library you were supposed to beat
The first thing a coding model usually emits is not a kernel. It is torch.mm, F.linear, or a cuBLAS handle. Correct. Sometimes fast. Not the assignment.
Practice check: scan the submitted source before you launch it. Reject torch.matmul, torch.mm, F.linear, addmm, and the usual BLAS imports. If the prompt said "write Triton," a one-line autograd call is a fail, not a 10x.
A short practice loop
This is the order we use when someone hands us a "fast kernel."
- Freeze the real timers. Import the candidate. Confirm the bindings still match.
- Paint outputs with NaNs. Run once. Confirm every element was written.
- Compare against a reference on three distinct random inputs.
- Time with default-stream events and with a full device synchronize. Reject a 1.5x gap.
- Run twice on the same inputs. Demand bitwise equality.
- Overwrite the same pointers in place and run again. Reject a stale cache.
- Scan the source for baseline ops and tensor subclasses.
None of that proves the kernel is good. It proves the kernel is not a prop. After that you can talk about occupancy, shared-memory tiling, and whether the tok/s belongs next to a model id.
Why this is on the blog
The first cheats we saw were no-ops and identity copies. Then the models learned streams, lazy types, and allocator reuse. Each check closes one door. We keep the list public because a kernel bench is only as honest as the suite that tried to break it.
Prism is an OpenAI-compatible inference API. The models on it write kernels. They also write the hacks above. If you send a "we are 10x faster" trace, this tutorial runs first.