One day a scan took almost seven minutes. Nothing failed, nothing alerted, and the report looked fine. That made it worth looking into.

What happened #

The scan had raised 41 findings, and the code asked for every interpretation in one reply, as a single JSON object.

The reply outgrew the call's 8,192-token output cap. The model stopped mid-object, the JSON didn't parse, and our LLM layer did what it was built to do with an invalid reply: retry on the fallback provider. That happened twice.

The scan took 407 seconds. Most of it went on generating long replies that got cut off and then thrown away, and a different model than intended wrote the final report.

All of this was silent:

• No exception reached the user, because the fallback worked.
• The report looked normal, because the fallback model was good enough.
• The only symptom was latency, and scans run in the background where nobody watches latency.

Why it slipped past testing #

The output size grows with the number of findings. Test brands raised 6 to 12 findings, which fit easily. A real brand with a lot going wrong raised 41. The prompt hadn't changed, but the output was several times bigger.

Fix 1: treat "stopped at the cap" as its own failure #

A reply cut off by the output cap is not the same as malformed JSON. The API tells you why it stopped (stop_reason: "max_tokens"), so:

• It's logged as llm_output_truncated, with its own count on the dashboard.
• Retrying the same request on the same provider is pointless, because it will hit the same cap. For batched calls, a cut-off reply goes straight to the fallback instead of a live retry that is bound to fail the same way.

Fix 2: split the work into groups #

Findings are now interpreted in groups of 12, all sent at once:

• Each group's reply stays well inside the cap, however many findings a scan raises.
• Each group gets its own number validation and retry, so one bad group doesn't redo the other three.
• Every group uses the same skill set, so the system prompts are identical and share the prompt cache.

The same scan then ran with no truncation and no fallback in 253 seconds instead of 407.

Then I split it too far #

Live chat turns had the same shape on a smaller scale: one call explaining six findings took 34 to 36 seconds, the slowest step in a turn. The obvious next step was one call per finding, all at once, so the step takes only as long as the slowest finding.

It was faster, and the answers got worse. Explaining findings together had been doing work I hadn't noticed:

• An action stopped naming the ad set that another finding said to scale.
• A gap between two data sources stopped pointing at the tracking problem another finding had already found.
• Two findings gave contradicting causes for the same drop.
• A finding recommended running an analysis that a sibling finding already was.

I tried giving each call every finding as shared, cached context, with a short part naming the one finding to explain. That brought back most of the links, but it added prompt complexity and new edge cases (citing another finding's numbers, for one). The same day, I reverted to groups.

What I took from it #

• Output size is a function of your input. If a reply lists one item per input, its worst case is set by your largest customer, not your test fixtures.
• Check the stop reason before you parse. Cut-off JSON is a capacity problem, not a formatting one, and needs a different response.
• Fallbacks hide problems. Record when a fallback fires, and alert on it.
• Parallel calls lose shared reasoning. Splitting a prompt splits what the model can connect. Make each batch as small as the limit requires, and no smaller.

Related: Stopping an LLM from making up numbers, on the validation each of these groups runs.

Click the game first so it receives key presses. Left mouse paints, right mouse erases, 1–7 picks an element, the mouse wheel sets the brush size, Space pauses and C clears. It's built for a mouse. Open it full screen if the frame is too small.

Try this: paint a block of wood, then paint a little fire just under it.

None of what you see is scripted. There's no physics engine, no fluid solver and no fire effect. Each cell follows a few rules about its immediate neighbors, and the sloped dunes, levelling lakes and spreading fires all come from those rules.

The world is a grid of bytes #

The world is a fixed 320×180 grid, scaled up 4× on screen. Each cell is five bytes:

struct Cell {
    ElementId type;         // sand, water, stone, ...
    uint8_t shade;          // random color variation, rolled when the particle spawns
    uint8_t life;           // remaining frames, for fire and smoke
    uint8_t lastUpdate;     // frame stamp: already moved this frame?
    uint8_t lastDisplaced;  // frame stamp: already pushed aside this frame?
};

The grid holds 57,600 cells, about 280 KB. Every frame updates the whole grid in place, cell by cell. There's no second buffer and nothing runs in parallel.

One pass per frame, bottom to top #

Each frame scans the rows from the bottom up. This matters for anything that falls. If you scanned top-down, a grain moved from row 10 to row 11 would be visited again on row 11 and fall a second time, then a third, reaching the floor in one frame. Going bottom-up, the grain lands in a row that has already been scanned.

Gases rise, though, so for them bottom-up is the wrong order. That's what lastUpdate is for. Before a particle moves, it's stamped with the current frame number, and the stamp moves with it. Any cell already stamped with this frame is skipped. So every particle moves at most once per frame, whichever way it's going.

Within a row, the scan direction is picked at random every frame. With a fixed left-to-right scan, particles on the left would always move first and take the free cells. Sand piles would lean and water would drift to one side. A random direction removes that bias.

Four movement rules #

Every element belongs to one of four movement classes, and the simulation has exactly one rule for each:

• None (stone, wood): never moves.
• Powder (sand): move down if possible; otherwise try down-left or down-right, in random order. That's all it takes to make sloped piles: a grain landing on a peak rolls off to one side until it finds a gap, so piles settle at a natural angle.
• Liquid (water): the powder rule, and if all three cells below are blocked, flow sideways up to dispersion cells along the row (5 for water). This sideways flow levels a pile of water into a flat surface. Without it, water would just be wet sand.
• Gas (fire, smoke): the liquid rule flipped upside down, plus a random wobble of one cell left or right on each step up. The wobble is why smoke billows and flames flicker instead of rising in straight columns.

Density, and a bug that teleported water #

A particle can move into an empty cell, or swap places with a lighter liquid or gas. Each element has a density (sand 150, water 100, smoke 2, fire 1), so sand sinks through water, and water and sand both fall through smoke. Solids never get pushed, and gases only rise into empty space.

The simple version of this rule has a bug. Pour a steady stream of sand into a lake and water gets pulled up through the stream to the top in a single frame. The grain at the bottom swapped with the water below it. Then the grain above it swapped with that same water cell, now one row higher. Then the next grain did the same, all the way up. The water never got updated; it was carried along by swaps.

The fix is the second frame stamp, lastDisplaced. A particle that gets pushed aside is stamped, and a stamped particle can't be pushed again until the next frame. Now the water moves up one cell per frame, as you'd expect.

Fire is three elements #

Fire is the most interesting behavior here. It's built from three generic properties that any element can use:

• Lifetime. Fire and smoke are given a random lifespan when they spawn (15–45 frames for fire). When it runs out, they turn into their decay product: fire becomes smoke, and smoke becomes empty space.
• Burning. Each frame, a burning cell checks its 8 neighbors. Each flammable neighbor has a small chance of catching fire (wood: 8 out of 256 per frame). If any neighbor puts out fire (water does), the burning cell decays immediately.
• Emission. Some cells randomly spawn another element in the cell just above them.

Wood doesn't turn into fire when it catches. It turns into ember: an element that stays in place, sets its neighbors alight, gives off flames upward, and after 60–160 frames turns into smoke. So a burning tree glows, throws flames and collapses into smoke, and the fire moves slowly through the wood. Wood's low flammability is what keeps the spread slow. Raise it and a forest goes up in a few frames.

Elements are data #

Because those behaviors are generic, an element is a row in a table, not code:

{.name = "Wood",
 .color = {115, 74, 42, 255},
 .colorJitter = 16,
 .density = 255,
 .flammability = 8,
 .burnsInto = ElementId::Ember},

To add oil, you add one enum value and one table row: a liquid with density 80 (so it floats on water) and some flammability. The palette, number keys and simulation all pick it up automatically, and a static_assert fails the build if the table and the enum get out of sync.

Drawing it #

Rendering is the cheapest part. Each frame fills a 320×180 pixel buffer with one color per cell, uploads it as a single texture, and draws that texture scaled up with nearest-neighbor filtering. That's one texture upload and one draw call.

The color comes from the element's base color, brightened or darkened by the cell's shade byte. Shade is rolled once when a particle spawns, so a sand pile looks grainy and the grains keep their colors as they slide. Fire and embers instead get a new shade every frame, which makes them flicker without any animation code.

Running it in a browser #

The web build is the same C++ code, compiled to WebAssembly with Emscripten. Only one change was needed: browsers don't allow a blocking while loop, so the loop body moved into a function the browser calls once per frame. The download is about 380 KB, and the code and build instructions are on GitHub.

The worst thing it can do is write a wrong number confidently. A media buyer who sees one wrong spend figure stops trusting every other figure on the screen. So the system follows one rule: code computes every number, the LLM only interprets.

Numbers are facts, not text #

Every number the analyst can show comes from code. Plain functions compute spend, conversion rate, the gap between two sources and so on, and each one produces a fact: a key, a value, a format (money, percent, count) and the step and source it came from.

The LLM never sees a number it is allowed to type. It writes placeholders instead:

Spend rose to {{fact:f_3a9c.spend}} while purchases fell to {{fact:f_3a9c.purchases}}.

Code swaps each placeholder for the fact's value, formatted in the brand's reporting currency. So numbers always render the same way, the LLM can't mistype them, and every value on screen links back to the step that produced it.

Validating every number #

After each LLM call, a validator reads the text:

• A placeholder with an unknown fact key fails.
• Any literal digit left in the text fails. The only exceptions are a small allowlist: list markers, date strings copied from the analysis window, and attribution labels like "7-day click".
• On failure, the model gets one retry that names the exact keys or tokens that broke the rule.
• If the retry fails too, the offending sentence is dropped and a warning is shown. A shorter answer is better than a wrong one.

The eval suite checks for zero ungrounded numbers on every recorded question, not just for the "right" answer.

The audit: five places numbers still got through #

Placeholders covered the obvious case. When I audited every LLM call in the pipeline, five places still let an LLM-written or LLM-chosen number through:

1. Dates and period lengths. The validator had exempted ISO dates, month names and "N days". So the model could write "over the last 45 days" when the window was 28. Now the answer step receives every window the turn actually used, and any date or duration outside them is a violation.
2. Named periods. The planner was turning "last month" or "Q3" into date ranges itself, which meant the LLM did date arithmetic. Common phrases ("last 30 days", "in September", "year to date", "yesterday") are now parsed in code. Only phrases that are ambiguous or match nothing go to the planner.
3. Conceptual answers. For "what's a good CTR?" there's no data, so the model reached for benchmarks it half-remembered. Those answers may now only repeat numbers the user wrote, and the prompt asks for explanations in words.
4. Follow-up suggestions. "Want me to look at why CPA is up 40%?" is still a number. Suggestions that state a figure without a fact behind it are dropped.
5. Planner filters. "Show me campaigns with ROAS below 2" is fine. But the planner also invented thresholds for vague wording like "underperforming campaigns". A gt/lt filter is now kept only if the user stated that number; otherwise the planner sorts and limits instead.

The pattern: the LLM chose a number in places that didn't look like "writing a number", such as a date range, a filter or a follow-up question.

Small rules that saved retries #

Strict validation costs retries, and each retry is another full LLM call. A few fixes in code removed the most common ones:

• Currency before a placeholder. The model kept writing INR {{fact:spend}} for a fact that already renders as "INR 60K". Code now strips the typed currency when the fact brings its own, rather than failing the whole draft.
• Figures from other findings. Each finding may name other findings by title. But it may only quote its own fact labels and text values, so it can't repeat another finding's "over 90%" outside that finding's context.
• Streaming. Answers now stream to the UI sentence by sentence. A sentence already on screen can't be rewritten, so a streamed sentence that fails validation is dropped instead of retried.

What it costs #

The prose is stiffer. Without the freedom to say "nearly double" or "about a third", the model writes more literal sentences. Some sentences get dropped. Retries add latency.

I'll take that trade every time. Users have to trust the numbers before the interpretation is worth anything to them.

I've released the core of this as an open-source Python library, factguard: fact placeholders, the validator with its date windows, the single retry and dropping the sentence.

Related: When an LLM hits its output cap, on what happened when one of these interpretation calls outgrew its token limit.