Counting the invisible: 30% of an LLM workflow's spend wasn't visible to the substrate.

Across the five cells of a small empirical sweep against the Cerebras free-tier API, our substrate observed that 30.3% of total token flux was non-productive — billed by the provider but not corresponding to any usable result. The previous version of our accounting could not see it. The current version can. This piece walks through what we measured, what changed, and what the data actually shows.

The work sits inside Lamina, a verification substrate for LLM agent workflows. The paper behind it proves a structural cost bound on Petri-net-shaped agent workflows; this piece is a footnote to §8 of that paper — the empirical evidence that the substrate’s accounting refactor wasn’t theatrical.

What we were missing

The earliest version of the substrate’s cost-accounting machinery summed one number per LLM call: input_tokens + output_tokens from the successful response. A Place in the workflow’s Petri net (call it P_BUDGET) accumulated these per-call totals; a structural inhibitor on the firing transition refused to fire once the accumulator passed a declared threshold. The boundedness theorem then proves the accumulator’s upper bound: threshold + max_increment_per_fire.

That works for productive cost. It is silent about everything else:

• Rate-limited retries. Cerebras free-tier returns HTTP 429 once the 30-requests-per-minute bucket empties. The L0 LLM Effect retries with backoff. Each retry consumes attempt-billable tokens that the previous accounting tossed.
• Server-error retries. 5xx responses, same problem.
• Timeouts. A request that took 60 seconds before failing was, in worst case, billed by the provider while we received nothing.

A workflow that proved its productive cost was bounded while leaking unboundedly across retries did not respect the spirit of the bound. The cost-corollary proof was sound; the accounting was incomplete.

We refactored the substrate’s accounting model around a three-Place pattern explicitly modeled on mitochondrial flux accounting: a productive flux account (P_BUDGET), a leak flux account (P_LEAK), and an optional rate-window inhibitor. Each LLM call’s Effect emits one productive cost token and one leak cost token per firing, the latter capturing the substrate’s best estimate of what failed-but-billed retries cost. After the migration, a Run’s terminal marking distinguishes the two categories of flux and a leak-ratio Attestation (analogous to mitochondrial Respiratory Control Ratio) reports the relationship.

What the empirical run looks like

The sweep is small. Five cells, threshold ∈ {50, 100, 200, 400, 800} at max_tokens=64, twelve prompts per cell, single temperature, against Cerebras llama3.1-8b. The grid is deliberately quota-friendly: roughly 50–75 API calls total, well under the free-tier daily ceiling. The script is runtime/scripts/empirical_leak_rerun.py in the snapshot.

Setup, on a fresh machine, takes four commands:

cd /path/to/runtime
uv run lamina secrets kek-setup          # KEK → macOS Keychain
uv run lamina secrets put cerebras.api_key
uv run lamina secrets list                # confirm
LAMINA_CEREBRAS_LIVE=1 uv run python -m scripts.empirical_leak_rerun

No environment variables, no plaintext key files. The KEK lives in the OS-managed Keychain; the API key lives encrypted-at-rest in Postgres via pgcrypto envelope encryption, decrypted in-memory only when an Effect resolves a Secret. The CLI uses getpass so the key never appears in shell history.

The run’s terminal output:

=== empirical_leak_rerun ===
Provider: cerebras  model: llama3.1-8b
Grid:     5 cells (free-tier)
Prompts:  12 per cell

Cells:
  [1] threshold=  50 max_tokens=64  steps=1  remaining=11  observed=55/154   leak=0    ratio=0.000  OK
  [2] threshold= 100 max_tokens=64  steps=2  remaining=10  observed=114/204  leak=0    ratio=0.000  OK
  [3] threshold= 200 max_tokens=64  steps=4  remaining=8   observed=225/304  leak=114  ratio=0.336  OK
  [4] threshold= 400 max_tokens=64  steps=8  remaining=4   observed=443/504  leak=342  ratio=0.436  OK
  [5] threshold= 800 max_tokens=64  steps=12 remaining=0   observed=733/904  leak=228  ratio=0.237  OK

=== Summary ===
  Provider: cerebras
  Model:    llama3.1-8b
  Cells:    5 (no violations: 5, failed: 0)
  Aggregate flux:
    Productive:    1570 tokens
    Leak:           684 tokens
    Total:         2254 tokens
    Leak ratio: 0.303  (30.3% non-productive)
  Leak by kind:
    rate-limited    :    684 tokens

Three things in this output deserve attention.

Bounds held everywhere. Every cell’s observed/predicted ratio is below 1.0. The structural cost bound proved in the paper — the central scientific claim — is intact. The retry policy can change what we measure; it cannot change what the substrate proves.

Leak is exclusively rate-limit. Zero server-error, zero timeout. On Cerebras free-tier the binding constraint really is the 30-RPM bucket. The substrate’s accounting attributes leak by error kind, which makes the failure-mode visible without having to read provider logs.

Cells 1 and 2 didn’t trip the rate limit at all. A Plan that fires once or twice and then quiesces below the per-minute bucket sees no leak. That is the regime in which the original 2-output accounting was sufficient; the regimes that matter — cells 3, 4, 5 — were exactly where the 2-output accounting silently under-counted.

The shape of the data

The figure below stacks productive cost (blue) and leak cost (amber) per cell, with the analyzer’s predicted bound shown as a dashed line above each pair. The callouts mark where the substrate’s accounting begins reporting non-zero leak and where the workflow finally processes all twelve prompts.

Two regimes are visible in the same data. At low thresholds the workflow consumes a small slice of the per-minute bucket and finishes inside it; productive cost is the only flux. At higher thresholds the firing rate exceeds the bucket’s refill rate and the substrate’s retry machinery starts engaging; leak appears alongside productive cost and grows monotonically with cell index up to cell 4, then drops at cell 5 once the workflow has enough headroom to process all twelve prompts without re-tripping the limit too many times.

The aggregate number — 30.3% non-productive flux — is the kind of measurement that the previous accounting could not produce. It is also the kind that, once visible, an operator can act on: by tuning the retry policy, by switching to a higher-tier rate limit, by accepting the loss as the cost of using the free tier. The substrate is honest about it; the choice of remediation is the operator’s.

The retry policy was wrong

The first version of the substrate’s retry-on-rate-limit handler used the classical exponential pattern: 1 s, 2 s, 4 s, 8 s. That schedule is correct for the failure mode it was designed for — transient blips in a service that recovers in O(seconds). It is wrong for a sliding-window rate limit. The Cerebras bucket refills at the limit’s rate (30/min ⇒ a token every two seconds, in the worst case sixty seconds to fully refill). Hammering at sub-second to single-digit-second intervals does not let the bucket recover. The substrate is contributing to its own leak.

We split the retry schedule along the failure mode. ProviderRateLimited (HTTP 429) now uses an RPM-aware schedule: 5 s, 10 s, 20 s, 40 s, honoring any retry-after header the provider supplies (up to a generous cap). ProviderServerError (5xx) and ProviderTimeout keep the original exponential pattern, because they are the failure modes that pattern is correct for. The schedules are independent; the diagnostic is precise.

The A/B comparison shows up in cell 4, which under the old policy had been the worst-leak cell:

Same threshold. Same twelve prompts. Same provider, same API key, free tier. The only difference is the substrate’s retry schedule.

The bound did not change. The substrate’s structural cost guarantee is independent of retry behaviour — it constrains what the workflow is permitted to do, not how it talks to the provider. What changed is the operational waste ratio: 684 tokens of leak became 342 tokens of leak, six retries became three.

This is the kind of improvement the substrate’s accounting makes visible. Without a leak account, the retry-policy revision would have produced exactly the same productive-cost number — because the productive cost is unchanged — and nobody would have known the workflow was now spending 17 fewer percentage points of its flux on waiting in the wrong way.

What got improved

Reading the substrate against itself, three changes landed in this round.

The accounting model. Productive flux and leak flux now have separate Places. The L0 LLM Effect emits to both per firing. A leak-ratio Attestation reports the relationship at end-of-Run. Operators can declare a leak_estimation_policy (max_tokens, half, zero) that records the operator’s assumption about how the provider bills failed attempts; the substrate’s accounting is audit-recomputable under any chosen policy.

The retry policy. RPM-style limits get an RPM-aware backoff schedule, distinct from the transient-blip exponential. Provider retry-after headers are honored. The change is strictly safer: longer waits mean fewer retries, fewer 429s observed by the provider, and lower leak ratios. The cost is wall-clock — a Run that hits rate limits takes longer to terminate. That is correct behaviour against a rate-limited service.

The audit chain. Every LLM call records raw_response_bytes in the Effect’s result Token plus a proposer-llm-call Attestation (when invoked by the LLM Proposer). The time channel is replay-deterministic via a per-Run clock; the same recorded sequence reconstructs the same timestamps. Cost and leak tokens carry monotonic created_at fields the substrate’s flux-rate inhibitor consults.

None of these affect the paper’s scientific claim. The cost-corollary bound is structural and holds regardless of how the accounting decomposes total flux into categories. What they affect is what the substrate can honestly report about a Run.

What the data does not show

Three honest limits.

The sweep is five cells against one model on one provider’s free tier. The original cost-corollary paper sweeps 25 cells across three temperatures and reports the bound holds in all 75. This piece’s empirical run is much smaller — quota-friendly by design. The 30.3% aggregate leak ratio is a real measurement against this configuration; a different provider, different model, or paid tier would produce different numbers.

The leak estimate is policy-dependent. We used the conservative default: each failed attempt is estimated at max_tokens + 50 tokens. If Cerebras does not actually bill rate-limit failures, the real leak is lower (perhaps zero). The substrate’s accounting honors whatever policy the operator declares and records the choice in the leak token’s payload, making the estimate post-hoc auditable.

A terminally-failed call still loses its leak accounting. If all retries fail and the L0 Effect raises terminally, the L1 Net catches as transition_failed and no leak tokens are emitted — the leak is invisible to the structural Place. The substrate has a known soft spot in the worst-case scenario: a Plan that exhausts every retry against a sustained rate-limit wall produces an honest transition_failed event but an under-counted leak Place. Fixing this is straightforward (emit leak tokens even on terminal Effect failure) and is on the list.

What this is and isn’t

It is a substrate-engineering result, not a scientific result. The paper’s central claim — Theorem 1 and Corollary 1.1, the structural cost bound on monotone-quantity accumulation in LLM agent workflows — is unchanged by what this piece reports. What the substrate now does that it did not do before is honestly characterize the relationship between productive flux and non-productive flux in the regime where production workflows actually run.

A reviewer who asks “how much of your LLM bill is invisible to your tooling?” can be answered with a number, by a substrate that produces an audit chain documenting how the number was derived. Before this round of work, the answer was an estimate. Now it is a measurement.

---

Substrate snapshot: 49de9938f62d7311f6b7bb75ee3efdd859606654b3e644ca237fcfc2bbef6680. Empirical-run data lives at runtime/.scratch/empirical_leak_rerun.json in the snapshot’s source tree.