30 September 2026, round 12
In the previous result, combining guessed tokens with shared steps mixed text between requests. This test runs the fix on the full model and measures what guessing does once machines share steps.
Result: the fix holds, and no request contained another's text. But under load guessing now lowers the total: 16 requests together delivered 233 tokens per second without guesses and 174 with four guesses per step.
| Model | Qwen3.5-397B-A17B, 4-bit, 3 stages of 20 layers |
|---|---|
| Machines | 3 × RTX PRO 6000 Blackwell with 96 GB, and a relay, in one datacenter |
| Settings | 16 sessions per machine, shared steps, shared attention buffer, up to 4 positions undone without saving state |
| Load | 16 requests at once, 256 tokens each |
| Guesses per step | Total tokens per second, whole run | Tokens per trip | Guesses accepted | Same tokens as alone | Text from another request |
|---|---|---|---|---|---|
| none | 233 | 1.00 | - | 7 of 16 | none |
| 2 | 201 | 2.54 | 2468 of 3149 | 3 of 16 | none |
| 4 | 174 | 3.26 | 2839 of 4929 | 8 of 16 | none |
One request alone on the same build: 47 tokens per second without guesses and 100 with four.
With one shared attention buffer, the model runtime merges requests whose slot numbers have gaps between them into one step. The states it keeps for undoing guesses come out wrong in such a step, and a request then continues from another request's context. Guesses alone, shared steps alone, and the shared buffer alone each worked; the three together did not. Requests with guesses now share a step only with neighbours in adjacent slots. We reproduced the fault and the fix on a small model before this run.
Earlier pages report totals as the sum of each request's own speed. That overstates the total when requests finish at different times, which guessing causes: it read 379 for the four-guess row above, whose whole run delivered 174. For runs without guesses the two are closer: 273 against 233 here, and 403 against 300 in the previous result. This page counts tokens over the whole run.
Full report and raw data: docs/test-results, round 12.
Sangama is open source under the MIT License. Contact: diljit@diljitpr.net · Discord