Sangama → Results

Qwen3.5-397B: guesses under load

30 September 2026, round 12

In the previous result, combining guessed tokens with shared steps mixed text between requests. This test runs the fix on the full model and measures what guessing does once machines share steps.

Result: the fix holds, and no request contained another's text. But under load guessing now lowers the total: 16 requests together delivered 233 tokens per second without guesses and 174 with four guesses per step.


Setup

ModelQwen3.5-397B-A17B, 4-bit, 3 stages of 20 layers
Machines3 × RTX PRO 6000 Blackwell with 96 GB, and a relay, in one datacenter
Settings16 sessions per machine, shared steps, shared attention buffer, up to 4 positions undone without saving state
Load16 requests at once, 256 tokens each

Numbers

Guesses per stepTotal tokens per second, whole runTokens per tripGuesses acceptedSame tokens as aloneText from another request
none2331.00-7 of 16none
22012.542468 of 31493 of 16none
41743.262839 of 49298 of 16none

One request alone on the same build: 47 tokens per second without guesses and 100 with four.

What went wrong before

With one shared attention buffer, the model runtime merges requests whose slot numbers have gaps between them into one step. The states it keeps for undoing guesses come out wrong in such a step, and a request then continues from another request's context. Guesses alone, shared steps alone, and the shared buffer alone each worked; the three together did not. Requests with guesses now share a step only with neighbours in adjacent slots. We reproduced the fault and the fix on a small model before this run.

A note on our totals

Earlier pages report totals as the sum of each request's own speed. That overstates the total when requests finish at different times, which guessing causes: it read 379 for the four-guess row above, whose whole run delivered 174. For runs without guesses the two are closer: 273 against 233 here, and 403 against 300 in the previous result. This page counts tokens over the whole run.

Next

Full report and raw data: docs/test-results, round 12.


Sangama is open source under the MIT License. Contact: diljit@diljitpr.net · Discord