Sangama → Results
Qwen3.5-397B: several requests per step
30 September 2026, round 11
The three-stage test found that total throughput stopped at about 120 tokens per second because the last GPU handled one frame at a time. This test builds the fix and measures it, one change at a time, on the same kind of fleet: three 96 GB GPUs and a relay in one datacenter.
Result: with each machine working on several waiting requests in one step, 48 requests together ran at 403 tokens per second in total, up from 119. With 64 requests the total was 449. One request alone is unchanged at 48 to 50.
One change at a time
| Change | 16 requests | 48 requests |
| Start: one frame per step | 119 | 119 |
| Waiting requests share a step | 174 | 184 |
| Requests grouped in the order the runtime can merge | 184 | 210 |
| Smaller groups, so all three machines work at once | 195 | 222 |
| One shared attention buffer, so any requests can merge | 263 | 403 |
Total tokens per second: the sum of each request's decode speed, plain decoding, 128 tokens per request. Over whole runs, prompts and start-up included, 48 requests gave 300 tokens per second and 64 gave 339.
- Sharing a step. Frames that arrive while a GPU is busy now run together. The first attempt failed when 31 prompts arrived at once, 866 positions against a limit of 512; groups are now split to fit.
- Order matters. With a separate attention buffer per request, the model runtime merges requests into one step only when their slot numbers are consecutive. Sorting by slot, and then one shared buffer, removed that limit. A group of 16 then costs about 20 ms per machine.
- Smaller groups. One group of everything waiting makes all requests move through the machines as a single wave, so one machine works while the others idle. Smaller groups run back to back keep all three busy.
Is the output still right?
- No request contained another request's text, in every run checked.
- About a third matched their solo output exactly. The rest differ at close word choices, for example "Pipeline parallelism" in bold against the same words without bold as the first token. Working on several requests together changes the arithmetic slightly, as working on several positions does.
What did not work
- A confidence cutoff for guesses gave nothing. We tried stopping the model's draft layer when it was unsure (cutoffs of 0.3, 0.5 and 0.75). The first version spent more time computing the confidence than it saved. With that fixed, a low cutoff trimmed almost nothing and a high one cut good guesses with bad ones.
- Guessing combined with shared steps was wrong. With guesses, shared steps and the shared buffer all on, 13 of 32 requests picked up text from another request's prompt. We discarded those numbers. The cause and the re-test are in the next result.
Full report and raw data: docs/test-results, round 11.
Sangama is open source under the MIT License. Contact: diljit@diljitpr.net · Discord