Sangama → Results

Qwen3.5-397B: several requests per step

30 September 2026, round 11

The three-stage test found that total throughput stopped at about 120 tokens per second because the last GPU handled one frame at a time. This test builds the fix and measures it, one change at a time, on the same kind of fleet: three 96 GB GPUs and a relay in one datacenter.

Result: with each machine working on several waiting requests in one step, 48 requests together ran at 403 tokens per second in total, up from 119. With 64 requests the total was 449. One request alone is unchanged at 48 to 50.


One change at a time

Change16 requests48 requests
Start: one frame per step119119
Waiting requests share a step174184
Requests grouped in the order the runtime can merge184210
Smaller groups, so all three machines work at once195222
One shared attention buffer, so any requests can merge263403

Total tokens per second: the sum of each request's decode speed, plain decoding, 128 tokens per request. Over whole runs, prompts and start-up included, 48 requests gave 300 tokens per second and 64 gave 339.

Is the output still right?

What did not work

Full report and raw data: docs/test-results, round 11.


Sangama is open source under the MIT License. Contact: diljit@diljitpr.net · Discord