30 September 2026, round 14
In our previous test, one request crossing three US states ran at about 10 tokens per second. Almost all of each trip was network time. When the model guesses the next few tokens and one trip checks them all, the same network time yields several tokens. We had not measured that across a distance since making wrong guesses cheap to undo.
Result:
| Model | Qwen3.5-397B-A17B, 4-bit, 3 stages of 20 layers |
|---|---|
| Stages | 96 GB GPUs in Maryland, Wisconsin and Texas |
| Relay and client | One machine in Colorado |
| Network | Direct connections over QUIC |
| Cost | About $4.95 an hour |
A trip without guesses took 82 ms: 15 ms of GPU computing and 67 ms of network and handling.
Tokens per second, 128 tokens per run. "Guesses asked for" is what the client requests; the last machine decides how many to make.
| Guesses asked for | Code | Explanation | Factual answer |
|---|---|---|---|
| none | 12.2 | 12.2 | 12.2 |
| 2 | 27.9 | 25.8 | 27.0 |
| 4 | 41.2 | 30.2 | 31.0 |
| 6 | 46.0 | 28.8 | 29.0 |
| 8 | 48.6 | 28.2 | 28.2 |
More guesses help code, where most are right, and hurt prose, where the later ones are usually wrong and still cost computing. With a fixed eight guesses, code ran at 50 tokens per second and prose at 25 to 27.
The last machine now decides for each request. It keeps a record of how often that request's first guess, first two, and so on were right, estimates how long a trip takes with each count, and picks the count that gives the most tokens per second. With eight asked for, prose ran at 28 and code kept 49.
We tried two simpler rules first. One cut guesses whenever the last GPU was busy, which lost speed here. One used a single success rate for all requests, which made code guess too little after prose. The rule has only been tested where the network dominates.
| Requests at once | Total without guesses | Total with guesses |
|---|---|---|
| 4 | 40 | 72-80 (not verified) |
| 8 | 73 | 124-137 (not verified) |
| 16 | 128-133 | 144-173 (not verified) |
| 32 | 205 | 202 (not verified) |
Tokens per second over the whole run, start-up included.
The fault. In two of the runs with guesses, 2 of 16 requests picked up text from another request and stopped early. One answer about splitting a model across computers ended with "the capital city of Australia".
Early in the test, the link to one machine was dropping packets, and one request over TCP ran at 5.5 tokens per second. By the time we switched to QUIC the link had recovered, so we did not get to compare the two under loss. On the clean link both ran at 12.
Full report and raw data: docs/test-results.
Sangama is open source under the MIT License. Contact: diljit@diljitpr.net · Discord