Sangama → Results

Qwen3.5-397B on 20 GPUs, through a relay

30 September 2026

We ran Qwen3.5-397B-A17B, one of the largest open models, split across 20 rented GPUs with 16 GB each. Every connection between computers was forced through a relay, as it would be for machines behind home routers. No single machine held more than 3 of the model's 60 layers.

Result: it works, and gives the same answer every time. Our first run decoded at 0.6 tokens per second. After six rounds of changes it runs at 5.0 to 5.5 tokens per second, with the first token in 1.3 to 6.2 seconds and token-for-token identical output.

Setup

 client -> relay -> worker 1 -> relay -> worker 2 -> ... -> relay -> worker 20
                                                                        |
 client <--------------------- next token ------------------------------+

Speed

Decode speed in tokens per second, then time to first token:

Single runs, greedy decoding. Every configuration produced exactly the same tokens as the first full-precision run.

Correction, 30 September: that held for these prompts and settings, but not in general. A later test found that how tokens are batched can change this model's output at close word choices.

Where the time goes

The GPUs spend 21 milliseconds on each token, all 20 together. The model activates only 17 billion of its 397 billion parameters per token, so the computing is cheap. Almost everything else is the network: a token must pass through 20 computers in order, and each hop crosses the relay.

• First run: about 1,500 ms per token, 98.6% of it network.
• Now: 180-200 ms per token.
• GPUs mostly idle: during generation each GPU used about 12.5 GB of memory, drew about 63 W, and showed near 0% utilisation.
• Tiny traffic: each worker sent and received about 0.1 Mbit/s.

So a 397B model fits on 20 machines with 16 GB each, and those machines sit mostly idle. The limit is how long each message takes to cross the network, not bandwidth or computing power.

What made it faster

1. Push-forward (round 2). Each worker used to wait for the rest of the chain to finish before replying, so every token crossed every hop twice. Now each worker passes its result on and replies at once, and the client collects the token from the last worker: 2.2×.
2. Half-size activations (round 2). The numbers passed between workers are sent in 16-bit (BF16) instead of 32-bit format; the model was trained in BF16. Another 1.4-2×, and long prompts stopped timing out.
3. Reliable reconnects (round 3). After a worker restarted, the client tried an old cached address and never recovered. It now always dials through the relay, and recovered by itself after all 20 workers were restarted.
4. Pipelined prompts (round 4). Long prompts are sent in 64-token pieces, so different workers process different pieces at the same time. The first token of a 459-token prompt came 2.7× sooner.
5. Nearby workers (rounds 5 and 6). We measured each worker's distance to the relay and replaced the farthest with machines in the relay's datacenter, including the last stage, whose distance counts twice per token. Decode speed more than doubled.

Speculative decoding: correct, but slower (for now)

Speculative decoding guesses several tokens ahead and has the model check all the guesses in one pass. When the guesses are right, one trip through all 20 workers yields several tokens instead of one. Since our time goes almost entirely to that trip, this was the obvious thing to try.

How it works here:

• The client guesses up to 4 tokens per step by looking back in the text: if the last 2-4 tokens appeared earlier, it guesses whatever followed them then.
• Before checking a batch of guesses, each worker saves a snapshot of its state, including the model's recurrent (linear-attention) state.
• The last worker returns the model's own choice at every position, so the client sees how many guesses were right.
• If some were wrong, the next message tells every worker to restore its snapshot and replay only the accepted tokens locally. This needs no extra network trip.

Correctness: every speculative run produced exactly the same tokens as normal decoding.

Speed: the model accepted few of these guesses, so saving, restoring and replaying on all 20 workers cost more than the trips it saved.

Warm back-to-back runs, measured before round 6, while the network was slower than it is now.

Status: off by default. To pay off it needs better guesses: Qwen3.5 ships its own multi-token-prediction layer, reported to be right 70-80% of the time, which would mean roughly 2-2.4× faster decoding. That layer is not in our published slices yet. Snapshots also need to get cheaper, by saving only the recurrent state and trimming the rest.

What the testing caught

• Relay rate limits. Default relay rate limits silently disconnected most workers within minutes when many computers shared one network address. Fixed with explicit limits.
• Discovery traffic. Every worker was asking every other worker what it held, through the relay: about 170 connections where the route needs 39. Each request took about a second. Workers behind a relay now ask the relay only, and a request takes 2-40 ms.
• A dead worker blocked the others. After one worker vanished, the client could not reach the healthy ones until it was restarted. Fixed in round 3.
• Rented machines come and go. One relay box died, and one host reclaimed two GPUs mid-test. Replacements downloaded their layers and joined in 5-15 minutes. The output stayed identical.

Next

• Better guesses for speculative decoding. Done in the next test: the model's own multi-token-prediction layer gives up to 5.3 tokens per trip.
• Automatic placement. Have the scheduler pick workers close to the relay, or to each other, as we did by hand in round 5.
• Direct connections on home networks. Our rented machines could not accept incoming connections, so hole punching could not be tested here.

Full report, per-machine metrics and raw data: docs/test-results.

Sangama is open source under the MIT License. Contact: hello@sangama.co · Discord