Qwen3.5-397B: many requests at once, and drafted tokens

30 September 2026

The previous test ran Qwen3.5-397B across 20 rented GPUs with 16 GB each, every connection through a relay. It served one request at a time, and the GPUs sat almost idle. This test asks three questions about the same pipeline:

How much can it serve when many people use it at once?
How much faster is one request when the model guesses several tokens ahead?
Do the two combine?

Result:

Many requests at once: 32 requests together gave 35.6 tokens per second in total, against 1.25 for one request alone. The best run reached 90.9 tokens per second with 51 requests. 96 requests ran together and every one produced the same tokens as when run alone.
Drafted tokens: the model's own draft layer produced up to 5.3 tokens per trip through the 20 machines, and made a single request up to 2.2× faster.

This fleet is slower per request than the previous one (1.25 against 5.4 tokens per second), because its machines are spread over three US states. The previous fleet's provider ended its listings that morning. The comparisons below are within this fleet.

Setup

Model	Qwen3.5-397B-A17B, 4-bit (Q4_K_M), 20 stages of 3 layers each
Workers	20 rented GPUs (RTX 4090, 5090, 3090 Ti, 5080, L4), each capped at 16 GB; the last stage gets 30 GB when it holds the draft layer
Network	Every hop through one relay in Nevada; workers in Nevada, California and Utah, 1 to 56 ms from it
One request	About 800 ms per token, 23 ms of it GPU computing

Many requests at once

A token spends about 1 ms on each GPU and the rest of its 800 ms on the network, so each machine can work on other requests in the meantime. Each worker now keeps up to 96 separate sessions in memory.

Requests at once	Total tokens per second	Each request, tokens per second	Same tokens as alone
1	1.3	1.25	1 of 1
4	6.2	1.55	4 of 4
16	30	1.88	16 of 16
32	60	1.87	32 of 32
51	90.9	1.79	51 of 51
64	56.4	0.88	64 of 64
96	67.6	0.70	96 of 96

Total is the sum of each request's decode speed once all are running. Counting prompts and start-up as well, 32 requests gave 35.6 and 96 gave 56.6 tokens per second. The 51-request row is a run of 96 in which 45 clients failed to start; the other 51 ran together.

Up to about 50 requests, each keeps its full speed, so the total grows with the number of requests.
Past about 50, each request slows and the total stops rising. Processors were 10-20% busy and GPUs under 7%, so the limit is not computing. We have not found it yet.
Memory: 96 sessions fit in 14.0-15.3 GB per worker, including 12.6 GB of model weights.

Drafted tokens

Qwen3.5 has a small extra layer trained to predict the token after next. The last machine in the pipeline uses it to guess a few tokens ahead and sends the guesses back with each token. The next trip through the 20 machines checks all the guesses at once, keeps the ones the model agrees with, and discards the rest.

Prompt	Guesses per step	Tokens per trip	Guesses accepted	Tokens per second	Speed-up
Code	none	1.0	-	1.25	-
	2	2.8	41 of 42	1.82	1.5×
	4	4.3	49 of 55	2.29	1.8×
	6	5.3	52 of 66	2.69	2.2×
Explanation	none	1.0	-	1.24	-
	2	2.2	35 of 55	1.43	1.2×
	6	3.2	44 of 105	1.60	1.3×
Factual	none	1.0	-	1.22	-
	2	2.6	39 of 47	1.65	1.4×
	6	2.9	42 of 121	1.44	1.2×

The guesses are good. The first one or two are almost always right. Later guesses in a chain are right less often.
Checking guesses is not free yet. A trip that checks guesses took 1.5 to 2 seconds, against 0.8 for a plain trip. Every machine saves its state first and, when a guess is rejected, restores it and redoes the accepted tokens. That is why 5.3 tokens per trip gave 2.2× and not 5.3×. Making this cheap is the next step.
A 4-bit draft layer is enough. An 8-bit copy was accepted no more often.
The published model file leaves the draft layer out. We converted it from the original release (5.7 GB). The last machine needs about 19.5 GB with it.

Both together

Many requests at once, each with 4 guesses per step:

Requests at once	Total tokens per second, no guesses	With guesses
16	21.7	38.9
32	57.1	49.9
32, repeated	56.5	59.6
48	51.3	83.1

Every request completed, and 56% of guesses were accepted, the same as for one request. But the gain is not consistent: 1.8× at 16 requests, 1.6× at 48, and none at 32. Each figure is a single run, and plain runs on these shared rented machines varied by as much between runs. We need more repeats before quoting a speed-up under load.

A correction: the output depends on batching

Runs with guesses were fluent, but often not token-for-token identical to plain runs. They differ at close word choices, for example "iteratively." against "using an iterative approach.".

Drafting does not cause this. Plain decoding gave different tokens at the same place when we only changed how many prompt tokens were sent at a time (8 instead of 64). Repeating a run with the same settings always gave the same tokens. Processing several positions together changes the arithmetic very slightly, and in this model that can flip a near-tie.

Our earlier page said every configuration produced exactly the same tokens. That was true for the prompts and settings we tested then, but it is not true in general for this model.

What the testing caught

Rented machines have end dates. The previous relay and ten workers stopped when their provider's listings ended, and could not be restarted. We rebuilt on listings with weeks left.
Listed locations can be wrong. Two machines listed in California added 220 ms and over 400 ms per hop. Connecting to the relay looked instant from one of them, because a local proxy answered; timing a real reply showed the delay.
Limits sized for one request. Our network layer allowed 4 requests in flight per link, then 16 calls per machine, then 64 sessions. Each was found by adding requests until something refused.
The model runtime kept memory for all 60 layers in every session, about 306 MB each, although a worker runs only 3. It now keeps about 12-20 MB per session.

Cheaper checking of guesses. Done in the next test: undoing wrong guesses now costs under half a millisecond per trip.
Find the limit near 50 requests.
Place machines by measured delay and listing end date, automatically.

Full report and raw data: docs/test-results.

Sangama is open source under the MIT License. Contact: hello@sangama.co · Discord