A small model across CPUs, GPUs and the Internet

29 September 2026

Measured on 29 September 2026 with the Qwen2.5-0.5B-Instruct test model, split across two workers. The full list of test scenarios and raw results is in the repository.

Same answer everywhere. At full precision, every combination produced exactly the same tokens: CPU, Apple Metal, NVIDIA CUDA and Vulkan, with either of the two engines, mixed in any order.
One machine, two GPUs: 103 tokens per second (220 with 4-bit weights).
Across the Internet: 3.1 tokens per second between a Mac on a home network and a GPU in a US datacenter, 260-370 ms apart. Distance, not computing power, sets the pace.
Through a relay: 7 out of 7 generations were exact over four minutes of continuous use, with every connection forced through the relay.

Route (layers 0-11 → 12-23)	Where	Result	Speed
Candle Metal → Candle Metal	One Mac	Reference output	-
Candle CUDA → Candle CUDA	Two GPUs, one machine	Identical	55 tokens/s
llama.cpp CUDA → llama.cpp CUDA	Two GPUs, one machine	Identical	103 tokens/s
llama.cpp Vulkan → llama.cpp Vulkan	Two GPUs, one machine	Identical	106 tokens/s
llama.cpp Vulkan → Candle CUDA	Two GPUs, one machine	Identical	86 tokens/s
Candle Metal → Candle CUDA	Home Mac → US datacenter	Identical	3.1 tokens/s
Candle CUDA → Candle Metal	US datacenter → home Mac	Identical	1.2 tokens/s
Candle Metal → Candle CUDA, all through a relay	Home Mac, US GPU, relay	Identical, 7 of 7	0.6 tokens/s
4-bit and full-precision workers mixed	Any	Refused, by design	-

Speeds are single runs of a small test model and include network delay; they are not benchmarks. 4-bit output is only reproducible with the same weight file on the same kind of GPU, so a route must use one published file.

What the testing caught

GPU context bug: NVIDIA CUDA workers failed on every conversation after the first. Found, fixed, and re-tested over five conversations in a row.
Relay time limit: relay connections closed after two minutes, cutting off requests in progress. Raised to one hour; four minutes of continuous traffic then ran cleanly.
Mixed weight files: two differently compressed copies of the same model were accepted in one route. Workers now report a checksum, and such routes are refused.

Sangama is open source under the MIT License. Contact: hello@sangama.co · Discord