SnowLLM

Local LLM inference, hand-tuned for one chip. The Ryzen AI Max.

$curl -fsSL https://snowllm.dev/install.sh | sh

Checks the machine before downloading anything, then installs into ~/.local/share/snowllm and links snowllm into ~/.local/bin. Re-run to upgrade.

Linux x86_64 — tested Windows 11 + WSL2 — not verified · ROCm reaches the GPU through /dev/dxg there rather than /dev/kfd, so set it up with AMD's WSL guide first. Nobody has run SnowLLM this way.

76.7tok/s
Model
Qwen3.6-35B-A3B-FP8
Machine
Ryzen AI Max+ 395
Shape
8K in / 1K out, one user
Spec
MTP depth 2, AL 2.983
1K 8K 32K 128K 220K 86.9 49.3
Output tok/s by input context length, one user — the five measured points, evenly spaced
Measured One Ryzen AI Max+ 395. Qwen3.6-35B-A3B-FP8. ROCm torch 2.12.0. Median of 3 reps.

Every number here has its conditions attached.

Speculative decoding is on at depth 2 in all rows below. Prefill and decode are reported separately, split at the last request's first token.

Concurrency — 8K in, 1K out

UsersOutput tok/s
1 user76.7
2 users130.0
3 users143.5
4 users176.9

Total output tok/s. Prefill holds 3430–3467 tok/s throughout; TTFT runs 2.1 s at one user to 5.3 s at four.

Context length — one user, 1K output tokens each

Input contextOutput tok/s
1K tokens86.9
8K tokens76.9
32K tokens72.2
128K tokens56.3
220K tokens49.3

Rows are the prompt length; every run generates 1K tokens on top of it. Prefill falls from 2315 to 1227 tok/s across the sweep, and time to first token at 220K is 168 s.

Accuracy AIME 2026, 30 problems, avg@4. Graded on the last boxed integer.

91.7% — against a model card that reports 92.7 at twice the depth.

110 of 120 samples, ±4.2. Qwen reports 92.7 avg@8 for the same checkpoint. The gap sits inside the error bar, so treat it as agreement, not as a difference.

28 of 30 problems scored 3/4 or better. One problem has never converged on this checkpoint and alone accounts for 8% of the run's output tokens.

Hardware Tested on one machine. No cross-hardware comparison implied.

It runs on one chip, and one model.

  • Linux x86_64 with an AMD gfx1151 GPU — the Ryzen AI Max 300 series, Strix Halo
  • ROCm 7.x, and Python 3.10–3.14
  • Qwen3.6-35B-A3B-FP8, the only supported model today

Only the Ryzen AI Max+ 395 has been tested. The rest of the 300 series shares the same GPU and should work, but nobody has checked. The installer refuses to run on anything else rather than fail later.

Run A local safetensors checkpoint, then one command.

Point it at a checkpoint.

hf download Qwen/Qwen3.6-35B-A3B-FP8 --local-dir ~/models/Qwen3.6-35B-A3B-FP8
snowllm ~/models/Qwen3.6-35B-A3B-FP8

Serves an OpenAI-compatible API. snowllm --help lists every flag.

If huggingface.co is slow from where you are, hf honours the HF_ENDPOINT environment variable — point it at a mirror you trust.

Flags The ones worth knowing before the first run.
FlagWhat it does
--max-num-seqs Concurrency ceiling, up to 256. A memory choice: each request pins linear-attention state for its whole life.
--max-model-len Context length. A per-request ceiling, not a reservation — the KV pool is shared, and a request that outgrows it is preempted and re-prefilled.
--gpu-memory-utilization Fraction of the GPU the server may occupy, 0.9 by default.
--prefix-cache-gib Host RAM for remembering shared prompt prefixes, 4 by default. A repeat behind a 16K prefix costs 0.36 s instead of 5.1 s; the first one pays about 1%.
--num-spec Speculative depth. The optimum depends on the workload; 2 is a robust default.
--kv-cache-dtype bf16 or int8. int8 halves KV bytes: more decode throughput at long context, less prefill.
Caveats What the numbers above do not cover.
  • One machine, one thermal state, TDP unpinned. Run-to-run spread is 6.8–11.1% with speculation on.
  • Concurrency was measured to four users, no further.
  • The speculative rows sit at 99% draft acceptance on synthetic prompts. Expect 10–20% below that on real text.
  • Speed and accuracy are separate runs. No row pairs throughput with a quality metric.
Next
  • More models
  • Tensor parallelism at any degree
  • NPU/GPU co-working prefill
  • Native INT4 quantization