Measured, losses included.
How Raven compares with the fastest stacks we know, on the workloads the road to 1.0 is judged by. Each section names its machine, the versions compared and the commit measured, and the benchmark code is in the repository.
01 — the goal post
JIT-compiled gpt-oss, at least as fast as four baselines.
Beta 1's goal post, JIT compilation on GPU, is met. The next goal post on the road to 1.0: gpt-oss-20b, compiled by Rune.jit, at least as fast as tinygrad, JAX, llama.cpp and vLLM on NVIDIA (through CUDA and through NVIDIA's kernel driver directly) and on AMD. We are not there yet, and this table is where we track it.
| as fast as | NVIDIA · CUDA | NVIDIA · NV (no CUDA runtime) | AMD |
|---|---|---|---|
| llama.cpp | behinddecode 10.2 against 6.83 ms (b12032) | behinddecode 12.0 against 6.83 ms (bench median; 10.7 ms of GPU time) | behinddecode 12.7 against 10.38 ms (b12048, Vulkan) |
| tinygrad | not measured— | not measured— | blockedtinygrad needs Resizable BAR on this machine |
| JAX | not measured— | not measured— | not measured— |
| vLLM | not measured— | not measured— | not measured— |
gpt-oss-20b, decode at position 512. Every cell is behind or not yet measured; the goal is met when the whole table reads “as fast or faster”.
02 — measured · 2026-11
gpt-oss-20b on an NVIDIA RTX 5000 Ada.
| measurement | raven · CUDA | raven · NV | llama.cpp · CUDA |
|---|---|---|---|
| Decode, position 512 of a 1,024 cache | 10.2 ms | 12.0 ms (bench median; 10.7 ms GPU time) | 6.83 ms |
| Decode, Q8_0 attention and head | 8.1–8.3 ms | not measured | — |
| Prefill, 512 tokens | 122 ms | not measured again | 56.0 ms |
| First call, kernels cached | 0.37–0.39 s | 0.37–0.54 s | 75 ms |
Machine “fudge”, under load. raven at 8ef1902f3 (decode), f2af3251b (Q8_0), 4e23903949 (prefill, least of 4 rounds), c29acaee3 (first call); NV at 31aa4521b. llama.cpp's first call has no warmup (59–151 ms).
03 — measured · 2026-11
gpt-oss-20b on an AMD Radeon AI PRO R9700.
where decode's time goes
Expert products take 5.4 ms at 212–254 GB/s, against llama.cpp's 3.0. Top-k's 20-kernel sort and about 37 small kernels a layer take about 1.5 ms; the argmax over the logits 0.63 ms.where prefill's time goes
Expert products take 326 ms on WMMA at 7–8 TFLOPS, against 177. The out projection takes no WMMA: 52 ms, against q's 10.Machine “nonnormal” (gfx1201) at 502431b668; attention on WMMA. First call: 1.03 s when compiling kernels, of which lowering and linking the first block take 69–131 ms on the host; llama.cpp 2.19 s without Mesa's shader cache. llama.cpp's ROCm build needs HIP libraries this machine lacks.
04 — measured · 2026-10-11
The GPU runtime: rig against PyTorch, JAX and tinygrad.
| measurement | rig: own driver / CUDA | PyTorch | JAX | tinygrad: own driver / CUDA |
|---|---|---|---|---|
| Install size, NVIDIA | 7.6 / 115 MiB | 5,458 MiB | 5,774 MiB | 461 / 496 MiB |
| Install size, AMD | 7.6 MiB | 15,045 MiB | — | — |
| Time to first result, NVIDIA | 70 / 134 ms | 720 ms | 538 ms | 336 / 279 ms |
| Time to first result, AMD | 53 ms | 1,106 ms | — | — |
| Time per launch, NVIDIA | 2.1 / 3.2 µs | 7.7 µs | 28.9 µs | 511 / 447 µs |
| Time per launch, AMD | 4.9 µs | 8.1 µs | — | — |
| Load a 256 MiB file, NVIDIA (a loss to PyTorch) | 46.0 / 49.3 ms | 32.9 ms | 169.4 ms | 62.0 / 99.9 ms |
| Load a 256 MiB file, AMD | 38.9 ms | 215.9 ms | — | — |
Medians of release builds, 2026-10-11. NVIDIA: an RTX 5000 Ada, where rig and tinygrad run through their own driver and through CUDA. AMD: a Radeon AI PRO R9700, where PyTorch is the only framework run. Kernels precompiled: compiling them at run time loads NVRTC or comgr. “—”: not run on that machine.
05 — measured · 2026-11-03
Tokenization: brot against gigatoken and HF tokenizers.
| OpenWebText, one thread | brot | reference |
|---|---|---|
| GPT-2 (a loss) | 474.4 MB/s | 816.6 MB/s · gigatoken |
| LLaMA | 190.1 MB/s | 71.4 MB/s · gigatoken |
| Mistral | 203.5 MB/s | 145.4 MB/s · gigatoken |
| BERT-base | 62.7 MB/s | 2.7 MB/s · HF tokenizers |
Machine “fudge” (Intel Core Ultra 5 235), shared. brot at f988690df; gigatoken at cd5ee648c6 (0.11.0); tokenizers 0.23.3. gigatoken refuses WordPiece, so BERT-base runs against HF tokenizers.
06 — measured · 2026-11
Dataframes: talon against DuckDB and Polars.
| workload | talon, against the faster baseline |
|---|---|
| Decision support, SF1 (22 queries) | 6.6× slower (geomean) |
| H2O group-by, 10⁷ rows | 2.1× slower |
| H2O joins | 2.0× slower |
Against the faster of DuckDB 1.5.6 and Polars 1.44.2, at 9b82c5b57a, under load. talon uses 3–10× the CPU.
from doc/performance.mdedit ↗