github ↗

roadmap

The road to 1.0.

1.0 is the release where Raven can replace the typical Python stack in production. Two workflows define it, and each has a goal post measured against the Python stack it replaces.

take part, or sponsor →

Releases

  1. alpha · 2025-11

    The libraries, eager on the host.

    FFT and a full linear algebra suite in Nx, dataframes in Talon, and the decision to port tinygrad to OCaml.

  2. beta 1 · done

    Compiled, on GPUs.

    A whole training step compiles into one GPU program, with every layer in Raven.

  3. toward 1.0 · now

    gpt-oss, against the fastest.

    JIT-compiled gpt-oss-20b at least as fast as tinygrad, JAX, llama.cpp and vLLM, on NVIDIA and AMD.

    where we are against it →

01 — where we are

The stack works end to end.

Beta 1's goal post, JIT compilation on GPU, is met: a whole training step compiles into a single GPU program. Every layer of it is Raven: Brot tokenizes the corpus, Kaun defines the model, Rune differentiates it, Tolk compiles it, Vega updates the weights, and Munin records the metrics. The July update →

The next goal post on the road to 1.0 raises the bar to gpt-oss-20b, a mixture-of-experts model, compiled end to end on NVIDIA and AMD. Performance tracks it against tinygrad, JAX, llama.cpp and vLLM.

02 — 1.0 workflow

Training: fine-tune open-weight LLMs.

goal post

Fine-tune Llama 3.1 70B at 90% or better of the throughput of the equivalent PyTorch stack.

  • LoRA fine-tuning on one 8×H100 node, against torch with PEFT
  • Full fine-tuning across two 8×H100 nodes, against torch with FSDP

2 of 10 milestones done · 3 in progress

  1. Llama-class models on Kaun

    doneRoPE, grouped-query attention, RMSNorm, SwiGLU, and a Llama with validated checkpoint import: Llama 3.2 1B and TinyLlama 1.1B match the reference, layer by layer.

  2. Memory-efficient attention

    doneA flash-attention-class fused kernel generated through Tolk, with O(n) memory at real context lengths. Done in Rune's attention kernels.

  3. Mixed-precision training

    in progressbf16 compute with fp32 master weights in Vega's optimizers, and Rune.remat's savings holding through the compiled path. Rune.remat already recomputes under Rune.jit, and Vega steps bf16 parameters.

  4. GEMM throughput at parity

    openTolk's generated bf16 GEMMs approaching cuBLAS throughput at Llama shapes.

  5. JIT-compiled linear algebra and FFT

    in progressCholesky, LU, QR, SVD, symmetric eigendecomposition, linear solves and FFTs compile today; general eigendecomposition and complex matrices remain.

  6. LoRA in Kaun

    openAdapters over any parameter tree, with merged-weight export.

  7. The data pipeline

    openBrot at Gigatoken-level throughput, and streaming tokenized datasets fast enough that GPUs never wait.

  8. Distributed training, in three steps

    in progressSingle-node data parallelism compiles under Rune.jit into one program today. Then FSDP-style sharding, then multi-node with our own collectives and a launcher.

  9. Checkpointing at scale

    openSharded save and resume of the full training state, so a multi-day run survives a node failure.

  10. The benchmark harness

    openPinned reference stacks, re-runnable benchmarks, published numbers and loss-parity checks.

03 — 1.0 workflow

Deployment: serve open-weight LLMs from one static binary.

goal post

Serve open-weight models behind an OpenAI-compatible API, as a dependency-free static binary, at 90% or better of vLLM's throughput.

  • Llama 3.1 70B at bf16, tensor-parallel on 4×H100
  • Llama 3.1 70B at INT4 on one H100, compiled ahead of time
  • DeepSeek-V4-Pro at native FP4/FP8, expert-parallel across two 8×H200 nodes

0 of 11 milestones done · 5 in progress

  1. In-process inference

    openSmall and mid-size models compiled ahead of time and linked into an OCaml service: running a model is a function call.

  2. The inference engine

    openKV cache, continuous batching and PagedAttention, following vLLM's design.

  3. The serving layer

    openAn OpenAI-compatible API with token streaming, backed by Raven-compiled open-weight models.

  4. Tensor-parallel inference

    in progressSharding a 70B model at bf16 across several GPUs. The Llama example compiles a tensor-parallel decode step today.

  5. Post-training quantization

    in progressWeight-only INT8 is done (Nx_quant.quantize_q8_0); calibrated INT4, which fits 70B on one GPU, is next.

  6. MoE serving

    in progressThe DeepSeek-V4 family in Kaun, then expert routing, expert-parallel execution, and FP8 and FP4 paths. The gpt-oss example already routes tokens to experts.

  7. Multi-node serving

    openThe engine spans nodes, reusing the collectives and launcher from distributed training.

  8. Ahead-of-time compilation

    openNative code on CPU, precompiled kernels on GPU: nothing gets JIT-compiled at startup.

  9. Weight streaming

    in progressCheckpoints load at disk bandwidth. Rig already maps files and overlaps disk reads with transfers; direct I/O and loading several GPUs at once remain.

  10. The driver-less CUDA path

    in progressTalking to the GPU without any CUDA userspace. Rig already drives NVIDIA GPUs this way and gpt-oss-20b decodes through it; kernels still compile through NVRTC. Built for 1.0, but 1.0 doesn't gate on it.

  11. The serving benchmark

    openThroughput and inter-token latency against a pinned vLLM, published and re-runnable.

When both workflows hold their bar, and every core package clears its readiness bar, we ship 1.0.

If your team wants to take part, as an early adopter, a production pilot or a sponsor:

get in touchsponsor raven

from doc/roadmap.mdedit ↗