github ↗
docs menu

Module Nx_io.Gguf

GGUF files.

A GGUF file holds a model's arrays and its metadata, key-values that describe the architecture, the tokenizer and the file itself, as GGML's specification defines them. Each array is stored as one GGUF type (Gguf.dtype). A scalar type stores each element whole, and a block format stores the elements of a row in blocks of a fixed number of elements and bytes. An array's logical shape is its shape in elements, in row-major order (Gguf.tensor_info).

An array of a scalar type loads at its logical shape. An array of a block format loads as its bytes, at uint8, at its logical shape with the last extent replaced by the size of a row in bytes. A Q8_0 block is 32 elements in 34 bytes, so a Q8_0 array of logical shape [|m; 64|] loads at shape [|m; 68|].

#type value =
  1. #| Uint8 of int
  2. #| Int8 of int
  3. #| Uint16 of int
  4. #| Int16 of int
  5. #| Uint32 of int
  6. #| Int32 of int
  7. #| Uint64 of int64
  8. #| Int64 of int64
  9. #| Float32 of float
  10. #| Float64 of float
  11. #| Bool of bool
  12. #| String of string
    (*

    The bytes stored, UTF-8 by the specification.

    *)
  13. #| Array of value array

The type for metadata values, one case per GGUF value type. An unsigned 64-bit value is held in int64 by its bits. An array's elements are of one type.

#type dtype =
  1. #| F32
  2. #| F16
  3. #| BF16
  4. #| F64
  5. #| I8
  6. #| I16
  7. #| I32
  8. #| I64
  9. #| Q4_0
  10. #| Q4_1
  11. #| Q5_0
  12. #| Q5_1
  13. #| Q8_0
  14. #| Q8_1
  15. #| Q2_K
  16. #| Q3_K
  17. #| Q4_K
  18. #| Q5_K
  19. #| Q6_K
  20. #| Q8_K
  21. #| IQ2_XXS
  22. #| IQ2_XS
  23. #| IQ3_XXS
  24. #| IQ1_S
  25. #| IQ4_NL
  26. #| IQ3_S
  27. #| IQ2_S
  28. #| IQ4_XS
  29. #| IQ1_M
  30. #| TQ1_0
  31. #| TQ2_0
  32. #| MXFP4

The type for GGUF storage types. F32, F16, BF16, F64, I8, I16, I32 and I64 are the scalar types, which load as float32, float16, bfloat16, float64, int8, int16, int32 and int64. The others are block formats.

#type tensor_info = {
  1. #dtype : dtype;
    (*

    The type the array is stored as.

    *)
  2. #shape : int array;
    (*

    The array's logical shape, the reverse of the dimensions the file lists, whose first varies fastest.

    *)
}

The type for an array's description in the file.

#type t

The type for the contents of a GGUF file.

#val version : t -> int

version g is g's format version, 2 or 3.

#val metadata : t -> (string * value) list

metadata g is g's key-values, in file order.

#val tensors : t -> Nx.host Archive.t

tensors g is the archive of g's arrays.

#val info : string -> t -> tensor_info

info name g is the description of g's array name.

Raises Failure if g has no array name.