docs menu

on this page

Module Talon.Column

Columns: one typed array of values, some of them null.

A column is an Arrow array over nx buffers. Its validity is an Nx.bit_t, true at each row that holds a value; its absence means no row is null. Its values are laid out by its type:

one element per row of a primitive nx array: Nx.bit for bool, the integer and float types, int32 positions in the dictionary for categoricals, int32 days for dates and int64 ticks for clocks, durations and datetimes. A tensor column is one (rows, …shape) array of its dtype;
offsets into a child for byte strings, text and lists: text is a list of bytes;
one child per field for records.

An extension column is laid out as its storage. The values under a null are unspecified and deterministic: a reader that needs defined values masks them with the validity, as Arrow requires. Under a null record, its fields' slots are unspecified, their values and validity alike. Columns are immutable, and share their buffers with the tensors and layouts that read them: a column holds values of its own over that memory, so donating a tensor it took or gave ends that tensor alone, and the donation computes into fresh memory (Nx.donate).

A column is canonical when its buffers hold exactly its rows: offsets from 0, values exactly the rows', a validity only when a row is null, and every buffer of elements narrower than a byte (Nx.bit, the int4 dtypes) from the first bit of its storage with every bit past its last element clear. Two canonical columns whose rows hold the same values, those under their nulls included, have the same layout, byte for byte.

A column's buffers are bytes on the host, which formats, text kernels and C stubs read in place: every nx value a column takes or gives, tensors, validities, offsets and positions, is a host value (Nx.host). A caller places a value of another device set on the host first, with Nx.place Nx.Host.on.

#type t

The type for columns.

#val type_ : t -> Type.any

type_ c is the type of c's values.

#val length : t -> int

length c is the number of rows of c.

#val null_count : t -> int

null_count c is the number of null rows of c. The first call may read c's validity once; later calls cost O(1). It raises Invalid_argument for a validity Rune.jit or Rune.vmap traces (Compiled, mapped and differentiated runs), whose bits have no elements to count.

OCaml values

#val v : 'a Type.t -> 'a array -> t

v ty vs is the column of type ty holding vs, without nulls. A float32 or float16 value is stored rounded to the nearest value of the type, ties to even.

Raises Invalid_argument naming the row if ty does not hold a value of vs (Type.holds): text that is not UTF-8, a string outside a categorical's dictionary, an integer outside the type's range, a span that is not a whole number of the unit, a record of other fields. An extension type has no values, so a column of it holds only nulls.

#val of_options : 'a Type.t -> 'a option array -> t

of_options ty vs is like v, with a null for each None.

#val values : 'a Kind.t -> t -> 'a array

values k c is c's values read as k.

Raises Invalid_argument if k does not read c's type (see Kind.provably_equal: no kind reads an extension column), if a row of c is null, or, naming the row, if a value is outside what k reads: an integer outside OCaml's int, a stamp or a span outside Time's range, a list with a null element.

#val options : 'a Kind.t -> t -> 'a option array

options k c is like values, with None for each null.

Tensors and ragged arrays

#val of_tensor : ?validity:Nx.host Nx.bit_t -> ('a, 'b, Nx.host) Nx.t -> t

of_tensor ?validity x is the column of x's rows:

for a 1-D x, of the type of x's dtype: bool for bit and bool, int8 to uint64, float16 to float64;
for x of shape (n, …shape), a tensor column of x's dtype and cell shape shape.

It shares x when x's dtype is the storage of its type; a 1-D bool x is packed into Nx.bit once.

validity marks the rows that hold a value; it defaults to every row.

Raises Invalid_argument if x is a scalar, if x is 1-D of a dtype that no scalar type stores (bfloat16, the float8 and int4 dtypes, complex), or if validity's shape is not x's rows.

#val to_tensor : ('a, 'b) Nx.dtype -> t -> ('a, 'b, Nx.host) Nx.t

to_tensor dt c is c's values as stored, in O(1) after one read of the null count the first time: numbers, booleans as Nx.bit, the days or ticks of temporal values, the codes of a categorical (its dictionary is in its type), and (rows, …shape) for a tensor column. It shares c's buffer, which must not be written.

Raises Invalid_argument if dt is not c's storage dtype (Nx.cast converts the result), if c is not stored one element per row, or if c has a null; under Rune.jit, for a traced validity (Compiled, mapped and differentiated runs), the null raises when the call returns.

#val validity : t -> Nx.host Nx.bit_t option

validity c is true at row i iff row i of c holds a value. None means no row is null. Some v may be true everywhere.

#val ragged : ('a, 'b) Nx.dtype -> t -> ('a, 'b, Nx.host) Nx_ragged.t

ragged dt c is c's rows as a ragged array without a copy, one row per row of c: a list column's offsets and its elements, stored as dt, or a string or binary column's offsets and bytes, dt being Nx.uint8. An extension of such a type is read as its storage. It reads the offsets once, as Nx_ragged.v checks them, and no element. It shares c's buffers, which must not be written.

Raises Invalid_argument if c is neither a list whose elements are stored as dt nor text or bytes with dt Nx.uint8, or if a row of c or an element of a row is null.

val of_ragged : 
  ?validity:Nx.host Nx.bit_t ->
  ('a, 'b, Nx.host) Nx_ragged.t ->
  t

of_ragged ?validity r is the list column whose rows are r's rows, null where validity is false: a list of the element type of_tensor gives r's values, such as list[int32] for Nx.int32 values, and of a tensor type for values of more than one axis. It shares r's buffers, packing bool values as of_tensor does. validity defaults to every row valid. ragged dt (of_ragged r) has r's rows, dt being the dtype the column stores r's values in.

Raises Invalid_argument if validity's shape is not r's number of rows, or if r's dtype has no talon type, as of_tensor does.

Layouts

A layout is a column's Arrow buffers, as formats read and write them.

#type layout =

#| Fixed of {
1. #validity : Nx.host Nx.bit_t option;
2. #values : Nx.host Nx.packed;
}
(*
One element per row, or one cell for a tensor column.
*)
#| Varsize of {
1. #validity : Nx.host Nx.bit_t option;
2. #offsets : Nx.host Nx.int64_t;
3. #child : t;
}
(*
Row r is the child's rows offsets.{r} to offsets.{r + 1} - 1: the elements of a list, or, for string and binary, the bytes, a uint8 child without nulls.
*)
#| Children of {
1. #validity : Nx.host Nx.bit_t option;
2. #length : int;
3. #fields : (string * t) list;
}
(*
length rows, with one child per field of a record, in the record type's order.
*)

The type for layouts.

#val layout : t -> layout

layout c is c's layout, in O(1).

#val of_layout : Type.any -> layout -> (t, int * string) Stdlib.result

of_layout ty l is the column of type ty laid out as l, without a copy, or Error (row, reason) for the first row whose value ty does not hold: text that is not UTF-8, a code outside a categorical's dictionary, a time of day outside the day. A row is checked only where it is not null. reason is a phrase, as in invalid UTF-8 at byte 3. A child holds its own values, so only l's own values are checked.

Raises Invalid_argument if l does not lay out ty: values of another dtype or cell shape than ty's storage, a validity of a shape other than [|n|] for n rows, offsets that are not 1-D, start below 0, decrease or reach past the child, a child of another type (for text, a uint8 column with a null), or fields of other names, types or lengths than ty's; under Rune.jit, for traced offsets of binary (Compiled, mapped and differentiated runs), offsets out of place raise when the call returns, naming Nx_ragged.v.

#val parse : Type.any -> t -> (t, int * string) Stdlib.result

parse ty c is the column of type ty whose rows are the values that the rows of the string or binary column c write, null where c is null, or Error (row, reason) at the first non-null row that is not ty's text or holds a value ty does not, reason a phrase such as not a number. Formats that read text call it, mapping row to their own location. The text of each type is:

bool: true or false;
int8 to uint64: a decimal integer with an optional sign, in the type's range;
float16 to float64: a decimal number with an optional sign, digits on at least one side of an optional point and an optional exponent, or inf, infinity or nan in any case with an optional sign, rounded to the nearest value of the type, ties to even, beyond its range to an infinity;
string: the bytes, valid UTF-8; binary: the bytes;
a categorical: one of the dictionary's strings;
date: YYYY-MM-DD, a year outside 0000 to 9999 signed and of at least four digits, as Time.Date.pp writes it: -0044-03-15;
datetime[u] (a wall clock) and datetime[u, UTC] (an instant): a date of any year, T or a space, hh:mm:ss and an optional fraction of one to nine digits, then, for an instant only, Z or ±hh:mm; a whole number of u in u's range, the int64 ticks, which bound the year.

Raises Invalid_argument if c is neither string nor binary, or ty is another type.

#val print : t -> t

print c is the column of the canonical texts of c's rows, null where c is null, which parse reads back: parse (type_ c) (print c) is c. Formats that write text call it. It is c itself for string and binary, and a string column otherwise. Each value is written in the text parse reads:

a float in the fewest significant digits that read back to it at its type's width, without an exponent from 1e-7 up to 1e21 (150, 0.0015, -0, 1e+21), or nan, inf or -inf;
a datetime with the fewest fraction digits that are exact, and Z for an instant, every tick of its unit included, past Time.Date's years;
an integer in full, past OCaml's int included.

Raises Invalid_argument if c's type is not one parse reads.