docs menu

on this page

Module Talon.Type

Column types.

A column's type is what it stores: float32, datetime[ms, UTC], list[string]. A type is indexed by its kind, the OCaml type its cells read as (Kind): float32 is a float t. Whatever fixes what a column stores therefore also fixes what OCaml reads from it. Collections of types of several kinds, such as a Schema.t, hold them as any.

The set of types is closed. A new type is an extension: a named type stored as one of these, identified by its name, its metadata and its storage type together, as in Arrow. Computing on an extension column takes its declaration, a value of the Ext module.

The constructors of t are exposed for matching and private: types are built with the constructors below, which check their arguments.

Under open Talon, this module shadows Stdlib.Type.

Types

#type unit_ =

#| S
(*
Seconds.
*)
#| Ms
(*
Milliseconds.
*)
#| Us
(*
Microseconds.
*)
#| Ns
(*
Nanoseconds.
*)

The type for the units of temporal ticks.

#type ext

The type that extension cells read as. It has no values: an extension column is read only through the extension's declaration, and its kind binds no handle.

#type 'a t = private

#| Bool : bool t
(*
Booleans, stored as Nx.bit, eight to a byte.
*)
#| Int8 : int t
(*
Signed 8-bit integers.
*)
#| Int16 : int t
(*
Signed 16-bit integers.
*)
#| Int32 : int t
(*
Signed 32-bit integers.
*)
#| Int64 : int t
(*
Signed 64-bit integers. A value outside OCaml's int range fails when it is read as int.
*)
#| Uint8 : int t
(*
Unsigned 8-bit integers.
*)
#| Uint16 : int t
(*
Unsigned 16-bit integers.
*)
#| Uint32 : int t
(*
Unsigned 32-bit integers. A value outside OCaml's int range fails when it is read as int.
*)
#| Uint64 : int t
(*
Unsigned 64-bit integers. A value outside OCaml's int range fails when it is read as int.
*)
#| Float16 : float t
(*
IEEE 754 binary16 floats.
*)
#| Float32 : float t
(*
IEEE 754 binary32 floats.
*)
#| Float64 : float t
(*
IEEE 754 binary64 floats.
*)
#| String : string t
(*
UTF-8 text. Text is validated when it enters talon, and invalid bytes belong in Binary.
*)
#| Binary : Binary.t t
(*
Byte strings.
*)
#| Categorical : string iarray -> string t
(*
Strings from a dictionary, stored as int32 positions in it. The dictionary holds distinct UTF-8 strings, and its order is the order of the values.
*)
#| Date : Time.date t
(*
Dates, stored as int32 days since 1970-01-01.
*)
#| Clock : unit_ -> Time.span t
(*
Times of day, stored as int64 ticks of the unit since midnight. A time of day reads as its span from midnight.
*)
#| Duration : unit_ -> Time.span t
(*
Signed durations, stored as int64 ticks of the unit.
*)
#| Datetime : {
1. #unit_ : unit_;
2. #clock : 'c Time.clock;
} -> 'c Time.stamp t
(*
Stamps (Instants and wall clocks), stored as int64 ticks of unit_ since 1970-01-01 00:00:00 of clock: instants when clock is Time.Utc and wall clocks when it is Time.Local. A tick outside the range of Time.stamp fails when it is read.
*)
#| List : 'a t -> 'a array t
(*
Lists of values of the element type, read as arrays.
*)
#| Record : (string * any) list -> Record.t t
(*
Records with the given fields in order, Arrow's struct. Field names are distinct UTF-8 strings.
*)
#| Tensor : ('a, 'b) Nx.dtype * int iarray -> ('a, 'b, Nx.host) Nx.t t
(*
Tensors of one dtype and one shape, one per cell, stored as Arrow's canonical arrow.fixed_shape_tensor. The shape has at least one dimension, and no dimension is negative.
*)
#| Ext : {
1. #name : string;
2. #metadata : string;
3. #storage : 's t;
} -> ext t
(*
Extension types: values stored as storage and identified by name, metadata and storage together. The name is non-empty UTF-8, and storage is not an extension type.
*)

The type for column types whose cells read as 'a.

#and any =

#| Any : 'a t -> any
(*
The type for types of any kind.
*)

Constructors

#val bool : bool t

bool is Bool.

#val int8 : int t

int8 is Int8.

#val int16 : int t

int16 is Int16.

#val int32 : int t

int32 is Int32.

#val int64 : int t

int64 is Int64.

#val uint8 : int t

uint8 is Uint8.

#val uint16 : int t

uint16 is Uint16.

#val uint32 : int t

uint32 is Uint32.

#val uint64 : int t

uint64 is Uint64.

#val float16 : float t

float16 is Float16.

#val float32 : float t

float32 is Float32.

#val float64 : float t

float64 is Float64.

#val string : string t

string is String.

#val binary : Binary.t t

binary is Binary.

#val categorical : string array -> string t

categorical d is Categorical with a copy of the dictionary d, which may be empty.

Raises Invalid_argument if d holds a string twice or a string that is not valid UTF-8, or holds more than 2³¹ - 1 strings.

#val date : Time.date t

date is Date.

#val clock : unit_ -> Time.span t

clock u is Clock in the unit u.

#val duration : unit_ -> Time.span t

duration u is Duration in the unit u.

#val instant : unit_ -> Time.instant t

instant u is Datetime of instants in the unit u.

#val wall : unit_ -> Time.wall t

wall u is Datetime of wall clocks in the unit u.

#val list : 'a t -> 'a array t

list t is List of t.

#val record : (string * any) list -> Record.t t

record fields is Record with fields, in order. A record type may have no fields, and a field's name may be empty.

Raises Invalid_argument if two fields have the same name, or a name is not valid UTF-8.

#val tensor : ('a, 'b) Nx.dtype -> int array -> ('a, 'b, Nx.host) Nx.t t

tensor dt shape is Tensor of dt with a copy of shape.

Raises Invalid_argument if shape is empty or holds a negative dimension.

#val ext : name:string -> ?metadata:string -> 'a t -> ext t

ext ~name ?metadata storage is Ext named name, with metadata (defaults to "") and stored as storage. Format readers build the types of extension columns with it.

Raises Invalid_argument if name is empty or not valid UTF-8, or if storage is an extension type.

Kinds

#val kind : 'a t -> 'a Kind.t

kind t is the kind that t's cells read as: Kind.bool, Kind.int for the integer types, Kind.float for the float types, Kind.string for String and Categorical, Kind.binary, Kind.date, Kind.span for Clock and Duration, Kind.instant and Kind.wall for Datetime by its clock, Kind.list (kind e) for List e, Record.kind for every Record, and Kind.tensor dt for Tensor (dt, _). For Ext it is the extension kind, which reads nothing (see Kind.provably_equal).

Values

#val holds : 'a t -> 'a -> bool

holds t v is true iff t holds v, that is iff v can be stored as t:

an integer is in the type's range;
a float is NaN, infinite, or rounds to a finite value at the type's precision;
a string is valid UTF-8, and for Categorical it is in the dictionary;
an instant or a span is a whole number of the type's unit, and for Clock it is at least zero and less than one day;
each element of a list is held by the element type;
a record has the type's field names in order, and each of its non-null fields has the kind of the field's type and is held by it, an extension field by its storage value;
a tensor has the type's shape.

Booleans, byte strings and dates are always held. Ext has no values. This is the test a literal passes when it takes the type of the operand it meets, and the test values pass when they become a column.

holds t does its work on t once, such as indexing a categorical's dictionary: apply it to t once and use the result for many values.

#val compare_value : 'a t -> 'a -> 'a -> int

compare_value t v0 v1 orders v0 and v1 by talon's total order on the values of t, the order that sorting, comparisons and grouping use:

floats order neg_infinity < … < infinity < nan. -0. equals 0., and every NaN equals every other;
integers, dates, spans and instants order by value;
false comes before true;
strings and byte strings order by their bytes, which for UTF-8 text is code point order;
Categorical values order by their position in the dictionary;
lists order lexicographically, a list coming before every list it is a proper prefix of;
records order lexicographically by their fields, in the type's order;
tensors order lexicographically by their elements in row-major order: floats as above, complex numbers by their real then their imaginary part, each as a float, and other elements by value, unsigned integers as unsigned.

A null record field comes after every value and equals another null field; talon's order puts nulls last at the top level and inside lists too, where this function does not see them. An extension field orders by its storage value. That order serves key identity; sort keys and ordering reductions over a type that is or contains an extension not declared ordered are refused when the verb is applied.

Key identity. Two non-null values are the same key iff compare_value t v0 v1 = 0. Null is one more key, the same as itself only.

compare_value t does its work on t once, such as indexing a categorical's dictionary: apply it to t once and use the result for many comparisons.

Raises Invalid_argument if a Categorical value is not in the dictionary, a record does not have the type's field names in order or holds a non-null field of another kind, or a tensor does not have the type's shape.

Operands

#val common : 'a t list -> 'a t option

common ts is the type at which operands of the types ts meet in an operation, and None if they do not meet or ts is empty. The result does not depend on the order of ts. Scalar types meet at the one of them that contains all the others: a type contains another when it can store every value the other can store, with the same meaning. Lists meet at the list of their elements' common type, and records with the same field names in the same order at the record of their fields' common types. One type contains another when:

they are equal;
int8 in int16 in int32 in int64, uint8 in uint16 in uint32 in uint64, and uint8 in int16, uint16 in int32, uint32 in int64;
float16 in float32 in float64;
a Categorical in String, and in a categorical whose dictionary begins with its own;
a Clock in a clock of a finer unit.

So int8 and uint8 do not meet, while int8, uint8 and int16 meet at int16. uint64 does not meet int64, although every uint64 that OCaml reads is an int64. Clock and Duration never meet, and neither do datetimes or durations that differ in unit, since a coarser unit stores ticks a finer one cannot: cast first. A wall clock and an instant never meet.

Equality and formatting

#val equal : 'a t -> 'b t -> bool

equal t0 t1 is true iff t0 and t1 are the same type: the same constructor with equal arguments. Dictionaries are equal when they hold the same strings in the same order, and record types when they have the same field names with equal types in the same order.

#val pp : Stdlib.Format.formatter -> 'a t -> unit

pp ppf t formats t as schemas and plans show it:

bool, int8 to uint64, float16 to float64, string, binary and date;
categorical["AA", "B6"], with at most the first eight strings of the dictionary, followed by an ellipsis and the dictionary's size when it holds more: categorical["a", "b", "c", "d", "e", "f", "g", "h", … 26];
clock[ns], duration[ms], datetime[us] for a wall clock and datetime[us, UTC] for an instant, with the units s, ms, us and ns;
list[float64], record[carrier string, delay float64] and tensor[float32, 3×4];
ext[ymir.epoch, float64], and ext[units.mass "kg", float64] when the metadata is not empty.

A field name or an extension name formats as is when it is non-empty and holds no ASCII space, comma, bracket, double quote, backslash or ASCII control byte. Otherwise it is quoted, as categories and metadata always are: between double quotes, with double quotes and backslashes preceded by a backslash, and control bytes and bytes that are not part of valid UTF-8 written as \x and two hexadecimal digits.

#val pp_name : Stdlib.Format.formatter -> string -> unit

pp_name ppf n formats the name n of a field or a column as pp formats a field name: as is, or quoted when it must be. Formats print their columns' names with it.

#val pp_quoted : Stdlib.Format.formatter -> string -> unit

pp_quoted ppf s formats s quoted, as pp quotes a category. Messages quote names and texts with it.