Module `Talon.Col`

Column creation and manipulation.

Columns are the building blocks of dataframes, each storing a homogeneous sequence of values with consistent null handling.

type t

The type for columns.

Columns store homogeneous data with consistent null handling:

Numeric data backed by 1D Nx tensors with an optional null mask.
String data as string option array.
Boolean data as bool option array.

Generic constructors

val numeric : ('a, 'b) Nx.dtype -> 'a array -> t

numeric dtype arr is a numeric column from arr with dtype dtype.

val numeric_opt : ('a, 'b) Nx.dtype -> 'a option array -> t

numeric_opt dtype arr is a nullable numeric column from arr with dtype dtype. None values are recorded in the null mask.

From arrays (non-nullable)

Create columns from arrays without introducing null masks. Values are taken literally; to represent missing data, use the _opt constructors instead.

val float32 : float array -> t

float32 arr is a non-nullable float32 column from arr.

The resulting column has no null mask. All values, including nan, are treated as regular data.

val float64 : float array -> t

float64 arr is a non-nullable float64 column from arr.

The resulting column has no null mask. All values, including nan, are treated as regular data.

val int32 : int32 array -> t

int32 arr is a non-nullable int32 column from arr.

The resulting column has no null mask.

val int64 : int64 array -> t

int64 arr is a non-nullable int64 column from arr.

The resulting column has no null mask.

val bool : bool array -> t

bool arr is a non-nullable boolean column from arr.

All values are wrapped as Some value, creating a column with no nulls.

val string : string array -> t

string arr is a non-nullable string column from arr.

All values are wrapped as Some value, creating a column with no nulls.

From option arrays (nullable)

Create columns from option arrays with explicit null representation. Numeric types attach a null mask (while storing placeholder values in the tensor), whereas string and boolean types preserve the option structure.

val float32_opt : float option array -> t

float32_opt arr is a nullable float32 column from arr.

None values are recorded in the null mask. Placeholder nan values are stored in the tensor; callers must rely on the mask (via option accessors or Agg helpers) to detect nulls.

val float64_opt : float option array -> t

float64_opt arr is a nullable float64 column from arr.

None values are recorded in the null mask. Placeholder nan values are stored in the tensor; callers must rely on the mask to detect nulls.

val int32_opt : int32 option array -> t

int32_opt arr is a nullable int32 column from arr.

None values are recorded in the null mask. The tensor stores Int32.min_int as placeholder, but the mask is authoritative when checking for nulls.

val int64_opt : int64 option array -> t

int64_opt arr is a nullable int64 column from arr.

None values are recorded in the null mask. The tensor stores Int64.min_int as placeholder, but the mask is authoritative when checking for nulls.

val bool_opt : bool option array -> t

bool_opt arr is a nullable boolean column from arr.

The option array is used directly without conversion. O(1).

val string_opt : string option array -> t

string_opt arr is a nullable string column from arr.

The option array is used directly without conversion. O(1).

Properties

val length : t -> int

length col is the number of elements in col.

val null_mask : t -> bool array option

null_mask col is the null mask of col, if any.

Returns Some mask when an explicit mask was attached via a nullable constructor, None otherwise.

val dtype : 
  t ->
  [ `Float32 | `Float64 | `Int32 | `Int64 | `Bool | `String | `Other ]

dtype col is the column's data type as a poly-variant tag.

val is_null_at : t -> int -> bool

is_null_at col i is true iff the value at index i is null.

Checks the null mask for numeric columns, or tests for None in string/boolean columns.

From tensors

val of_tensor : ('a, 'b) Nx.t -> t

of_tensor t is a non-nullable column from the 1D tensor t.

The tensor's dtype is preserved. Existing payload values (including NaNs or extremal integers) remain regular data.

Raises Invalid_argument if t is not 1D. O(1) — the tensor is used directly without copying.

Null handling

val has_nulls : t -> bool

has_nulls col is true iff col contains at least one null value.

Checks the null mask for numeric columns, or scans for None in string/boolean columns.

val null_count : t -> int

null_count col is the number of null values in col.

val drop_nulls : t -> t

drop_nulls col is col with all null values removed.

The column type is preserved.

val fill_nulls : t -> value:t -> t

fill_nulls col ~value is col with null values replaced by the first element of value.

value must be a single-element column of the same type as col. Raises Invalid_argument if column types don't match.

See also Talon.fill_null for a more convenient scalar-based API at the dataframe level.

Column transforms

val cumsum : t -> t

cumsum col is the cumulative sum of col, preserving the dtype.

val cumprod : t -> t

cumprod col is the cumulative product of col, preserving the dtype.

val diff : ?periods:int -> t -> t

diff ?periods col is the element-wise difference between consecutive values. periods defaults to 1.

val pct_change : ?periods:int -> t -> t

pct_change ?periods col is the fractional change between consecutive values. nan where the previous value is zero. periods defaults to 1. Result is always float64.

val shift : periods:int -> t -> t

shift ~periods col is col with values shifted by periods positions. Positive shifts move values down (inserting nulls at the top), negative shifts move values up.

Extraction

val to_tensor : ('a, 'b) Nx.dtype -> t -> ('a, 'b) Nx.t option

to_tensor dtype col is the underlying tensor if col is numeric and its dtype matches dtype.

val to_string_array : t -> string option array option

to_string_array col is the underlying string option array if col is a string column.

val to_bool_array : t -> bool option array option

to_bool_array col is the underlying bool option array if col is a boolean column.

val to_string_fn : ?null:string -> t -> int -> string

to_string_fn ?null col is a function that formats the value at index i as a string.

The underlying array is extracted once so repeated calls are O(1). null defaults to "<null>".

val pp : Stdlib.Format.formatter -> t -> unit

pp formats a column for inspection. Shows the dtype, length, and up to 5 values.