Module Talon

Dataframe library for tabular data manipulation.

Dataframes are immutable collections of named, typed columns with equal length. Columns can hold numeric tensors (via Nx), strings, or booleans, each with explicit null semantics.

type t

The type for dataframes.

Dataframes are immutable tabular data structures with named, typed columns. All columns in a dataframe have the same length.

type 'a row

The type for row-wise computations producing values of type 'a.

Row computations form an applicative functor, allowing composition of independent computations from multiple columns.

Columns

module Col : sig ... end

Column creation and manipulation.

DataFrame creation

val empty : t

empty is an empty dataframe with no rows or columns.

Neutral element for concat.

val create : (string * Col.t) list -> t

create pairs is a dataframe from (name, column) pairs.

Column names must be unique (case-sensitive) and all columns must have the same length.

Raises Invalid_argument if duplicate column names exist or column lengths differ.

val of_tensors : ?names:string list -> ('a, 'b) Nx.t list -> t

of_tensors ?names tensors is a dataframe from 1D tensors.

All tensors must have the same shape and dtype. names defaults to "col0", "col1", etc.

Raises Invalid_argument if tensors have inconsistent shapes, any tensor is not 1D, names are not unique, or the wrong number of names is provided.

val of_nx : ?names:string list -> ('a, 'b) Nx.t -> t

of_nx ?names tensor is a dataframe from a 2D tensor.

Each column of the tensor becomes a dataframe column. names defaults to "col0", "col1", etc.

Raises Invalid_argument if tensor is not 2D or names are not unique.

Shape and inspection

val shape : t -> int * int

shape df is (rows, columns).

val num_rows : t -> int

num_rows df is the number of rows in df.

val num_columns : t -> int

num_columns df is the number of columns in df.

val column_names : t -> string list

column_names df is the column names of df in order.

val column_types : t -> (string * [ `Float32 | `Float64 | `Int32 | `Int64 | `Bool | `String | `Other ]) list

column_types df is the column names paired with their detected types.

val select_columns : t -> [ `Numeric | `Float | `Int | `Bool | `String ] -> string list

select_columns df category is the column names matching category.

Categories:

  • `Numeric: all numeric types (float32, float64, int32, int64)
  • `Float: floating-point types only (float32, float64)
  • `Int: integer types only (int32, int64)
  • `Bool: boolean columns
  • `String: string columns
val is_empty : t -> bool

is_empty df is true iff df has no rows.

Note. A dataframe can have columns but zero rows and still be considered empty.

Column access and manipulation

val get_column : t -> string -> Col.t option

get_column df name is the column named name in df, if any.

val get_column_exn : t -> string -> Col.t

get_column_exn df name is the column named name in df.

Raises Not_found if the column does not exist.

val to_array : ('a, 'b) Nx.dtype -> t -> string -> 'a array option

to_array dtype df name is the numeric column name as a typed array if the column exists and matches dtype.

Null values retain their placeholder representation (NaN for floats, sentinel values for integers). See to_opt_array to distinguish nulls from data.

val to_opt_array : ('a, 'b) Nx.dtype -> t -> string -> 'a option array option

to_opt_array dtype df name is the numeric column name as an option array if the column exists and matches dtype. None for null elements, Some v for present values.

val to_bool_array : t -> string -> bool option array option

to_bool_array df name is the bool option array for column name if it exists and is bool type.

val to_string_array : t -> string -> string option array option

to_string_array df name is the string option array for column name if it exists and is string type.

val has_column : t -> string -> bool

has_column df name is true iff df has a column named name.

val add_column : t -> string -> Col.t -> t

add_column df name col is df with column name added or replaced.

Raises Invalid_argument if Col.length col differs from num_rows df.

val drop_column : t -> string -> t

drop_column df name is df without column name.

Note. Returns df unchanged if the column does not exist.

val drop_columns : t -> string list -> t

drop_columns df names is df without the named columns.

Non-existent columns are silently ignored.

val rename_column : t -> old_name:string -> new_name:string -> t

rename_column df ~old_name ~new_name is df with column old_name renamed to new_name.

Raises Not_found if old_name does not exist. Raises Invalid_argument if new_name already exists as a different column.

val select : ?strict:bool -> t -> string list -> t

select ?strict df names is the sub-dataframe with only the named columns, in the order given by names.

strict defaults to true: raises Not_found if any name is missing. When false, missing columns are silently skipped.

val reorder_columns : t -> string list -> t

reorder_columns df names is df with columns reordered so that names appear first (in that order), followed by any remaining columns in their original relative order.

Raises Not_found if any name in the list does not exist.

Row-wise operations

The Row module provides a declarative way to express computations over dataframe rows.

module Row : sig ... end

Row-wise computations using an applicative interface.

Row filtering and transformation

val head : ?n:int -> t -> t

head ?n df is the first n rows of df. n defaults to 5.

If n exceeds the number of rows, returns the entire dataframe.

val tail : ?n:int -> t -> t

tail ?n df is the last n rows of df. n defaults to 5.

If n exceeds the number of rows, returns the entire dataframe.

val slice : t -> start:int -> stop:int -> t

slice df ~start ~stop is the rows from start (inclusive) to stop (exclusive).

Raises Invalid_argument if start < 0, stop < start, or indices are out of bounds.

val take : t -> int array -> t

take df indices is the rows of df at the given 0-based indices.

Indices may repeat (to duplicate rows) and need not be sorted.

Raises Invalid_argument if any index is out of bounds.

val sample : ?n:int -> ?frac:float -> ?replace:bool -> ?seed:int -> t -> t

sample ?n ?frac ?replace ?seed df is a random sample of rows from df.

Exactly one of n or frac must be specified:

  • n: exact number of rows to sample.
  • frac: fraction of rows to sample (in [0;1]).
  • replace defaults to false.
  • seed: random seed for reproducible sampling.

Raises Invalid_argument if both n and frac are specified, neither is specified, frac is outside [0;1], or n > num_rows df when replace is false.

val filter : t -> bool array -> t

filter df mask is the rows of df where mask is true.

Raises Invalid_argument if Array.length mask differs from num_rows df.

val filter_by : t -> bool row -> t

filter_by df pred is the rows of df where pred is true.

Raises the same exceptions as the column accessors used in pred.

val drop_nulls : ?subset:string list -> t -> t

drop_nulls ?subset df is df with rows containing null values removed.

When subset is provided, only those columns are checked for nulls. Otherwise all columns are checked. A row is dropped if any checked column is null at that position.

val fill_null : t -> string -> with_value: [ `Float of float | `Int32 of int32 | `Int64 of int64 | `String of string | `Bool of bool ] -> t

fill_null df col_name ~with_value is df with null values in column col_name replaced by with_value.

The value type must match the column type. Raises Invalid_argument if the column does not exist or the types do not match.

val drop_duplicates : ?subset:string list -> t -> t

drop_duplicates ?subset df is df with duplicate rows removed, keeping the first occurrence.

When subset is provided, only those columns are considered for equality. Raises Not_found if any column in subset does not exist.

val concat : axis:[ `Rows | `Columns ] -> t list -> t

concat ~axis dfs is the concatenation of dfs.

  • `Rows: all dataframes must have the same columns; rows are stacked.
  • `Columns: all dataframes must have the same number of rows; columns are combined. Column names must be unique across dataframes.

Raises Invalid_argument if dfs is empty or the dataframes are incompatible for the chosen axis.

val map : t -> ('a, 'b) Nx.dtype -> 'a row -> ('a, 'b) Nx.t

map df dtype f is a 1D tensor of the given dtype obtained by applying f to each row of df.

val with_column : t -> string -> ('a, 'b) Nx.dtype -> 'a row -> t

with_column df name dtype f is df with a column name whose values are produced by applying f to each row. If a column named name already exists, it is replaced.

val with_string_column : t -> string -> string row -> t

with_string_column df name f is df with a string column name whose values are produced by f.

val with_bool_column : t -> string -> bool row -> t

with_bool_column df name f is df with a boolean column name whose values are produced by f.

val with_columns : t -> (string * Col.t) list -> t

with_columns df cols is df with the given columns added or replaced.

Raises Invalid_argument if any column length differs from num_rows df.

val iter : t -> unit row -> unit

iter df f applies f to each row of df for side effects.

val fold : t -> init:'acc -> f:('acc -> 'acc) row -> 'acc

fold df ~init ~f folds f over the rows of df with accumulator init.

Sorting and grouping

val sort : t -> 'a row -> compare:('a -> 'a -> int) -> t

sort df key ~compare is df with rows sorted by the values produced by key, ordered according to compare.

O(n log n) in the number of rows.

val sort_values : ?ascending:bool -> t -> string -> t

sort_values ?ascending df name is df with rows sorted by column name.

ascending defaults to true. Null values are always sorted to the end regardless of direction.

Raises Not_found if the column does not exist. O(n log n).

val group_by : t -> 'key row -> ('key * t) list

group_by df key is the list of (k, sub_df) pairs obtained by grouping the rows of df by the values produced by key.

The order of groups is not guaranteed. Rows within each group maintain their original relative order.

Column transforms

val cumsum : t -> string -> t

cumsum df name is df with column name replaced by its cumulative sum, preserving the column's dtype.

Raises Not_found if the column does not exist.

val cumprod : t -> string -> t

cumprod df name is df with column name replaced by its cumulative product, preserving the column's dtype.

Raises Not_found if the column does not exist.

val diff : t -> string -> ?periods:int -> unit -> t

diff df name ?periods () is df with column name replaced by the element-wise difference between consecutive values. periods defaults to 1.

Raises Not_found if the column does not exist.

val pct_change : t -> string -> ?periods:int -> unit -> t

pct_change df name ?periods () is df with column name replaced by the fractional change between consecutive values. nan where the previous value is zero. periods defaults to 1. Result column is always float64.

Raises Not_found if the column does not exist.

val shift : t -> string -> periods:int -> t

shift df name ~periods is df with column name shifted by periods positions. Positive shifts move values down (inserting nulls at the top), negative shifts move values up.

Raises Not_found if the column does not exist.

Column inspection

val is_null : t -> string -> Col.t

is_null df name is a boolean column where true indicates a null value at that position.

val value_counts : t -> string -> t

value_counts df name is a two-column dataframe with columns "value" and "count", containing the unique non-null values and their frequencies, sorted by count descending.

Aggregations

module Agg : sig ... end

Column-wise aggregation operations.

Joins and merges

val join : t -> t -> on:string -> ?right_on:string -> how:[ `Inner | `Left | `Right | `Outer ] -> ?suffixes:(string * string) -> unit -> t

join df1 df2 ~on ?right_on ~how ?suffixes () joins two dataframes on key columns.

on names the key column in df1. right_on names the key column in df2; defaults to on when both dataframes share the same column name.

Join types:

  • `Inner: rows where key exists in both dataframes.
  • `Left: all rows from df1, null-filled for missing df2 rows.
  • `Right: all rows from df2, null-filled for missing df1 rows.
  • `Outer: all rows from both, null-filled where missing.

Null keys never match (null != null). Duplicate column names receive suffixes (default: "_x", "_y").

Raises Not_found if a key column is missing. Raises Invalid_argument if key columns have incompatible types.

Pivot and reshape

val pivot : t -> index:string -> columns:string -> values:string -> ?agg_func:[ `Sum | `Mean | `Count | `Min | `Max ] -> unit -> t

pivot df ~index ~columns ~values ?agg_func () is a pivot table from df.

  • index: column whose values become row identifiers.
  • columns: column whose unique values become new column names.
  • values: column containing the data to fill the table.
  • agg_func defaults to `Sum for numeric, `Count for others.

Raises Not_found if any specified column does not exist. Raises Invalid_argument if values is incompatible with agg_func.

val melt : t -> ?id_vars:string list -> ?value_vars:string list -> ?var_name:string -> ?value_name:string -> unit -> t

melt df ?id_vars ?value_vars ?var_name ?value_name () unpivots df from wide to long format.

  • id_vars: columns to keep as identifiers (default: all non-value_vars).
  • value_vars: columns to melt (default: all non-id_vars).
  • var_name defaults to "variable".
  • value_name defaults to "value".

Raises Not_found if any specified column does not exist. Raises Invalid_argument if id_vars and value_vars overlap.

Converting

val to_nx : t -> (float, Stdlib.Bigarray.float32_elt) Nx.t

to_nx df is a 2D float32 tensor from the numeric columns of df.

Rows correspond to dataframe rows, columns to numeric dataframe columns (in order). All numeric types are cast to float32. Null values become nan. String and boolean columns are ignored.

Raises Invalid_argument if df contains no numeric columns.

Formatting and inspecting

val pp : ?max_rows:int -> ?max_cols:int -> Stdlib.Format.formatter -> t -> unit

pp ?max_rows ?max_cols ppf df formats df as a table on ppf.

max_rows defaults to 10. max_cols defaults to 10.

val to_string : ?max_rows:int -> ?max_cols:int -> t -> string

to_string ?max_rows ?max_cols df is df formatted as a table string.

val print : ?max_rows:int -> ?max_cols:int -> t -> unit

print ?max_rows ?max_cols df is pp ?max_rows ?max_cols Format.std_formatter df.

val describe : t -> t

describe df is a dataframe of summary statistics for the numeric columns of df.

Rows are: count, mean, std, min, 25%, 50%, 75%, max. String and boolean columns are ignored.

val cast_column : t -> string -> ('a, 'b) Nx.dtype -> t

cast_column df name dtype is df with column name converted to the numeric dtype.

Null values are preserved through the conversion.

Raises Not_found if the column does not exist. Raises Invalid_argument if the source column is not numeric.

val to_html : ?max_rows:int -> ?max_cols:int -> t -> string

to_html ?max_rows ?max_cols df is df formatted as an HTML table string.

Generates a <table> element with <thead> and <tbody>. Truncated rows and columns show ellipsis markers. A trailing <p> shows the shape when the table is truncated.

max_rows defaults to 20. max_cols defaults to 10.

val pp_display : Stdlib.Format.formatter -> t -> unit

pp_display ppf df formats df as a data URI containing an HTML table.

This printer is intended for notebook environments (Quill) where the data URI pattern is detected and rendered as rich HTML output. Uses fixed defaults (max_rows=20, max_cols=10). For custom limits, use to_html directly.

val pp_info : Stdlib.Format.formatter -> t -> unit

pp_info ppf df formats detailed information about df on ppf: shape, column names and types, null counts, and memory usage.

val info : t -> unit

info df is pp_info Format.std_formatter df.