Module Talon.Expr

The shape of expressions with one value per row of their frame.

The shape of expressions with one value per frame.

The type for expressions of shape 's whose values read as 'a.

The type for outputs of shape 's: named expressions, as select, derive and aggregate take them.

name := e outputs e as the column name. It binds more loosely than every operator, so "late" := delay > float 15. needs no parentheses. e needs a column type: a const or ($) result needs store, and an option result is never one.

Raises Invalid_argument if name is not valid UTF-8.

keep sel outputs the columns that sel selects, unchanged and under their names. It keeps a column of any type, extensions included.

across k sel f is the outputs f n (Col.v k n) for each name n that sel selects, in order: across Kind.float Sel.(prefix "wk") (fun n x -> n := over (rank x)). A selected column that k does not bind is a problem: narrow sel with Sel.of_kind. f runs when the verb is applied and must be pure.

The type for functions of one column of any type.

each sel { column } is the outputs column n x for each name n that sel selects, in order, where x reads the column n at its own type, extensions included: each Sel.all { column = (fun n x -> n := rows - count x) }. column is polymorphic, so it applies only operations that take every type; each is checked against the column's type when the verb is applied: sum of a string column is a problem. An extension column is read without its declaration, so operations that order its values (min, (<), rank, …) or compute with them (sum, mean, …) are problems, and those that count, move, select or compare them for equality apply. column runs when the verb is applied and must be pure.

int n is the integer n.

float x is the float x. It may round to the type it takes.

bool b is the boolean b.

string s is the text s.

Raises Invalid_argument if s is not valid UTF-8.

instant t is the instant t.

wall t is the wall clock t.

span d is the span d.

date d is the date d.

null is null, typed by what it meets.

a + b is the sum of a and b, wrapping on overflow as nx's integers do. Its type is the operands' common type, as for (-), (*), (/) and (mod).

a - b is the difference of a and b.

a * b is the product of a and b.

a / b is the quotient of a by b, truncated toward zero, and null where b is zero.

a mod b is the remainder of a by b, of a's sign, and null where b is zero.

a +. b is the IEEE 754 sum of a and b. Its type is the operands' common type, as for (-.), (*.), (/.) and (**).

a -. b is the difference of a and b.

a *. b is the product of a and b.

a /. b is the quotient of a by b.

a ** b is a to the power b.

a = b is true iff a and b are equal in talon's total order (Type.compare_value), so nan = nan and -0. = 0., and null if either is null. The operands meet at their common type.

a <> b is not (a = b).

a < b is true iff a comes before b in talon's total order, so x > float 15. holds for NaN, and null if either is null. Operands of an extension type need a declaration made with ~ordered:true.

a > b is b < a.

a <= b is a < b || a = b.

a >= b is b <= a.

a && b is Kleene's conjunction: false if either is false, else null if either is null. Both operands are computed.

a || b is Kleene's disjunction: true if either is true, else null if either is null.

not a is the negation of a, null where a is.

if_ c a b is a where c is true and b where c is false or null. a and b meet at their common type.

is_null a is true where a is null and false elsewhere. It is never null.

coalesce es is the first of es that is not null, or null. es meet at their common type.

is_in vs a is true iff a is the same key as one of vs (see Type.compare_value), so it is false, never null, where a is null: not (is_in vs a) is true there, whereas not (a = v0 || a = v1) is null. a's type must hold each of vs; an extension's values are encoded with its declaration.

cut edges a is the bin of a among edges, as int64. The bin is i where edges.(i) <= a < edges.(i + 1) in talon's total order, so k + 1 edges bound k bins, each holding its lower edge, and adjacent bins share an edge. It is null where a is null, below the first edge, or at or above the last, which NaN is: cut [| 0.; 0.5; 1. |] x is 0 for 0.2, 1 for 0.5 and null for 1..

Each edge is a literal of a's type, which must hold it and may round it, so cut edges a = int i exactly where a >= edges.(i) && a < edges.(i + 1). As that type stores them, the edges must ascend strictly and hold no NaN; -0. and 0. are one edge. a's type is one that (<) orders. edges is copied.

Raises Invalid_argument if edges has fewer than two elements.

cast ty a converts a's values to ty:

• between bool, integer and float types: integers take exact values only, so a fractional, infinite or NaN float, or a value out of range, is a data error; floats round to nearest; bool takes 0 and 1 only, and gives 0 and 1;
• between string and categorical types: the text is kept, and a categorical takes only the strings of its dictionary, and another string is a data error;
• between datetimes of one clock, between durations, and between clocks: values are kept, and a value that is not a whole number of the new unit is a data error;
• between clocks and durations: a time of day is its span from midnight, converted as between durations, and a duration outside [0, 1 day) is a data error for a clock;
• between a duration and int64: the integer is the duration's ticks in its own unit, exactly, for every int64. To read another unit, cast between durations first, as cast Type.int64 (cast (Type.duration Ms) d) does for milliseconds;
• between dates and wall-clock datetimes: a date becomes the datetime of its midnight, and a datetime becomes the day it falls on, so 1969-12-31T23:59:59 becomes 1969-12-31. A value out of the new type's range is a data error;
• lists element by element, records with the same field names field by field, and tensors of the same shape element by element, as Nx.cast does.

Any other pair is a problem that names the function that converts it, if one does: Str.parse and Temporal.parse for text to values, and Temporal.format for dates, clocks and datetimes to text, and Temporal.wall or Temporal.localize between the clocks. An instant falls on a day only in a zone, so it casts neither to nor from a date. a needs a column type.

The type for elementwise nx functions of one argument that preserve its dtype, such as Nx.exp, on host values as columns hold.

nx { f } a applies f to a's values with nx's semantics, IEEE's for floats: nx { f = Nx.exp } x. a has an nx dtype: an integer, float or bool type, bool computing as Nx.bool. When the verb is applied, talon calls f once on a traced value of that dtype and records the operations f performs; an f that moves, reduces or reshapes its argument is a problem, and so is one that ignores it, which would lose its nulls.

The type for elementwise nx functions of two arguments of one dtype that preserve it, such as Nx.atan2.

nx2 { f2 } a b is like nx for two arguments, which meet at their common type: nx2 { f2 = Nx.atan2 } y x.

rows is the number of rows of the frame, as int64.

count a is the number of non-null values of a, as int64.

sum a is the sum of a's values: int64 over integers, a's type over floats and durations. Integer sums wrap as nx's integers do; a duration sum that overflows is a data error. Other types are a problem.

min a is the least of a's values in talon's total order, of a's type. An extension type needs an ordered declaration.

max a is the greatest of a's values, like min.

first a is a's first non-null value in frame order.

last a is a's last non-null value in frame order.

only a is a's one distinct non-null value, and a data error naming the frame when a has several.

mean a is the arithmetic mean of a's values, as float64: their sum divided by their count. Integers of at most 32 bits sum exactly in int64 in a frame of fewer than 2³¹ values, and the sum rounds once to float64; other values sum in float64 in row order. a is an integer or float expression, as for std, var, median and quantile; other types are a problem.

std a is the sample standard deviation of a's values, dividing by n - 1, and null for fewer than two values. It is infinite only where the standard deviation exceeds max_float. Its bits depend on the frame's values in row order alone. Over traced values, and under Rune.jit, it is a two-pass form, equal within rounding.

Raises Invalid_argument over a float64 operand under Rune.jit, or over traced float64 values under Rune.vmap, as that form reads them to detect overflow.

var a is the sample variance of a's values, like std. It is infinite only where the variance exceeds max_float.

median a is quantile 0.5 a.

quantile p a is the p-quantile of a's values, interpolating linearly between the two nearest ranks.

Raises Invalid_argument if p is not in [0;1].

n_unique a is the number of distinct keys of a, null being one key, as int64.

arg_min a is the zero-based position in the frame of a's first least value, as int64. An extension type needs an ordered declaration.

arg_max a is the position of a's first greatest value, like arg_min.

pivot key v is, per frame, one record with a field per category of key's categorical type, in dictionary order, named as the category. The field holds v at the frame's row whose key is that category, and is null where the frame has no such row. A null v in a cell is a null field, so a missing cell and a null value look alike; a bool true pivoted beside v tells them apart.

The record's type, and the plan's schema, follow from key's type alone and never from its values: cast key to the categorical naming the fields, or read them from the data with Kit.categorize. Every non-null value of v lands in a cell of its row, or the run fails: a row with a null key and a null v holds no cell, so a left join's unmatched row leaves its frame's fields null; a null key with a non-null v has no field to hold the value and is a data error, as is any value outside the dictionary at the cast. Two rows of one frame with one key are a data error; reduce them first with Query.aggregate on the frame's keys and key. Filtering before pivot removes the groups left with no row.

A data error reports the failing frame's first row, naming the key.

Cost: O(n + g k) for n rows, g frames and k categories. Under over or window a frame's record holds its k cells, so the rows together hold their frames' k cells.

Its problems are a key whose type is not categorical, and a v with no column type, or none reachable. aggregate with it blocks, holding the columns it reads. Raises Invalid_argument under Rune.jit, and under Rune.vmap over traced keys (Compiled, mapped and differentiated runs): its failures read them.

collect a is a's values in frame order as a list of a's type, with one element per row of the frame and a null element for a null value. It is never null, and is the empty list over no rows. a must have a column type, and an extension column is read with each.

Cost. O(n). Under over each row holds its partition's list, and the lists together hold the sum of the partitions' squared sizes. Under window every row holds its window's list, and memory holds them all together.

Raises Invalid_argument under Rune.jit (Compiled, mapped and differentiated runs).

over ~by ~order e is e evaluated in each partition of the enclosing frame (Frames). Its type is e's. by defaults to no columns, which is one partition. order defaults to none, which is the frame's order.

At the top of derive or filter the frame is the whole input, so over (mean x) is the input's mean and it blocks the pipeline (Query).

Its problems are a column named twice in by or in order, and an order key on a column whose type is or holds an extension type.

Raises Invalid_argument over traced columns (Compiled, mapped and differentiated runs) if by is nonempty.

shift n a is a's value n rows earlier in the frame, or -n rows later if n is negative, and null past the frame's edges.

rank a is the 1-based rank of a's value among the frame's non-null values in talon's total order, ties taking the lowest rank (SQL's RANK), as int64, and null where a is null. An extension type needs an ordered declaration.

window ~on ~before ~after ~exclude_key r is, at each row, r over the row's window (Windows). Its type is r's. Without before the window has no lower limit, and after defaults to 0. With ~exclude_key:true it leaves out the rows whose key is ki, which without on is the row itself.

window (sum x) is the running sum, and window ~on:ts ~before:(span (Time.Span.days 7)) ~exclude_key:true (sum x) sums the seven days before ts. Without on the key is the row's place, so over ~order sets it, as in over ~order:[ Order.asc "day" ] (window ~before:(int 28) (mean x)), the mean of a day and the 27 before it.

It blocks the pipeline as over does. Keys and bounds and Float keys and results give its keys and bounds, and Cost its cost.

Its problems are an on of another type than Keys and bounds allows, a bound of another kind, type or unit, and over, shift, rank or window inside r, outside a per_list's body.

per_list ls r is, at each row, r over the frame of the elements of the lists that the outputs ls compute (Lists). Its type is r's. A list's norm is:

let xs = Col.v Kind.(list float) "xs" and x = Col.float "x"
let norm = per_list [ "x" := xs ] (sum (x *. x)) ** float 0.5

per_list [ "t" := spikes ] (mean (t -. shift 1 t)) is a spike train's mean interval. Nulls and floats gives its nulls and float results, and Cost its cost.

Where its outputs hold no over, shift, rank or window, its value at a row reads that row alone, so it streams. It may stand in a window's operand. A failure in r is at the row of its list.

Its problems are a hiding output (Outputs), an output that is not a list, no output, two outputs of one name, and an over, shift, rank or window in r that reads no output.

Raises Invalid_argument under Rune.jit when it reads a row's list, and under Rune.vmap where the lists' offsets or validity are traced (Compiled, mapped and differentiated runs).

list es is the list of es' values at each row, in order, as a list of the type where the types of es meet (Type.common), with a null element for each null value. It is never null: list [ x; null ] where x is null is a list of two null elements. A literal or null element takes that type, and store gives one where nothing does. A list operand stays one element, so a list of lists nests and never flattens.

It computes under Rune.jit and Rune.vmap, since the lists' sizes follow from the columns' lengths, but reading a list raises under Rune.jit as for any list column: in a result, and under per_list and Query.unnest (Compiled, mapped and differentiated runs).

Its problems are an empty es, and operands that do not meet.

record os is the record of the outputs os, its fields named as they are, in order: record [ "x" := x; keep (Sel.names [ "flux" ]) ]. It is never null, even where every field is. if_ c (record os) null is null where c is not true.

Its problems are those of binding os, and two outputs of one name.

field k n r is the field n of r read as k, null where r is null or the field is: field Kind.float "flux" band. k binds the field as a Col handle binds a column, so field reads fields of types without an extension, and within reads any.

Its problems are a field that r lacks and one that k does not read.

within r o is the outputs o bound over r's fields, so that inside o a name reads the field of that name and no column of the frame, null where r is null or the field is. within r (keep Sel.all) spreads a record, within r (each Sel.all { column = (fun n x -> "b_" ^ n := x) }) renames, and within r ("snr" := Col.float "flux" /. Col.float "err") computes. To combine a field with a column of the frame, read the field with field outside within.

Its problems are those of binding o against r's fields, a name that r lacks included, an r that is not a record, and, under Query.derive and Query.unnest, an output named as a column of the input, which would replace that column.

const v is v on every row. It is typed by what it meets, an extension's value being encoded with its declaration; a type that does not hold v is a problem.

f $ a applies f's function to a's value, once per row where no argument is null, and is null elsewhere: const mk $ carrier $ delay. An argument is decoded with its column type, so it needs one, unless it is itself an OCaml value: a const, a $ result or an option. The result is typed by what it meets, or by store.

option a is Some v where a is v and None where a is null. It is never null, and is an argument of ($) or what Query.values decodes. a needs a column type.

of_option a is v where a is Some v and null where it is None. It is typed by what it meets, or by store.

store ty a is a typed as ty, as if a met an operand of type ty: a literal, null, const, $ or of_option result takes ty, and an a of a type that ty contains is widened to it. A literal or const value that ty does not hold is a problem, and a $ or of_option result that it does not hold a data error.

Text.

Temporal values.

hash ~seed a is the hash of each row of a (Hashing), as int64, and is never null. A column whose bytes nx cannot read, a traced one other than a categorical, raises Invalid_argument naming Expr.hash (Compiled, mapped and differentiated runs).

pp ppf e formats e as it is written inside Expr.( … ), with fewest parentheses: (amount -. over (mean amount)) /. over (std amount). Besides:

• a handle formats as its column name, quoted as an OCaml string unless it is an OCaml lowercase identifier that is neither a keyword nor a value of this module;
• a literal formats as its value: 15, 15., nan, "text", true, 2024-03-15, 2024-03-15T09:30:00, 15m;
• an OCaml value of const formats as <const>, and the functions of nx and nx2 as <fn>;
• an is_in list and cut edges format with their operand's type once bound, and as […] before.