Module Tolk_nn.Layer_norm

Layer normalization over the last axis.

type t = {
  1. weight : Tolk_frontend.Tensor.t option;
    (*

    Per-feature scale, shape (dim).

    *)
  2. bias : Tolk_frontend.Tensor.t option;
    (*

    Per-feature shift, shape (dim).

    *)
  3. eps : float;
  4. dim : int;
}

A layer-norm layer. weight and bias are None when the layer was created without an elementwise affine transform.

val create : ?eps:float -> ?elementwise_affine:bool -> int -> t

create dim is a layer norm over a last axis of size dim. eps (default 1e-5) stabilises the variance denominator. elementwise_affine (default true) adds a learned per-feature scale (initialised to ones) and shift (initialised to zeros).

apply ln x normalises x along its last axis to zero mean and unit variance (biased estimator), then applies the affine transform when present.

  • raises Invalid_argument

    if the last axis of x does not have size dim.