Module Rig_mlx5_abi

ConnectX NICs' formats, as values.

A process and a ConnectX NIC (the mlx5 family) share five formats in memory: the work entries a queue pair's send ring holds, the receive entries its receive ring holds, the completions a completion queue's ring holds, the doorbell records that count them, and the doorbell registers of the NIC's access regions, whose stores tell the NIC that work waits. Each value here describes bytes, and its caller writes or reads them. Nothing here acts on a NIC.

 send ring: n entries of 64 bytes        doorbell record    access region
 | entry i | entry i+1 | ...   --post-->  send count  --ring-->  register
                                                                   |
 completion ring: m entries of 64 bytes  <--- the NIC writes ------'
 | completion j | ...           --poll-->  consumed count

A process writes entries at its ring's producer count, stores the count in the queue pair's doorbell record (Doorbell), then stores the first 8 bytes of the last entry in the queue pair's doorbell register (Uar). The NIC runs the entries in order, and writes a completion for each entry that asks for one and for every entry that fails. A process posts receives by writing entries at its receive ring's producer count and storing the count in the queue pair's doorbell record, with no store to a register. The NIC fills the oldest posted receive with each message the peer sends, and writes a completion for every receive. The process reads a completion once Completion.owned says the NIC has written it, and stores how many it read in the completion queue's doorbell record.

Conventions

Every field the NIC reads or writes is big-endian, and the encoders and decoders here read and write it so. Data carried inline is copied as given.

Misuse raises Invalid_argument: an integer outside the range its function states, or bytes outside the buffer.

The library has no state, so any domain may use any of its values.

References

rdma-core v56.0's providers/mlx5/mlx5dv.h and its mlx5dv manual pages: work entries, completions and doorbell records, as rdma-core publishes them for direct access to the NIC.

type buffer =
  (char, Stdlib.Bigarray.int8_unsigned_elt, Stdlib.Bigarray.c_layout)
    Stdlib.Bigarray.Array1.t

The type for memory a process and a NIC share, such as a ring.

#type ring =

#| Send
(*
The send ring, of work entries (Entry).
*)
#| Receive
(*
The receive ring, of receive entries (Receive).
*)

The type for a queue pair's rings.

#module Entry : sig ... end

A work entry is one operation of a queue pair's send ring.

#module Receive : sig ... end

A receive entry is one receive of a queue pair's receive ring: the memory the next message from the peer lands in.

#module Completion : sig ... end

A completion is what the NIC writes once a work entry completes or fails.

#module Doorbell : sig ... end

A doorbell record holds a count that a process stores for the NIC in its memory.

#module Uar : sig ... end

An access region (UAR) is a page of the NIC through which a process rings doorbells.