github ↗
docs menu
on this page

Module Rig_mlx5

ConnectX NICs of this machine, opened through Linux's RDMA subsystem.

Here a NIC is one open port of a ConnectX NIC (the mlx5 family). The kernel's mlx5_ib driver owns the NIC and shares it with other programs. open_ opens the NIC through its user verbs file, /dev/infiniband/uverbsN, and has the kernel make the NIC's objects with the verbs ioctls. The NIC then moves bytes between this process's memory and a peer's by RDMA writes and reads, and by sends into the receives the peer posted, on reliable-connected queue pairs.

This library is untested on real ConnectX NICs.

Three kinds of objects carry the work:

  • A region (Region) is host memory of this process, or a device's memory exported as a dma-buf, that the NIC may read or write. Work entries name a region's bytes by the keys the NIC gave it.
  • A completion queue (Cq) is a ring the NIC writes completions in, which the process polls.
  • A queue pair (Qp) is a send ring the process writes work entries in, and a receive ring of the receives it posts if it was made with one, connected to one queue pair of a peer. Posting is stores to memory and one store to the NIC's doorbell register, with no system call. The NIC runs a queue pair's entries in order, and their writes land in that order in a region registered In_order (Region.order).

A transfer between two processes goes like this:

  1. Each process opens its NIC (open_), makes a completion queue and a queue pair on it, and registers the memory it shares (Region.register).
  2. The processes exchange their queue pairs' endpoints (Qp.endpoint) and their regions' remote addresses (Region.remote) by other means, and connect (Qp.connect), each saying what its peer may do through its queue pair.
  3. A process posts entries (Qp.post), rings the doorbell (Qp.ring) and polls the completion queue (Cq.poll), or arms it (Cq.arm) and waits (wait). A process its peer sends to posts receives (Qp.receive) for the messages.
  4. It destroys the queue pair, then the completion queue and the regions, or closes the NIC (close).
ocaml
let nic = Result.get_ok (Rig_mlx5.open_ "mlx5_0") in
let cq = Result.get_ok (Rig_mlx5.Cq.make nic 256) in
let qp = Result.get_ok (Rig_mlx5.Qp.make cq ~receives:0 128) in
(* send Qp.endpoint qp to the peer, receive its endpoint [peer]; the
   peer connects its queue pair with [Remote_write] *)
Result.get_ok (Rig_mlx5.Qp.connect qp Local peer);
(* [src] is Region.local of a region here, [dst] the peer's
   Region.remote *)
Rig_mlx5.Qp.post qp
  Rig_mlx5_abi.Entry.{ op = Write { src; dst }; signal = true };
Rig_mlx5.Qp.ring qp

The kernel makes every object, and the process maps the rings, the doorbell records and the NIC's access regions. When the process's file of the NIC closes, at close or at the process's death, the kernel destroys every object of the NIC, its regions included, so nothing of the NIC outlives the process.

Domains

Any domain may make any call, concurrently with other domains, with these exceptions:

  • Qp.post and Qp.ring of one queue pair run one call at a time, and so does Qp.receive.
  • Cq.poll and Cq.arm of one completion queue run one call at a time.
  • wait of one NIC runs one call at a time.
  • destroy and deregister must run after every other call on their object has returned, and close after every other call on the NIC and its objects has returned.

A queue pair may be posted while another domain polls its completion queue.

close, Region.register, Region.deregister, Cq.make, Cq.destroy, Qp.make, Qp.destroy and wait, called from a callback inside a lock (Rig's domains section), raise Invalid_argument, having changed nothing (Drivers).

Errors

A request the kernel refuses returns Error msg, with msg naming the call and the kernel's reason. A completion queue or queue pair larger than the NIC holds is also Error msg, with msg naming the bound. A queue pair or completion queue that fails while it runs is reported by wait, and an entry that fails by its completion (Rig_mlx5_abi.Completion.status).

Misuse raises Invalid_argument: a call on a destroyed completion queue or queue pair, on a deregistered region, or on any object of a closed NIC. A second destroy or deregister on an open NIC calls nothing and returns what the first returned, or Error msg if an exception ended the first, whose object then counts as destroyed or deregistered and keeps what the kernel might still hold, as after a refusal. A second close does nothing.

A child of fork must not use its parent's NICs or their objects. Their rings are not mapped in the child.

References

  • InfiniBand Trade Association. InfiniBand Architecture Specification, Volume 1: reliable-connected queue pairs, packet sequence numbers, acknowledgement timeouts and retries; Annex A17, RoCE v2.
  • rdma-core's providers/mlx5/mlx5dv.h and its mlx5dv manual pages: the formats the process and the NIC share (Rig_mlx5_abi).
  • The Linux kernel's Userspace verbs access documentation: the verbs files and the pinning of registered memory.

NICs

A NIC is named as the kernel names its RDMA device, such as "mlx5_0". A ConnectX function has one port, and this library opens port 1.

The machine's files. names and open_ read this machine's files: the RDMA devices under /sys/class/infiniband and /sys/class/infiniband_verbs, and the verbs files under /dev/infiniband. Off Linux this machine has none of these files, so names () is Ok [] and open_ returns Error.

Privileges. Opening needs read and write access to the NIC's verbs file and no other privilege. Registering memory locks its pages, so the process's locked-memory limit (RLIMIT_MEMLOCK) bounds the bytes it registers unless the process has CAP_IPC_LOCK.

Kernels. Registering dma-buf memory needs Linux 5.12 or later.

#type t

The type for open NIC ports. Each has one protection domain, which its regions and queue pairs share.

#val names : unit -> (string list, string) Stdlib.result

names () is the names of this machine's RDMA devices whose PCI function the mlx5_core driver holds, sorted as strings, so "mlx5_10" comes before "mlx5_2". It is Ok [] where /sys/class/infiniband does not exist, and Error msg if it cannot be read, msg naming it and the reason. Listing them changes nothing on the machine.

#val open_ : string -> (t, string) Stdlib.result

open_ name opens port 1 of the NIC name and maps its access regions, the NIC's pages through which the process rings doorbells.

The result is Error msg if:

  • no RDMA device name exists, or mlx5_core does not hold its PCI function;
  • it has no verbs file, or its verbs file cannot be opened, msg naming the file and the reason;
  • the kernel refuses a call, msg naming the call and the reason;
  • port 1 is not active, or is an Ethernet port with no RoCE v2 global identifier;
  • the kernel gives fewer doorbell registers than one system page holds.
#val name : t -> string

name nic is the kernel's name of nic, such as "mlx5_0".

#val bus : t -> string

bus nic is the bus address of nic's PCI function, such as "0000:17:00.0".

link nic is the link layer of nic's port. On `Ethernet (RoCE v2) every packet carries a global route header.

#val max_message : t -> int

max_message nic is the most bytes one message of nic's port carries, as the kernel reported at open_: the most one write, read or send moves.

#val close : t -> unit

close nic destroys every object of nic and closes it. The NIC reads and writes no memory of this process once close returns. Later calls on nic and its objects raise Invalid_argument, except name, bus and link, which still return, and close, which does nothing.

Regions

#module Region : sig ... end

A region is memory the NIC may read or write.

Completion queues

#module Cq : sig ... end

A completion queue is a ring where the NIC reports the entries that completed.

Queue pairs

#module Qp : sig ... end

A queue pair is a send ring, and a receive ring if it was made with one, connected to one queue pair of a peer.

Events

#type event =
  1. #| Completion of Cq.t
    (*

    An armed completion queue holds a completion.

    *)
  2. #| Failure of string
    (*

    The NIC failed, a queue pair failed, or a completion queue overflowed, as the message names.

    *)
  3. #| Port of string
    (*

    The port's state changed, as the message says.

    *)

The type for events of a NIC.

#val wait : t -> ms:int -> event list

wait nic ~ms is the events of nic since the last wait, once there is one, or [] if none came within ms milliseconds. An armed completion queue raises one Completion per Cq.arm. Other domains run while it waits.

Raises Invalid_argument if ms < 0.