docs menu

Module Rig_bnxt.Qp

A queue pair is a send ring and a receive ring connected to one queue pair of a peer. Each ring holds 4095 entries: Qp.room says how many more it takes, and an entry's slot is free once Qp.poll or Qp.wait has taken its completion. They take the completions of a ring in the order of its entries. A message, the bytes of one send or read, is at most 2³¹ bytes.

A queue pair fails when one of its entries fails, when the NIC reports its failure, when its completions hold one of a kind its ring does not take, or when the NIC's machine fails. The NIC then runs none of its entries that have no completion, of either ring, nor those posted later; after a completion of an unknown kind, it may run them until Qp.destroy. Completions keep the order of the entries: those the NIC completed before the failure keep their result, the entry that failed carries its error, and the rest complete with the failure's Error. A failed entry may have moved some or all of its bytes. A failed queue pair stays failed. Qp.poll and Qp.wait send the firmware no request, a failure's included.

#type nic := t

#type t

The type for reliable-connected queue pairs.

#type endpoint = {

#qp : int;
(*
The queue pair's number.
*)
#psn : int;
(*
The packet sequence number its sends start at.
*)
#gid : string;
(*
Its NIC's 16-byte RoCE v2 address: the IPv4 address 10.x.y.z mapped into IPv6 (The fabric).
*)
#mac : string;
(*
Its NIC's 6-byte MAC address.
*)

}

The type for what a peer needs to connect to a queue pair. It is plain data, which a program sends to the peer by any means.

#type side =

#| Send
(*
The sends, and their completions.
*)
#| Receive
(*
The receives, and their completions.
*)

The type for a queue pair's rings.

#val make : nic -> (t, string) Stdlib.result

make nic is an unconnected queue pair of nic, whose sends start at a random packet sequence number.

Error msg if the machine has no memory for its rings or the firmware refuses it. A refused make gives back what it took, but for memory a firmware that stopped answering may still name (Errors).

#val endpoint : t -> endpoint

endpoint qp is what a peer's queue pair connects to qp with.

#val connect : t -> endpoint -> (unit, string) Stdlib.result

connect qp e connects qp to the peer's queue pair e and makes it ready to send. A send to the peer is retried while the peer has posted no receive, without end, and a send the peer does not acknowledge is retried 7 times, each after 67 ms, before its completion fails.

e may be the endpoint of a queue pair of qp's own NIC, qp's own included. qp then serves its peer's reads (read), which take bytes of this process's regions.

Error msg if the firmware refuses a change of qp's state. A connect the firmware refused part-way, or an exception ended, called again, takes qp back to reset and connects it from there.

Raises Invalid_argument if qp is connected, or e's gid is not 16 bytes or its mac not 6.

#val room : t -> side -> int

room qp side is how many more entries side's ring of qp takes.

#val send : t -> Region.t -> at:int -> int -> unit

send qp r ~at n posts the send of the n bytes at offset at of r to qp's peer, which lands them in the region of its next receive.

Raises Invalid_argument if qp is not connected, r is not a region of qp's NIC or is deregistered, the bytes do not lie in r, n is over 2{^31}, or room qp Send is 0.

#val read : t -> src:(Region.t * int) -> dst:(Region.t * int) -> int -> unit

read qp ~src:(s, i) ~dst:(r, at) n posts the read of the n bytes at offset i of s into the n bytes at offset at of r, on qp's Send ring, whose peer serves it.

Raises Invalid_argument as send does for r, and if s is not a region of qp's NIC or is deregistered, the bytes do not lie in s, or qp's peer is not a queue pair of qp's NIC.

#val receive : t -> Region.t -> at:int -> int -> unit

receive qp r ~at n posts the receive of up to n bytes at offset at of r, which the next send of qp's peer fills. A send longer than n fails the queue pair, and its peer's.

Raises Invalid_argument as send does, for room qp Receive.

#val poll : t -> side -> (int, string) Stdlib.result option

poll qp side takes the completion of the oldest entry of side whose completion has not been taken, if it has one. It is Some (Ok n) if the entry completed, with n the bytes it moved: a send's bytes, or the length of the message a receive took. It is Some (Error msg) if the entry failed or the queue pair has, with msg saying why, and None if the entry has no completion yet or no entry is due.

#val wait : t -> side -> us:int -> (int, string) Stdlib.result option

wait qp side ~us is poll once the oldest due entry of side has a completion, waiting for it at most us microseconds. It is None at the bound, the entry still due. A long wait sleeps between looks.

Raises Invalid_argument if us < 0 or no entry of side is due.

#val destroy : t -> (unit, string) Stdlib.result

destroy qp destroys qp and frees its rings. The NIC runs none of its entries and writes no memory for it once destroy returns Ok (). The entries of its peer's queue pair fail once their retries end.

It is Error msg if the firmware refuses, and so is every call after one that an exception ended. qp then counts as destroyed, and its rings stay allocated until close.