docs menu

on this page

Module Rig_bnxt

Broadcom NetXtreme RoCE NICs, driven over PCI with no kernel driver.

The library takes a NIC's PCI function from Linux, as Rig_amd_pci takes a GPU's, and does what the kernel's bnxt_en and bnxt_re drivers would do. It resets the NIC, gives its firmware the context memory and command queue of its RoCE engine, and gives the NIC its address on the fabric. The NIC then moves bytes between memory of its machine and a peer's, by sends and receives on reliable-connected queue pairs over RoCE v2, and between its own regions by reads. The NICs it drives are the BCM57608 (14e4:1760).

Hardware status. The library is written, never run on hardware.

Two kinds of objects carry the work:

A region (Region) is memory the NIC writes and reads: system memory the library allocates, memory of the process pinned for the NIC, or another device's memory, such as a GPU's behind its BAR. A byte of a region is named by its offset in it.
A queue pair (Qp) is a send ring and a receive ring, connected to one queue pair of a peer. A send posts bytes of a region to the peer, where they land in the region of the receive the peer posted next. Each ring has its completions, one per entry, in the order of the entries. Posting is stores to memory and one store to the NIC's doorbell register, with no system call.

A transfer between two machines goes like this:

Each machine opens its NIC (open_), makes a queue pair on it and allocates the regions it sends from or receives into.
The machines exchange their queue pairs' endpoints (Qp.endpoint) by other means, and connect (Qp.connect).
The receiver posts a receive (Qp.receive), the sender a send (Qp.send), and each waits for its completion (Qp.wait).
Each closes its NIC (close).

ocaml

let nic = Result.get_ok (Rig_bnxt.open_ 0) in
let src = Result.get_ok (Rig_bnxt.Region.host nic 4096) in
let qp = Result.get_ok (Rig_bnxt.Qp.make nic) in
(* send Qp.endpoint qp to the peer, receive its endpoint [peer] *)
Result.get_ok (Rig_bnxt.Qp.connect qp peer);
Rig_bnxt.Region.write src 0 "hello";
Rig_bnxt.Qp.send qp src ~at:0 5;
match Rig_bnxt.Qp.wait qp Rig_bnxt.Qp.Send ~us:1_000_000 with
| Some (Ok _) -> print_endline "sent"
| Some (Error why) -> prerr_endline why
| None -> prerr_endline "not sent within a second"

The fabric

A NIC is known on the fabric by its MAC address and by the IPv4 address 10.x.y.z, whose x.y.z are the last three bytes of its MAC address. A queue pair sends to its peer's MAC address directly, so two NICs reach each other on one Ethernet segment, at an MTU of 4096 bytes. The library trusts the fabric. A packet with the right queue pair and sequence number lands in the memory its receive names.

Domains

Any domain may make any call, concurrently with other domains, with these exceptions:

Qp.send, Qp.read, Qp.poll and Qp.wait of one queue pair's Send side run one call at a time, and Qp.receive, Qp.poll and Qp.wait of its Receive side too.
Region.deregister must run after every call that names the region has returned, Qp.destroy after every other call on the queue pair has returned, and close after every other call on the NIC and its objects has returned.

The NIC's requests to its firmware, which the other calls make, run one at a time. open_, detach and attach hold a lock of rig.pci's for their whole run (Rig_pci.Driver), a reset of the NIC included, so a call of rig's from a callback on their thread meanwhile is made inside a lock and raises Invalid_argument (Rig's domains section).

open_, close, detach, attach, Region.host, Region.pages, Region.device, Region.deregister, Qp.make, Qp.connect, Qp.poll, Qp.wait and Qp.destroy, called from a callback inside a lock (Rig's domains section), raise Invalid_argument, having changed nothing (Errors).

Errors

A request the firmware or the system refuses returns Error msg, with msg naming the request and the reason. A firmware that does not answer a request in time may still act on it, so every later request of the NIC returns Error msg, and the memory the request named stays allocated until close. A send or receive that fails is reported by its completion. An access to the NIC never fails, as Errors states.

Misuse raises Invalid_argument, such as a call on a closed NIC or its objects, but for name, bus, Region.length, Qp.endpoint and Qp.room, a call on a destroyed queue pair but for those two, or a post on a deregistered region or an unconnected queue pair. A second Region.deregister or Qp.destroy calls nothing and returns what the first returned, and a second close does nothing.

References

InfiniBand Trade Association. InfiniBand Architecture Specification, Volume 1: reliable-connected queue pairs, packet sequence numbers and retries; Annex A17, RoCE v2.

NICs

NIC i of a machine is the ith of its BCM57608 functions in bus order, whichever driver holds each. The NICs are named "BNXT", "BNXT:1", "BNXT:2", ….

#type t

The type for NICs the process holds.

val buses : 
  ?machine:Rig_pci.Machine.t ->
  unit ->
  (string list, string) Stdlib.result

buses ~machine () is the bus addresses of the NICs of machine (defaults to Rig_pci.Machine.this), in bus order. It is Ok [] for a machine that lists no PCI functions, such as this one off Linux.

Error why if the machine's PCI functions cannot be listed.

#val count : ?machine:Rig_pci.Machine.t -> unit -> (int, string) Stdlib.result

count ~machine () is the number of buses, or their Error. A machine with no NIC is Ok 0.

#val open_ : ?machine:Rig_pci.Machine.t -> int -> (t, string) Stdlib.result

open_ ~machine i holds NIC i of machine (defaults to Rig_pci.Machine.this) and boots it. The NIC must be detached (detach), and the process needs what What an open needs lists for a function. It needs no firmware file, since the NIC boots from the firmware in its flash. Every open resets the NIC, whatever ran on it before, and turns relaxed ordering off for every request the NIC starts, so that it writes memory in order (Region). On a root complex where relaxed ordering raises the NIC's write bandwidth, the NIC loses that gain. close gives it back as the open found it (Changes to the machine). The process holds the NIC until close, or until it exits, which closes it.

The open gives the NIC's firmware system memory, which stays taken until close, for the queue pairs and regions the NIC holds at once. Beyond the queue pairs the firmware keeps for itself, it holds every one the firmware holds where the firmware states limits of its own, and otherwise at most 65,536, plus those the firmware destroys fast. Its regions are the firmware's region entries less its address vectors where the firmware states limits and one count of address vectors, and otherwise half the entries, at most 262,144. A queue pair or a region past them is an Error of the call that makes it.

Error msg, with msg starting with the NIC's name, if machine's PCI functions cannot be listed, if it has no NIC i, if a process holds it, if its function cannot be taken or reset, if its relaxed ordering cannot be turned off, if the machine lacks the system memory the firmware's context takes, as Rig_pci.Function.alloc_dma states (taken physically, free huge pages, vm.nr_hugepages), or if its firmware refuses a request or does not answer in time.

Raises Invalid_argument if i < 0.

#val name : t -> string

name nic is nic's name, such as "BNXT:1".

#val bus : t -> string

bus nic is the bus address of nic's PCI function, such as "0000:41:00.0".

#val close : t -> unit

close nic unregisters the process from nic's firmware, turns its bus mastering off, frees its system memory, regions included, turns its relaxed ordering back on if open_ found it on, and releases its function. The NIC reads and writes no memory of the process once close returns, and the next open resets it.

Regions

#module Region : sig ... end

A region is memory the NIC writes and reads. It is local to its NIC: a peer's sends write it only through a receive this process posted, and only queue pairs connected to queue pairs of its own NIC read it (Qp.read).

Queue pairs

#module Qp : sig ... end

A queue pair is a send ring and a receive ring connected to one queue pair of a peer. Each ring holds 4095 entries: Qp.room says how many more it takes, and an entry's slot is free once Qp.poll or Qp.wait has taken its completion. They take the completions of a ring in the order of its entries. A message, the bytes of one send or read, is at most 2³¹ bytes.

Changes to the machine

detach and attach change this machine as Rig_pci.Driver describes, and their changes persist after the process. Each refuses a NIC a process holds, this one included, and needs CAP_SYS_ADMIN and write access to the files under /sys/bus/pci they write. Each returns Error msg if the machine's PCI functions cannot be listed.

open_ turns the NIC's relaxed ordering off, and close turns it back on if the open found it on. A process that dies without closing the NIC leaves it off, for the kernel's drivers too after attach, until the next open and close or a reboot.

#val detach : int -> (Rig_pci.Driver.progress, string) Stdlib.result

detach i takes NIC i of this machine from its kernel driver as Rig_pci.Driver.detach does. Its network interfaces and RDMA devices go until attach or a reboot. It leaves the other functions of the NIC's device, such as a card's other ports, and so returns Error msg, naming them and their drivers, while the device has any and the NIC is not bound to vfio-pci: a NIC taken physically must be alone on its device.

Raises Invalid_argument if i < 0.

#val attach : int -> (unit, string) Stdlib.result

attach i gives NIC i of this machine back to its kernel driver as Rig_pci.Driver.attach does.

Raises Invalid_argument if i < 0.