Module Rig_nv

A device of this library is one NVIDIA GPU. The GPU's resource manager (RM) is the part of NVIDIA's software that makes the GPU's objects, such as its channels, its memory and its address spaces, for its clients. A path is a library that reaches the RM one way and gives this library what it reached, as a path. Rig_nv_nvidia reaches it through NVIDIA's kernel driver, and Rig_nv_pci through the GPU's own system processor, with no kernel driver. Each path's opener calls make.

The device runs its work on two hardware channels, one for compute and one for copies. A channel fetches 64-bit entries from its ring, the GPFIFO, in order, and runs the 32-bit words of the segment each entry names (Rig_nv_abi.Method). rig hands the device a submission through the two C functions of rig_nv.h, rig_nv_room and rig_nv_submit. rig_nv_submit writes the submission's parts into the channels' rings and wakes the channels through the GPU's doorbell register, with stores to memory alone. Each submission ends with a release that writes its value into the timeline word.

Rig.submit -> rig_nv_room, rig_nv_submit      C, stores only
                  |                |
                  v                v
          compute channel     copy channel    ring entries name words
                  |                |
                  +-- release v ---+
                          |
                          v
                    timeline word             read by Rig_nv.sleep

A device lives from make, which allocates its channels and their memory, until rig calls stop, which frees the channels and asks the path to let go of the GPU. Between them rig submits, waits in sleep, allocates and frees memory, and loads images.

Any domain may call any function, at the same time as others, within the rules rig keeps for every driver (Rig_edge.Driver). A C caller other than rig must call rig_nv_room before each rig_nv_submit, with no other call of either on the device between them. A function that calls the RM returns the RM's refusal of its arguments as its result, None or Error, and raises Fault for any other failure; signaled never raises it, and make returns it as Error. A path's free that fails keeps the memory it was given, and the call that freed raises its Fault, grow also where it would answer Short. A call that fails for another reason reports that failure in the Fault's place. make returns its own Error, and grow raises its own exception. stop raises a free's Fault once it has stopped the device and given back every other memory. The memory an RM object names stays taken while the RM keeps the object, or while the path keeps the object's channel registered. A Fault from the RM's free or the path's unregister passes on as a free's does, but stop drops it and asks the path to stop (Faults and stopping).

The capability's local and the paths' openers, called from a callback inside a lock (Rig's domains section), raise Invalid_argument, having changed nothing (Drivers). Locks nest in this order, and a take out of it raises Invalid_argument, having taken nothing: an image's, which entry and unload hold; a device's local memory lock, which entry and stop hold; and a path's (Lock).

Compiled code finds the record its launch descriptors depend on under Rig_nv_abi.key:

let d = Result.get_ok (Rig.open_ (Rig_nv_nvidia.gpu 0)) in
let g = Option.get (Rig.capability d Rig_nv_abi.key) in
Printf.printf "%s, %d SMs\n" (Rig.arch d)
  (g.gpcs * g.tpcs_per_gpc * g.sms_per_tpc)

The C functions states how rig hands a driver work.

Driver

Facts

The device maps the 64 KiB of rig's array of words that holds its word through the path (map_host), so that its channels write it. On a path that maps no host memory, or when the word make was given lies outside the array, the word is eight bytes of the device's own Pinned memory, which starts at the given word's value and which rig reads through signaled. That word, in a page of the device's own, goes back when rig frees it (Rig_edge.facts). The device's values start after that value. The word holds, as an unsigned 64-bit integer in the host's byte order, the last value v such that the work of every value up to v has completed. A channel writes it once its work has completed and that work's writes are visible to the system. A read returns the value before a release or the value after it. This assumes the copy engine writes a 64-bit release as one store. The capability's local grows the local memory the compute channel gives kernels (Rig_nv_abi.Gpu.t.local). A growth takes effect at the next submission that uses the compute channel, after its waits and before its parts. After stop, local returns Error. The record also holds the classes and counts the path read of the GPU, and the windows onto shared and local memory the device set on its compute channel.

Submitting

"COMPUTE:0" runs Words and Launch parts, and "COPY:0" runs Words and Copy parts. A part of Words is ring entries that compiled code wrote (Rig_nv_abi.Gpfifo.entry).

The device keeps device order and prefix order (The C functions), and the release of v waits for the work of v to complete, so two submissions never overlap on the GPU. An untimed submission whose parts all run on the compute channel and whose last part is a launch, of a value whose low 32 bits are not 0, has that launch release v as it completes, which follows every launch before it. Such a v has the high 32 bits of v - 1, so the word goes from v - 1 to v even if the launch writes the 64 bits in two stores, which NVIDIA does not rule out. A part starts once the parts before it on its queue have completed, and reads what they wrote. Words run on their queue's channel. A copy runs on either channel and keeps its place in its queue's order, even a copy of zero bytes. On "COPY:0" a copy part of rows runs as one multi-line launch of the copy engine for each 2{^32} - 1 rows, and a copy of one row, or of rows that hold more than 2 GiB or lie 2{^32} bytes or more apart, as one launch of one line for each 2 GiB of each row. A fill writes its byte by the copy engine's remap of a constant, which reads no memory: in components of four bytes from the first multiple of 4 of its address, 2 GiB a launch, and of one byte in the at most three bytes at each end. On sm_80, sm_86, sm_87 and sm_89, a copy or a fill of one row of at least 256 KiB within the device's own Device memory runs instead as a kernel of the driver on the compute channel.

A wait names an aligned 64-bit word that the device's work addresses, and a value w. It holds the submission's work back until the word holds at least w in circular order (Rig_nv_abi.Method.acquire), whoever wrote it.

Each channel's ring holds 262,144 entries when make allocates the channel. A Words part of n words takes 1 + n/2 of them, and none if n is 0; launches and copies take none of their own. The driver's own words go into segments of the channel's segment ring, and each launch's descriptor and constant bank 0 into the compute channel's launch ring. rig_nv_room returns RIG_GROW for parts that exceed the segment ring, the launch ring or a channel's ring. The growth (grow) makes each such segment or launch ring anew, of the least power of two of bytes, at least a page, that holds twice what the parts take, and the next submission that needs it starts on it. A ring a growth made goes back, once a later growth replaced it, at the device's next growth or allocation after the device has reached the work of the submissions that used it. The 1 MiB rings the device opened with stay until it stops.

The RM fixes the entries of a channel's ring when it allocates the channel, so for parts that exceed a channel's ring the growth makes the device's two channels anew, in a new channel group, as make does: each ring holds the least power of two of entries that holds twice what the parts take, at most 2{^31}, and no fewer than the newest channels'. Once a submission needs the new channels, every submission on the device waits until the device has reached every value submitted before it; the next then starts on them, and places there the words a new channel owes before its parts: the engines' binding, the windows and the local memory. The channels they replace go back at the device's next growth or allocation. A growth that the path or the RM lacks memory or another resource for answers Short (Rig_edge.grown), and the device keeps its channels. Where the path's RM frees no object (rm's free is None), the channels a growth replaces stay until the device stops. Each growth at least doubles a channel's ring, from 262,144 entries to at most 2{^31}, so a device keeps at most 26 replaced sets: their rings, which take less than twice the largest of each channel, and the objects the RM made for each set's channels.

The driver holds at most 32,768 values in flight, a number it chooses for its stamp ring and its marks of each channel's work: rig_nv_room returns RIG_LATER while that many values submitted are not reached.

rig_nv_room returns RIG_NEVER for parts that take more than 2{^31} - 1 entries of a channel's ring, the most the ring holds: a channel holds the log2 of its ring's entries in 5 bits on each generation this driver runs. It returns it too for a part the device does not run:

• a part that is no words, copy or launch;
• a part on neither queue, a launch on "COPY:0", or a copy of one byte or more on "COMPUTE:0";
• ring entries of an odd number of words;
• an after index that is not below the part's own;
• a launch whose grid or group has an axis of size 0 or exceeds Rig_nv_abi.Qmd.max_size, whose group has more threads than Rig_nv_abi.Launch.max_threads, or whose dynamic shared memory, rounded up to 128 bytes, exceeds Rig_nv_abi.Launch.dynamic_shared.

Launches, whose constant bank 0 entry bounds, copies whose rows hold at most 2 GiB, with pitches and rows below 2{^32}, and fills of at most 2 GiB take no ring entry of their own, and the segment and launch rings grow to hold them. The driver's words for them close into an entry for each 4 MiB of segment, and a segment ring lies below 2{^40}, so their entries stay below the 262,144 a channel's ring holds from its open, and only memory bounds them. Words, and longer copies, take the entries their sizes take.

A launch's constant bank 0 is its kernel's, of at most Rig_nv_abi.Launch.max_bank_bytes: its first KiB holds the driver's parameters, and the kernel's own start where its compiler places them. A launch's bytes past those the kernel reads (Rig.Image.params) are left out.

rig_nv_room reads the word first, so RIG_LATER means that a value the device was given is not yet reached. rig_nv_submit never fails, since it only stores to memory, and returns RIG_COMMITTED, since the channels run work as soon as they fetch it, and rig never asks the device to commit. A submission of no parts writes its value after its waits and every earlier value.

While a profile is taken, each channel a value's work enters writes a stamp of the GPU's timer after the value's waits, and the channel that releases the value writes one before the release (Rig_nv_abi.Method.release_stamp). On COMPUTE a stamp waits for the channel's earlier work to complete. rig_nv_settle, the edge's settle, or a later rig_nv_submit converts the earliest start and the end to the host's clock, as The C functions states for settle. A stamp of a completed value that the work never wrote makes the next rig_nv_settle fail, and so does a failed read of the GPU's timer in it. Either loses the device.

A device measures the timer against the host's clock through the path's clock, and converts a stamp along the line through its first measure, at open, and its latest, once they lie a second apart, and at the timer's nominal rate of a nanosecond a tick before. A measure takes the time of a host clock between two readings of it. A re-measure whose readings lie more than 10 µs apart is dropped, and the measure at open keeps the first of sixteen reads within 10 µs, or the narrowest, which the first re-measure within 10 µs replaces. A time is thus within half its measure's interval, 5 µs once a measure within the bound is taken, plus the line's rate error over the time since the latest measure, at most 62.5 ms. That rate error is at most 10 µs over the measures' distance, 10 ppm, and the timer's own until they are a second apart. Through NVIDIA's kernel driver (Source) the driver takes the readings around its read of the timer's registers, and the device takes one more pair, to map the driver's clock to the host's. With Registers the device reads the registers itself, at open only, and its times drift from the host's clock at the timer's rate error since open.

A launch with dynamic shared memory runs in the multiprocessors' configuration of the most shared memory (Rig_nv_abi.Qmd.t.shared).

Memory

• Device memory is the GPU's, which the host does not address.
• Pinned memory is host memory that the GPU reads and writes coherently with the host.
• Mapped memory is the GPU's, which the host also addresses through the GPU's memory BAR. The host's stores to it reach the GPU before the work of any later submission of the device reads it.

alloc returns None if the GPU or the host lacks the memory, and for Mapped once the BAR has no room, or if it had none for the path's flush word (flush). alloc and map_host raise Invalid_argument for fewer than one byte. The device itself holds Pinned memory for its rings and its word, and 1 MiB for the descriptors and constant banks of its launches, in the BAR where the BAR had room at make and in Pinned memory otherwise.

locate returns the address of a region's first byte in the GPU's virtual address space (Rig_nv_abi.Gpu). Its handle is the same address, and its host address is None for Device memory. Freeing a view (map_peer) leaves the memory it maps and the other views in place. A view must be freed before the region it maps, as rig does (Rig_edge.Driver): freeing that region first may give its memory or its addresses out again while the view still reaches them.

peer is true iff paths of one key opened both devices and this device's path reaches the other GPU's memory (reaches of path), and map_peer then maps it. A view of Mapped memory orders the host's writes through the other GPU's BAR before this device's later work until the other device stops. map_peer of a stopped device's Mapped memory is Never, and is No_room where this device has no entry left for that order. Peer access between GPUs is untested. map_host answers Never if the path maps no host memory, and the path's refusal of the pages otherwise (map_host).

dma is the path's answer (dma). Another function reads what a value's work wrote once the value is reached: a copy's release writes the word once the copy engine's earlier copies are complete (Rig_nv_abi.Method.copy_release), and a compute release once its launches' writes are visible to the system. The device's later work reads what the function wrote once its writes are complete: a copy engine reads the memory itself, and each launch's descriptor invalidates the compute engine's data caches as the launch starts.

Images

An image is a cubin (Rig_nv_abi.Cubin). image returns Place (n, lay) for a cubin that Rig_nv_abi.Cubin.of_string accepts, where n is its Rig_nv_abi.Cubin.size, and Error msg otherwise. lay r relocates the cubin for the address of r. Every relocation must name a symbol the image defines, so CUDA's driver-supplied device functions, such as __assertfail and vprintf, are unavailable: an image that refers to one is refused even if the kernel to launch never calls it. A device opened with Rig_cuda.gpu runs such images.

The first submission on the compute channel after a lay invalidates the compute engine's instruction cache before its parts, since a cubin may load where another one's code was.

entry returns the address of the kernel's first instruction; the image starts there less the kernel's code offset (Rig_nv_abi.Cubin.kernel). Its params are the kernel's params_bytes, and its threads the launch's Rig_nv_abi.Launch.max_threads. It sets the function up for launch (Rig_nv_abi.Launch.make) and grows the compute channel's local memory to serve it (Rig_nv_abi.Gpu.t.local). It answers No_memory n, having made nothing, if the GPU has no memory for the n bytes of that local memory, and raises Invalid_argument with the reason if the function takes more than a launch can, such as a constant bank 0 of more than 64 KiB. unload frees the launch descriptors of the image's functions, and leaves its code region to rig.

Faults and stopping

The RM stops a channel that faults, such as on a read its page tables refuse, and writes the error into the channel's notifier, a page of Pinned memory. The multiprocessors and the MMU report their own faults to the RM, and a path may learn of others (check).

sleep reads the word and the notifiers every millisecond, and lets other domains run while it waits. As it starts to wait it asks the path for its report, and raises Fault if the path has one. Once a notifier holds an error it asks the path and the RM for their reports, and raises Fault. If the path has a report, Fault carries it. Otherwise Fault has a line for each distinct notifier error, by number and name, then one for each fault the multiprocessors or the MMU reported, or the RM's refusal to read them; an MMU line names the faulting address and access.

While no notifier holds an error, a sleep asks only the path. A fault the RM records without writing a notifier, or a refusal of the RM to read the faults, is then not raised by a sleep. A channel the RM stopped so keeps the word still: where the path bounds progress (hang_ms), rig loses the device with "no progress for n ms", without the RM's lines, and where the RM's watchdog runs (hang_ms is None), the wait does not end. Work that completes after such a fault raises nothing.

A wait for work that runs long lasts until the work ends, unless the path bounds progress: rig then loses the device past that bound.

stop ignores its fault. It ends the path's registration of the channels (unregister) and frees them, which makes the RM preempt their work, then asks the path to stop (stop). If the path unregistered and the RM freed the channels, or the path returns `Stopped, the word holds the last submitted value before stop returns, and the memory the device holds itself goes back to the path. Otherwise the word reaches that value when the channels stop or finish, and that memory stays allocated for the life of the process. Regions rig allocated return as Rig_edge.Driver.stop states, in both cases.

Either way the stop frees the C state the device ran work with, its templates and the channels' records, which the channels never read. The edge that reads the word, 80 bytes, stays for the process.

Once the word shows the last value and stop has returned, no work of the device writes it. A channel's release of a value follows its work, and the channels end their work with the last value's release. The stop raises the word itself only once the channels are freed or the path stopped them, or once both channels' notifiers hold an error: the RM stops a channel it reports an error for. A raised word would release the waits of a channel still running, whose own releases would then lower it, so with one channel stopped the word reaches the last value only as the other's work ends.

Paths

A path reaches a GPU's RM, allocates the RM's device, subdevice and virtual address space for the GPU, reads what the device needs, and gives it all to make as a path. make allocates everything else the device holds, its channels and their memory, through the path. What the driver does when a path or the RM refuses a call, such as a stop that answers `Unknown, is untested.

A path's locks

For a path's implementation. A path must take each lock it holds across OCaml code with Lock.protect, which counts it among the locks its thread holds, so that a call from a callback inside it is refused as it starts (Drivers). A path's lock comes last in the order of the module's preamble, and a path holds one of its locks at a time (Lock.protect).

Warning. A path's opener's make and its public functions that may take a lock must run their whole body inside Lock.guarded; one that does not may raise Invalid_argument from a lock it takes later, with what it changed before left changed, and a signal's handler may run anywhere in it.