Module Rig_amd

A device of this driver is one AMD GPU. Paths find and take GPUs, and the driver runs them. A path is a library that reaches a GPU in one way and gives the driver what it found there: the GPU's description, and functions that allocate the GPU's memory, make its hardware queues, wait for its interrupt and stop it (path). Rig_amd_amdgpu reaches a GPU through Linux's amdgpu kernel driver, and Rig_amd_pci drives the GPU's PCI function itself, with no kernel driver. Each offers an opener whose make fills a path and calls make.

From any path, make builds the same device. Two queues take rig's work. "COMPUTE:0" is a ring of PM4 or AQL packets that the GPU's compute engine reads, and "COPY:0" a ring of SDMA packets that a copy engine reads. Both rings lie in Pinned memory, so the driver's submit places packets and rings the queues' doorbells with plain stores. The argument segment is Pinned memory into which submit copies each launch's arguments. The last packet of a value's work is its release, which writes the value into the timeline word, the eight bytes of host memory that the opener's make is given, once the work has completed, and raises the path's interrupt, on which sleep blocks. What differs between paths, such as a bound on how long work runs or whether the GPU maps host memory, is a field of the path, which the device's facts carry (Facts). The sections below hold on every path.

rig --room, submit--> COMPUTE:0 ring, PM4 or AQL --+
                      COPY:0 ring, SDMA -----------+--> GPU
                                                         |
                          release of value v, interrupt  |
rig <--signaled, sleep-- timeline word, host memory <----+

path: the GPU's memory, its hardware queues, its interrupt, its stop

A device's life goes like this:

1. A path's opener reaches the GPU and fills a path.
2. make makes the device's rings and argument segment, and maps its timeline word, through the path. Rig.open_ runs these two steps through the opener's make.
3. rig gives each submission to the driver's room and submit (C functions). The GPU runs its parts and releases its value.
4. Once rig loses or closes the device, stop ends it (Stop).

Unless a value says otherwise, a function that calls the path returns the path's refusal of its arguments as None or Error, and raises Fault for any other failure of the GPU or the path. signaled, free and stop never raise it. Any domain may call any value concurrently with others, except the C functions and stop, which rig calls as Rig_edge.Driver states. sleep may run beside the C functions. The scratch and trace of the device's capability, and an opener's make, called from a callback inside a lock (Rig's domains section), raise Invalid_argument, having changed nothing (Drivers).

A program names a GPU through a path's opener:

match Rig.open_ (Rig_amd_amdgpu.gpu 0) with
| Error why -> prerr_endline why
| Ok d -> print_endline (Rig.arch d) (* such as "gfx1201" *)

The C functions states how rig hands a driver work for its queues, and Rig_edge what its timeline word shows. Rig_pci's section on what an open needs states what an open of a GPU with no kernel driver needs.

GPUs

An AMD GPU has two version numbers. Its processor (Rig_amd_abi.Gpu.processor), such as gfx942, names the code objects it runs. Its GC version, such as 9.4.3, names the block that runs its compute queues, whose registers and packets follow it. The Instinct MI300 series is processor gfx942 with GC 9.4.3. This driver names GPUs by processor, and Rig_amd_pci, which programs the GC, by GC version.

The driver drives gfx942, gfx950 (the MI350 series), and the processors of GFX11 and GFX12. On a GPU of one die its compute queue reads PM4 packets. On a GPU of several, such as the MI300X, it reads AQL packets, which every die runs (Rig_amd_abi.Capability.compute). The copy queue reads SDMA packets.

On a GPU of several dies, the AQL queue hands every kernel the device's scratch memory.

The driver has run on gfx1201 alone. Its AQL queue, with the dispatch, scratch and room of GPUs of several dies, has never run on hardware.

Order

"COMPUTE:0" runs words (Rig.Submission.words) and launches, and "COPY:0", the copy engine, runs words and copies. The parts of one queue run in array order. A part's after indices, which rig derives (Rig_edge.queue), order it after parts of the other queue, and two parts of different queues that after does not order may run in parallel.

A copy runs on the copy engine or on the compute engine, and keeps its place in the order of "COPY:0" whichever engine runs it, as does a copy of no bytes.

Value v's work starts once the work of v - 1 has completed, on both queues. A submission of no parts releases its value once v - 1 is reached.

A wait for another device's point holds the submission's work back until the aligned 64-bit word at the wait's address holds at least the point's value, compared as unsigned integers. The compute queue makes such waits only on a GPU whose GC and compute firmware (mec of path) this library knows to compare 64-bit words. On other GPUs the driver's waits lists neither stores nor hosts (Rig_edge.waits), and rig waits on the host.

Caches. The first part of a submission on the compute engine, and any part after a wait for the other queue, follows an invalidation of every cache of the GPU, its L2 included, after the L2 is written back. Any other launch follows at least an invalidation of the caches above the L2. A launch therefore reads what the host, the copy queue, other devices and the launches before it wrote. A value's release writes the L2 back, so the host and other devices read what its work wrote once the value is reached.

Memory

• Device memory is GPU memory, which the host does not address.
• Pinned memory is host memory that the GPU addresses, which the host reads as fast as other memory.
• Mapped memory is GPU memory that the host addresses through the GPU's memory BAR, write-combined, so the host writes it fast and reads it slowly. The host's writes reach the work submitted after them. alloc returns None for it on a path without such memory or whose hdp is None (path).

alloc returns None if the GPU or the host lacks the memory. alloc and map_host raise Invalid_argument for fewer than one byte. Freeing a mapping ends the mapping alone, and the memory it maps stays.

locate returns the GPU address of a region's first byte, and that address again as the handle, since the GPU names memory by address. The host address is None iff the region is Device memory or a map_peer view of Device memory.

peer and map_peer return true and a view at the same GPU address iff paths of one key opened both devices and this device's path reaches the other GPU's memory (reaches of path). A device flushes the host writes of at most 7 other GPUs, so map_peer returns None for Mapped memory of an eighth. A view of Mapped memory flushes the other GPU's host writes before this device's later work until the other device stops, and map_peer of a stopped device's Mapped memory is None. Peer access between two GPUs, with the flushes of each other's host writes, is untested, and so is a path whose hdp is None.

dma is the path's answer (dma of path), or Error where the path gives no register that flushes the GPU's host data path (hdp of path), since another function's writes through the memory BAR pass it as the host's do. Once a region is given, each submission flushes that path before the GPU reads, for the rest of the process, so the device's later work reads what the function wrote. The function reads what a value's work wrote once the value is reached, since its release writes the L2 back, and the host data path of the GPUs this driver drives keeps no read cache to invalidate.

map_host returns a region the GPU addresses, or Never if the path maps no host memory, and otherwise the path's refusal of the pages (map_host). Both paths map host pages for writing, so it answers Never for pages mapped read-only, such as an io device's pages over a read-only file. Rig_amd_amdgpu's kernel driver faults the pages in for writing, and Rig_amd_pci pins them for writing through VFIO.

Images

image returns Place (n, lay) for a code object, whose image takes n bytes (Rig_amd_abi.Code_object.size) of the GPU's own memory, from which its code runs. It returns Error msg:

• if the binary is not a code object, with Rig_amd_abi.Code_object.of_string's message;
• if it is for another processor, as "a code object for gfx90a; the GPU is gfx1201";
• if a kernel takes more local data share than the GPU has, naming the kernel and both sizes.

entry returns the address of the kernel's descriptor, the symbol f ^ ".kd", as code, which a dispatch names, the kernel's explicit_size as params and its max_threads as threads (Rig_amd_abi.Code_object.kernel). It makes the kernel's launch and grows the device's scratch memory to the kernel's private segment. It answers No_memory n, having made nothing, if the GPU has no memory for the n bytes of that scratch. It raises Invalid_argument for a kernel:

• that reads an implicit argument that is a runtime's service (Other of Rig_amd_abi.Code_object.hidden);
• that reads its dispatch packet, on a GPU of one die.

unload frees the launches.

Driver

The sections above state what this driver adds to each value's contract.

This driver

Facts

The device's facts (Rig_edge.facts) are:

• arch, the GPU's processor (Rig_amd_abi.Gpu.processor), such as "gfx1201";
• budget, the bytes of the GPU's own memory;
• queues, "COMPUTE:0", which runs Words and Launch, and "COPY:0", which runs Words and Copy, index 0 and 1 in C;
• copy_engine, Some 1, "COPY:0";
• completion, Store, since the queue that releases a value writes the word;
• waits, stores and hosts iff the compute queue compares 64-bit words (Order);
• may_block, false, since the C functions write memory and call no system function;
• hang_ms, the path's hang_ms, None on a path whose kernel driver bounds how long work runs;
• maps_host, true iff the path maps host memory (map_host of path);
• host_addresses, false, since the host does not address Device memory;
• alignment, 4096, since both paths allocate whole 4 KiB pages of the GPU, at addresses that start on one;
• cache_bytes, the path's cache_bytes: the L2's and the MALL's bytes as the kernel driver states them on the amdgpu path, None where it states no L2, and None on the PCI path, which reads neither;
• capability, capability, declared with Rig_amd_abi.declare;
• word, the device's timeline word, as Word. below states;
• bus, the path's bus, the GPU's PCI function.

Capability. The record holds the GPU, its clock, its compute queue's packets, its work-group processors that run work, and its trace buffers, made at the first trace. On an AQL queue its scratch grows the device's scratch memory, which the queue hands every kernel, and returns Error once stop has begun. On a PM4 queue the dispatch of each launch names the scratch.

Word. The device maps rig's whole array of words through the path (map_host of path). The path's first map of the array on a GPU makes the mapping, and every later one counts against it: a GPU maps the array once, at the host's addresses, while a device of it lives. On a path that maps no host memory, or when the word make was given lies outside the array, the word is eight bytes of the device's own Pinned memory, started at the given word's value, which rig reads through signaled. The device's values start after that value. The word holds an unsigned 64-bit integer in the host's byte order. A release writes it after its work's writes reached memory. A release on "COMPUTE:0" writes all 64 bits, and one on "COPY:0" the low 32 bits, so the driver releases on "COMPUTE:0" every value whose low 32 bits are 0, the only values whose high 32 bits change.

Sleep. sleep blocks on the path's interrupt, which every release raises, and lets other domains run while it waits. It may return on the interrupt of an earlier release or of other work of the GPU. The path reports a fault of the device's work, such as a page fault or a reset of the GPU, and sleep raises it as Fault with the path's report.

C functions

edge's room, submit and settle are rig_amd_room, rig_amd_submit and rig_amd_settle, declared in rig_amd.h. Its commit does nothing, since submit places each value's release and returns RIG_COMMITTED. A part is:

• words, whole packets of the queue's kind: PM4, or AQL in multiples of 16 words, on "COMPUTE:0", and SDMA on "COPY:0";
• on "COMPUTE:0", a launch. submit writes the kernel's arguments into the argument segment: the launch's parameters, each ref's address added, then the implicit arguments the kernel reads (Rig_amd_abi.Code_object.hidden) for a grid of whole groups that starts at work-item 0. A group takes the kernel's group segment plus the launch's shared memory of local data share;
• on "COPY:0", a copy between two GPU addresses, its handles, whose ranges do not overlap. A box of two rows or more whose pitches are multiples of 4 bytes is sub-window copies (Rig_amd_abi.Sdma.window) of up to most_rows rows each, in the largest element of up to 16 bytes that divides its width, its pitches and each side's address modulo 4, if its width and pitches fit the packet in those elements and each side's address modulo 4 plus its width is at most its pitch; any other box is linear copies of each row. A fill is a CONST_FILL of its whole dwords, as many bytes a packet as a linear copy moves, and a linear copy of each unaligned end, at most three bytes, from four bytes of the argument segment that hold its byte. It reads no other memory. The compute engine runs only copies of one row.

A submission takes fewer words than the compute ring holds, at most half the copy ring and at most half the argument segment, since its SDMA packets and argument bytes never wrap. The segment also holds 8 bytes per part, the word through which a part of one queue orders a part of the other. room returns RIG_NEVER for parts that exceed the largest rings the path makes (most_ring of path: 2 GiB through Rig_amd_amdgpu; through Rig_amd_pci 16 GiB for the compute queue and 8 GiB for the copy queue, but 16 MiB on SDMA 4, whose one queue that path programs keeps the ring the device opens with), or that the device does not run, and RIG_GROW for parts that exceed its rings or its segment. On SDMA 4 over Rig_amd_pci, a submission of copies past half that ring, about 170,000 linear copies of one packet each, never fits, and Rig.Buffer.submit_copy then submits a box of that many rows that sub-windows do not copy in several submissions.

grow then makes, of Pinned memory of the first power of two from 4096 bytes that holds twice what the parts need and at most most_ring, a segment, or for each ring they exceed a new queue of the path's on a new ring, with read and write positions of its own at 0 (queue of path). The next room that needs them installs them once the device reached every value submitted, and every room answers RIG_LATER until then, so the device's work ends while other threads submit: later work goes to the new queue, whose doorbell is another, and the replaced queue, which read every packet, runs beside it until the next growth or stop destroys it (retire of path) and gives back its ring. A growth whose parts need more than a queue a growth made and no room installed replaces that queue, which ran no work. It makes the new queue first and then destroys the one it replaces. Where the path answers Full for the new one, as over Rig_amd_pci, it destroys the old one first and makes the new one in its slot, so over Rig_amd_pci a path holds at most two queues of a kind. Segments and rings never shrink. grow answers Short n if the path lacks n bytes of memory, or another resource for a queue, having made nothing that stays and kept what a growth handed, and raises Fault if the path refuses a queue for another reason. A copy in linear copies takes a packet for each 1 GiB of each row it moves (4 MiB on SDMA engines before 4.4.2 and on 5.0 and 5.1). The device does not run:

• a copy of one byte or more on "COMPUTE:0";
• words on an AQL queue that are not whole packets;
• a launch whose parameters end before its kernel's explicit arguments do (explicit_size of Rig_amd_abi.Code_object.kernel);
• a launch whose grid or group has an empty axis, whose groups have more work-items than its kernel's max_threads, or whose shared memory exceeds the GPU's local data share less the kernel's group segment;
• on a GPU of several dies, a launch of 2{^32} work-items or more along an axis;
• a part with an after index not below its own.

On PM4 and SDMA rings, the queues read none of a submission's packets before all of them are placed, since submit moves the write position and rings the doorbell last. An AQL queue may read a packet as soon as its header is placed, and submit places each header after the rest of its packet. rig gives submit no waits where the compute queue cannot wait, as the device's facts state (Rig_edge.facts).

Times. While a profile is taken, each queue a value's work enters writes the GPU's clock after the value's waits, and the queue that releases the value writes it before the release: on "COMPUTE:0" once the dispatches before it have completed (PM4's COPY_DATA of the clock), on "COPY:0" after a fence has waited for its copies (SDMA's TIMESTAMP). settle, or a later submit, converts the earliest start and the end to the host's clock, in value order, with each no earlier than the end of the value converted before it, since a value's work starts once every earlier value's work has completed. A stamp of a completed value that the work never wrote makes the next settle fail, which loses the device.

A device measures the clock against the host's clock as the path's clock says, and converts a stamp along the line through its first measure, at open, and its latest, once they lie a second apart, and at clock_hz from the latest before. A measure through a Source takes the time of a host clock between two readings of it. A source may count another counter of the clock's rate, a constant apart from the clock, as KFD's does on GFX 12: the device takes the constant once, at open, from stamps its compute queue writes between two readings of the host's clock, the closest pair of eight, against a read of the source, and adds it to every read. That read is the first of sixteen before the stamps within 10 µs, else the first of sixteen after them within 10 µs, else the narrowest; it is also the measure at open, which, if wider than 10 µs, the first re-measure within 10 µs replaces in the line. A re-measure whose readings lie more than 10 µs apart is dropped. A time is thus within its measure's deviation plus the line's rate error over the time since the latest measure, at most 62.5 ms. A measure's deviation is half its interval, 5 µs once a measure within the bound is taken. The rate error is at most the two measures' deviations over their distance, 10 µs over a second (10 ppm) for deviations of 5 µs, and the clock's own until they lie a second apart. Through KFD, KFD samples its host clock after it reads the GPU's counter, and the device maps that sample to the host's clock with a pair of readings of its own. A measure's deviation is then half its interval plus KFD's gap between the counter read and its host sample. That gap is one-sided, usually one register read, and unbounded under preemption. Through a Source a time also holds the constant's error: half the stamps' interval plus the deviation of the read it was taken against, for the device's life. The constant stays fixed for that life, and a counter that moved against the clock would shift every time by the move: a conversion that puts a value's start before its submit, or its end after the conversion, by more than 1 ms leaves both its times 0, as for work the device cannot time, so a profile records no Work for it (Rig.Profile.event). A late read of the source, as KFD's under preemption, does the same. With Round_trip the device measures at open only, and its times drift from the host's clock at the clock's rate error since open.

Stop

stop passes fault to the path's stop, which destroys the device's queues. Once none runs, it writes the last value given to submit into the word and gives back the device's own memory. If the path could not destroy every queue, the device keeps its memory, and the word reaches that value only if the queues still run and complete their work.

Either way the stop frees the C state the device ran work with, its rings' and timed values' records, which the queues never read. The edge that reads the word, 24 bytes, stays for the process. A word of the device's own goes back when rig frees it (Rig_edge.facts).

Once the word shows the last value and stop has returned, no work of the device writes it: the queues release values in order, and the last value's release is the last write; the stop writes the word itself only once the path destroyed every queue.

Measures

A device made with measure (make) counts measure.counters around each launch of a value a profile times. It takes the GPU's stable power state (stable_power of path) and holds it as its path states, so every process's work on the GPU runs at its stable clocks meanwhile. Of the 8 SQ counters a GFX12 SQ counts at once, it counts at most 4, "SQ_WAVES" one of them, which it counts whether listed or not.

A measured launch starts once the work before it has completed, and the work after it on the compute queue starts once its counters are stored. Its submission's Work covers both. A copy part of the same value runs on the copy queue and waits only for what its after names, so it may run during the launch, and a counter of cycles or waits may then include its contention for memory. Work submitted while no profile is taken is not measured.

Its Counters event has the launch's own start and stop and each counter's values as Rig_amd_abi.Counter.t lays them out. The values are the launch's alone and exact unless another process counts on the GPU, or waves started under another process's counting still run. A launch records no Counters, and counts in an Unmeasured event, if "SQ_WAVES" differs from its grid's waves, if a value reads 2{^32} - 1, if the host had no memory for its records, or once the device's VMID is seen to have moved, until a count with no launch reads nothing under the filter of the launch after it: the GPU may then have counted another process's waves as the device's.

The device's facts list the counters, "SQ_WAVES" first, each with one value per place Rig_amd_abi.Counter.t counts it in.

Paths

Only path libraries use these values. A path reaches a GPU in one way, reads what the device needs, and gives it all to make as a path. What the driver does when a path refuses or fails a call, such as a stop that answers `Unknown or a free that raises, is untested.

A path's locks

For a path's implementation. A path that holds a lock across OCaml code must take it with Lock.protect, which counts it among its thread's locks (rig_hold in rig_edge.h), so that a call of rig's or of the driver's from a callback inside it is refused as the call starts and does not wait for good.

Warning. The path must take its locks in the order Lock.create states. Nothing checks it, and two threads that take two locks in opposite orders wait for each other for good. Only a take by the thread that holds the lock raises. A path's entries must run their whole body inside Lock.guarded; one that does not may raise Invalid_argument from a lock it takes later, with what it changed before left changed, and a signal's handler may run anywhere in it.

References

• LLVM's User Guide for AMDGPU Backend: code objects, kernel descriptors, the memory model of each GFX version.