Module Rig.Buffer
Buffers of device memory.
A buffer's memory is the whole allocation it lies in. A view is a buffer over part of another buffer's bytes, in the same allocation (Buffer.view). Views and borrows share their memory's stamps and claims, so work through any of them is work on that allocation. rig does not interpret the bytes.
A device's work addresses only buffers on that device. To use another device's memory, a submission names a borrow of it, which exists where the device can map that memory (Rig.reaches). A mapping is the access to an allocation that a device's driver makes for the device. A borrow that needs one holds the borrowing device's mapping of the allocation, which every live borrow of it on that device shares. The first borrow that needs a mapping pays for the mapping. The device's driver maps the pages and locks them where the device needs that. The mapping lasts while a borrow holding it lives. Once the last has ended (Buffer.unborrow) or been collected, rig releases the mapping after the borrowing device has run the work submitted until then. When the allocation returns, rig releases its mappings the same way and gives it back after them. A lost device keeps its mappings until it has stopped (Stops and forks). Empty memory borrows on every device of its machine but an io device, whatever Rig.reaches says, and an io device's on every other device.
Buffer.create, Buffer.of_string, Buffer.of_bigarray and Buffer.of_io make an allocation, with empty stamps, and Buffer.view and Buffer.borrow make further buffers over it. Its stamps order its uses (Order), and it returns as Reclamation states.
type tThe type for buffers. Compare buffers with ==. Polymorphic comparison does not look at their bytes: compare terminates, with an order that says nothing about contents, and (=) raises Invalid_argument when it reaches two buffers of one device, a buffer and itself included, and finds buffers of different devices unequal. Hashtbl.hash depends on a buffer's range and on its memory's addresses and state, which can change while the buffer lives, as when a device first borrows host memory.
Making buffers
type kind = Rig_edge.memory = - #
| Device - #
| Pinned(*Page-locked host memory that the device's driver allocates for the device, which both the device's work and the host address, with no flush needed between them. Which
*)budgetit counts in is stated in Reclamation. A device whose memory the host addresses, such asRig.host, aRig.memory_deviceor a Metal device, has none (create). Host memory reaches a device throughborrow. - #
| Mapped(*The device's own memory, which the host also addresses through a write-combined window, and which counts in the device's
*)budget. Work submitted after a host write sees it, and host reads are slow. Where the device has no such window, or the window or the device's budget has no room for the buffer, it isPinnedmemory, or, on a device whose memory the host addresses, the device's own memory. rig falls back without first freeing the device's cache.
The type for the kinds of a device's memory that create allocates.
create d n is a buffer of n new bytes of d's memory of kind memory (defaults to Device), with unspecified contents. An empty buffer allocates nothing. A non-empty one starts on d's alignment, and its memory runs on to whole words (Words). A host buffer of at least 64 KiB and at least four pages starts on a page, so devices can borrow it; a smaller one of at least 128 bytes starts on 128 bytes, so threads that write it in pieces of whole 128-byte blocks share no cache line, on Apple silicon's 128-byte lines or x86's 64-byte ones. On an io device the memory is what Rig_edge.Io.alloc makes, whatever memory.
Raises Invalid_argument if n < 0, if memory is Pinned and the host addresses d's memory, or if n > 0, d is an io device and its Rig_edge.Io.alloc raises it, and Lost if d is lost or its driver faults during the allocation.
of_bigarray ba is a buffer on host over ba's bytes, without a copy. It keeps ba reachable, and its memory is outside the claims (Claim). Each call makes a new allocation with stamps of its own, even over bytes another call wraps, so rig does not order work through one such buffer with work through the other. Wrap an array once and share the buffer through view and borrow.
of_io d k r ~access n is a buffer over the n bytes of r, a region of the io device d. A region is memory as d's library represents it, and the library declares its type under k (Rig_edge.Io.region_key). Its memory admits access, which is Read for a region d holds for reading, such as a file opened read-only. Its memory is outside the claims (Claim), and r returns to Rig_edge.Io.free once unreachable.
Raises Invalid_argument if d is not an io device, its library's key is not k, or n < 0, and Lost if d is lost or closed.
view b ~first ~length is the length bytes of b from its byte first on, in b's memory.
Raises Invalid_argument if b is dead, first or length is negative, or the bytes do not lie inside b's.
borrow d b is a buffer on d over b's bytes, without a copy, or None if d cannot map b's memory (Rig.reaches) or its driver never maps it, such as pages mapped read-only. A host buffer under 64 KiB may not start on a page (create), and host memory off a page does not borrow on a driver's device. It is Some b for b on d. The borrow keeps b reachable and shares its memory, so work through it is ordered as work on b, and a borrow of a borrow maps the memory the first one maps.
Raises Invalid_argument if b is dead or, on a device other than b's, a Claim.with_ holds its memory exclusive; Lost if d is lost or its driver faults during the borrow; Rig.Out_of_memory if d's driver has no room for the mapping (Reclamation); and Sys_error if asking an io device for b's pages fails transiently (Rig_edge.Io.pages).
free ~why b tries to return b's memory early, and is false, changing nothing, if b does not span its memory (spans), another claim holds it, or it is outside the claims (Claims). Otherwise it takes the memory exclusive and consumes it with the reason why ("freed" by default), which dead then answers, so every buffer over it is dead, the memory returns once the work using it is done, as collected memory does (Reclamation), and it is true. A library that consumed b for an operation passes the reason it consumed with, so that a later refusal of the memory names that operation.
Raises Invalid_argument if b is dead.
unborrow ~why b tries to end the borrow b early, and is false, changing nothing, if a claim or a hold through the borrow or one of its views keeps it, such as a host access or a submit through it on another domain, or a prepared submission that names it (Claim.hold). Otherwise every buffer of the borrow, its views included, is dead with the reason why ("unborrowed" by default), which dead then answers, the memory and its other buffers live on, the borrow's hold of its mapping ends as at its collection, and it is true. Claims of the memory through its other buffers neither refuse it nor see it. A program ends a borrow to unlock host memory it keeps.
Raises Invalid_argument if b is dead, is not a borrow on a device other than its memory's owner, or is a view of part of one.
Properties
These return a value for every buffer, a dead one included, and raise nothing.
device b is the device b is on: d for a buffer that create, of_io or borrow made on d, host for one that of_string or of_bigarray made, and its buffer's device for a view and for Claim.consume's result.
is_borrowed b is true iff b lies over memory that something other than b's device holds, either a bigarray (of_bigarray) or another device's memory (borrow).
spans b is true iff b's bytes are all of its memory, as for a buffer create, of_string, of_bigarray or of_io made, a borrow of one, or a view of all of one.
access b is the access b admits, which its views and borrows admit too. It is what b's memory admits: Read_write for memory create, of_string and of_bigarray make, and what of_io was given. A buffer Claim.consume returned from memory its claims did not hold exclusive admits only Read (Claims).
rig never writes through a Read buffer, and refuses a copy or work that would (copy, Submission.make, submit), and write of it raises. A write through an array export made breaks this, and reads of the memory then see unspecified bytes. A library that writes its caller's buffer checks this first, so that it reports the error itself.
dead b is Some why if b is dead, where why is the reason its memory was consumed with (Claim.consume), or its borrow ended with (unborrow), and None otherwise. A library that receives a buffer checks it first, so that it reports the error itself.
Host access
The host reads and writes a buffer's bytes in place where it addresses the buffer's memory: memory on host, a borrow on host included, and memory of a device of this machine that has a host address, such as a memory_device's. read and write take a read claim on the memory, so that no donation writes it meanwhile (Claims), wait for the work the access follows, call their function on the bytes, then release the claim when it returns or raises. The bytes are an array of length b / Bigarray.kind_size_in_bytes k elements of kind k, in the host's byte order, without a copy, and are ordered only during the call. An empty buffer is an empty array, whatever its memory. export waits itself, and wait orders other host code. Host access takes no value and adds no point to the stamps. The blits and fill (Copying) copy bytes to and from any memory, the host's or not.
A wait covers only the work submitted before it returns (Order), and work submitted while the function of read or write runs is not ordered after its accesses. Host access from several domains needs its own synchronization.
read, write and export raise Invalid_argument if b is dead, its bytes are not a whole number of elements of k, b has bytes that the host does not address or that do not start at a multiple of the elements' size (of one component's size for complex kinds), or a Claim.with_ holds the memory exclusive.
wait b access returns once the work that a host access access of b must follow is done. For Read it waits for every device's last write of b's memory, and for Read_write for every use of it by any device. It allocates nothing when that work is already done.
- Host memory and an io device's memory have work to wait for only where a device borrowed them.
- It does not wait for work through
Submission.unorderedrefs, which the caller orders. - It takes no claim, so a donation on another domain may write the memory once it returns, which
read,write, the copies, the blits andfillprevent.
Raises Invalid_argument if b is dead, and Lost where Memory after a loss says, before it waits on any device if the loss came first.
reached b access is true iff wait b access would return without blocking, as the work it waits for is done. Host access then follows that work as after wait. Otherwise it commits that work on each of its devices as Point.reached does, blocking as that commit blocks, so a loop on it ends once the work runs. It allocates nothing.
Raises Invalid_argument if b is dead, and Lost where wait does.
val read :
t ->
('a, 'b) Stdlib.Bigarray.kind ->
(('a, 'b, Stdlib.Bigarray.c_layout) Stdlib.Bigarray.Array1.t -> 'c) ->
'cread b k f is f a, where a is b's bytes as elements of k, once every device's last write of b's memory is done. a is the memory itself, so the host must address it where b has bytes; blit_to_bytes copies out of any memory. f must not write through a, since work that reads the memory may still run, and must not keep a after it returns, since nothing orders a later access through it.
val write :
t ->
('a, 'b) Stdlib.Bigarray.kind ->
(('a, 'b, Stdlib.Bigarray.c_layout) Stdlib.Bigarray.Array1.t -> 'c) ->
'cwrite b k f is f a, where a is b's bytes as elements of k, once every use of b's memory is done. Work submitted after write returns sees f's writes. f must not keep a after it returns, since nothing orders a later access through it.
Raises Invalid_argument also if b admits only Read (access).
val export :
t ->
('a, 'b) Stdlib.Bigarray.kind ->
('a, 'b, Stdlib.Bigarray.c_layout) Stdlib.Bigarray.Array1.texport b k is b's bytes as elements of k for good, once every use of b's memory is done, for code that keeps them, such as a library's array. The array keeps the memory alive while it is reachable. From then on the memory is outside the claims (Claim), and the caller orders access through the array with later work, as for of_bigarray. The array reads without any check of loss (Memory after a loss).
Raises Invalid_argument also if b's memory is not host's, which create made on host, or of_string or of_bigarray made. Another device's memory is refused even where the host addresses it, since every use of a lost device's memory raises Lost (Memory after a loss) and reads through an array cannot.
Copying
A copy is ordered as work that reads its source and writes its destination. Where the host addresses both sides, or one side is an io device's memory and the host addresses the other, the host or the io device copies. Otherwise a device copies on its copy engine, the queue that its driver names in copy_engine (Rig_edge.facts), and the copy takes the next value of that device's timeline (Rig.submitted). On another machine, a device of that machine copies.
A borrow of an io device's memory is its pages, whose reads by the host or by work are as the io device states once the object behind them changes, such as a file that a truncation cut short (Rig_disk). Memory an io device owns, its own or a borrow of its pages (borrow), moves only through that io device's read or write. A copy from or into it never reads or writes the pages, and no device maps them for it, so a copy from a file cut short raises what the io device's read raises. Two allocations of one object, such as two opens of one file, are two memories that share no byte (overlaps). A program must not copy between bytes of the object that both name, whose result is unspecified.
A device copies directly between memory on it, its own or a borrow on it, and memory the host does not page that the device maps, which is another device's memory of its driver and another driver's memory that the host addresses, such as its Pinned memory. Such a mapping lasts until the memory returns, also where a borrow made it.
Memory the host pages (the host's own, a bigarray's, an exported array's and a memory device's) reaches the copy engine of a device of this machine through a borrow on that device that the copy names (borrow). A copy that names the memory itself passes through staging memory, also while a borrow of it lives.
Any other copy passes through staging memory, host memory that rig makes in allocations of 32 MiB, each at the first copy whose bytes need it, and maps on each device that copies through it for the allocation's life. Between a device and memory the host pages, the host moves the bytes into or out of staging memory before the call returns. Such a copy is synchronous for the host and slower than a copy through a borrow of the same memory, which takes the device's direct route once the borrow has paid for its mapping. A program that copies one allocation many times borrows it once and copies through the borrow.
A copy holds one allocation, or two for more than 32 MiB, until the transfers the host makes for it are done. A copy of at most 32 MiB whose bytes the host writes into staging memory puts them after the bytes earlier such copies put in an allocation that no copy holds, where it has room, and waits for no work there; once rig has seen that allocation's work done, it puts them at its start. Where none has room, it takes an allocation whose work is done, else the one whose bytes were placed from its start longest ago, and waits for its work. A copy whose bytes a device writes into staging memory takes an allocation whose work is done, or a new one, and no copy puts bytes after them. Staging memory so grows by 32 or 64 MiB for each copy holding it at once, and by as much for each copy whose bytes a device wrote into it and whose work is still queued, or, on a device whose word rig reads through its driver, until a wait reads it. A copy of more than 32 MiB whose bytes the host writes waits before it reuses one of its two allocations until the work that last read it is done; where a device writes them, its queue waits.
A copy's allocations serve the copies after it. Let n be the most allocations a staging memory has had at once. When a copy gives its allocations back, every allocation that no copy holds and that no copy took within the last n takes returns as free returns memory. Copies that run one after another, or several at a time over and over, keep theirs.
Staging memory that no copy holds also returns with its device's cache: at free_cache of that device (host for host memory), at a set_budget that leaves the device over its budget, and from reclamation's second round (Reclamation). After copies to many devices at once, their allocations so stay until n later allocations have been taken or one of these returns them. An allocation returns once the work that used it is done.
A device that maps no host memory (maps_host in Rig_edge.facts) stages through memory of its own that both it and the host address, its Pinned memory, or its own memory where the host addresses it, kept and returned in the same way, and returned too once the device is lost and has stopped. Between two such devices, the host copies from one device's staging memory into the other's.
The blits copy between a buffer and an OCaml string or bytes, and submit_fill and fill set a buffer's bytes. They are ordered as copies. Where the host addresses the buffer (Host access) and no io device owns its memory, the host copies or sets the bytes once the work the copy follows is done. Otherwise the blits' bytes pass through staging memory, and a device copies them to or from the buffer as copy does. A fill of memory an io device owns is the io device's writes from host memory that holds the byte. Any other fill runs on the copy engine of the buffer's device, or, for a borrow on a device that runs no copy, of the device whose memory it borrows, and reads no memory.
submit_copy and submit_fill return once their work is queued, as Rig.submit returns, with a point that is reached once the work is done. copy, fill and the blits return once their bytes are where they go.
A copy, blit or fill holds a read claim on its buffers' memory until it returns, so that no donation writes the memory meanwhile (Claims). Work that a copy left queued on a device holds none, as no work on a device does: the stamps order a donation's write after it.
type box = {- #
src_offset : int;(*The byte of the source where the first row starts.
*) - #
dst_offset : int;(*The byte of the destination where it starts.
*) - #
width : int;(*The bytes of each row.
*) - #
rows : int;(*The number of rows.
*) - #
src_pitch : int;(*The bytes from the start of a row of the source to the start of the next.
*) - #
dst_pitch : int;(*The same in the destination.
*)
}The type for boxes: rows rows of width bytes. Row i starts at byte src_offset + i * src_pitch of the source and at byte dst_offset + i * dst_pitch of the destination. A box fits a source and a destination when its fields are non-negative, each pitch is at least width, and, if it has bytes, its last row ends within each side: src_offset + (rows - 1) * src_pitch + width is at most the source's length, and likewise for the destination, computed without overflow. A box of one row is a run of bytes (row).
row ~src_offset ~dst_offset n is the box of one row of n bytes, at src_offset and dst_offset (default to 0), with pitches of n.
submit_copy ~waits ~src ~dst boxes copies the bytes of each box of boxes from src into dst, one box after the other, and is a point that is reached once every byte is there. A byte of dst that two boxes write holds the later box's byte. The copy follows the points of waits (defaults to [||]) besides those its memory orders (Order), and submit_copy reads boxes once.
- Where a device copies directly, the copy is one submission on that device, or several in order where the driver takes none that large.
submit_copyreturns once they are queued, and the point is the last one's. - Where the copy passes through staging memory,
submit_copyreturns once the transfers the host makes are done and the others are queued. The point is that of its last transfer, which follows the others. - Where the host or an io device copies,
submit_copyreturns once the copy is done, and the point is a point ofRig.hostthat is reached. A copy of no bytes waits on the host for the points ofwaitsand returns such a point.
It blocks where Rig.submit blocks, where the host copies after the work the copy follows, where a transfer into staging memory waits for the work that last read that memory, and where the host moves bytes out of staging memory after a device's transfer into it, which it waits for. So a copy by a device of this machine into memory the host pages waits for its bytes, unless it names a borrow of that memory on the device (Copying). A device of another machine queues such a copy and does not wait. It allocates.
Memory outside the claims, such as a bigarray's (of_bigarray) or an exported array's (export), is read and written by the copy until its point is reached, so a program orders its own access through such an array after the point.
Raises Invalid_argument if src or dst is dead, they share a byte of memory, a box does not fit them, dst admits only Read, one side is an io device's memory and the other a borrow of it, a side's memory is held exclusive, the copy has bytes and no device can copy between the two sides, or one row of a box fits in no submission of the device's driver. Raises Lost if a device that runs a transfer is lost before its submission, or its submission loses it, or while the host waits for a transfer it moves bytes after, or if a point of waits is a lost device's and that device never reached it; Out_of_memory if staging memory cannot be allocated or mapped (Reclamation), or the device's queues must grow for a submission and it has no memory for the growth (Rig.submit); and what an io device's read or write raises. Where it raises after it queued some of the copy's submissions, those still run, so dst may hold some of the copy's bytes.
copy ~src ~dst copies src's bytes into dst and returns once they are there: submit_copy ~src ~dst [| row (length src) |], then a wait for its point, holding its read claims until it returns.
Raises Invalid_argument if they differ in length, and otherwise as submit_copy raises. Raises Lost also if the copy is lost during the wait.
blit_from_string s i b j n copies the n bytes of s from i into b from its byte j.
Raises Invalid_argument if the ranges are not valid, b is dead or it admits only Read, and otherwise as copy ~dst:b raises.
blit_to_bytes b i s j n copies the n bytes of b from its byte i into s from j.
Raises Invalid_argument if the ranges are not valid or b is dead, and otherwise as copy ~src:b raises.
submit_fill ~waits b c sets every byte of b to c, after the points of waits (defaults to [||]), and is a point that is reached once they are set, returned as submit_copy returns it. The host sets the bytes where it addresses b, and writes them through the io device for an io device's memory. Otherwise the copy engine of b's device sets them, reading no memory, or, where b is a borrow on a device that runs no copy, the copy engine of the device whose memory it borrows.
Raises Invalid_argument if b is dead or admits only Read, its memory is held exclusive, or b has bytes, the host does not address it and no device sets them. Raises Lost as submit_copy does, and what an io device's write raises.
fill b c sets every byte of b to c and returns once they are set: submit_fill b c, then a wait for its point, holding its read claim until it returns.
Raises as submit_fill raises, and Lost also if the fill is lost during the wait.
Low level
For the libraries that submit work. C code reads a buffer's host address, its length, where its bytes lie and the reason it is dead with rig_buffer_host, rig_buffer_bytes, rig_buffer_span and rig_buffer_why, claims its memory with rig_buffer_claim and rig_buffer_release, and waits under the claim with rig_buffer_wait, which runs wait. All are declared in rig.h.
Memory is handle-only when its driver names it by a handle and gives it no address (Rig_edge.location).
io b k is the region b's memory lies in, if it is an io device's memory of a library whose key is k, and None otherwise.
Raises Invalid_argument if b is dead.
aligned b ~at n is true iff the address at bytes past b's first byte, in the addresses of b's device's work, is a multiple of n, for n a power of two. It is arithmetic on the address, defined for every at, in b's bytes or outside them. Handle-only memory has no address here, and its allocation starts on the alignment of the device whose memory it is, so aligned answers for an n up to that alignment from b's offset in it. A borrow of it answers as its memory does: a device maps another's memory by pages, and no device's alignment exceeds a page, so the mapping keeps the allocation's offsets modulo that alignment.
Raises Invalid_argument if b is dead or an io device's memory, if n is not a positive power of two, or if b is handle-only and n exceeds that alignment.
type dma = Rig_edge.dma = - #
| Bus of (int * int) list(*The memory's bus addresses on its machine, as (address, bytes) pairs in order, each a multiple of 4096. They hold until the process exits.
*) - #
| Dmabuf of {}(*A new dma-buf file descriptor, close-on-exec, which the caller owns and closes, and the offset of the allocation's first byte in it. The dma-buf keeps the memory until its last reference goes, such as a NIC's registration of it.
*)
The type for how another PCI function of a buffer's machine, such as a network adapter, reaches its memory by DMA (dma).
dma b ~fn is how the PCI function at the bus address fn of b's machine, such as a network adapter, reaches the whole allocation b's memory lies in, which starts offset bytes before b's first byte, as the driver of the device whose memory it is describes it, or Error why where that driver cannot (Rig_edge.Driver.dma). A borrow's answer is its memory's owner's.
The memory is then outside the claims for good (Claim.share), also where the driver refused, and once described the allocation never returns to its device's cache.
Warning. No stamp records fn's reads and writes. The caller lets fn read the memory only once the work that last wrote it is reached, and lets work use bytes fn wrote only after fn's writes are done, submitting that work afterwards. A Dmabuf descriptor is the caller's to close.
Raises Invalid_argument if b is dead or empty, or is host's, a memory device's or an io device's memory, or if a Claim.with_ holds it exclusive, and Lost if the device whose memory it is is lost.
handle b is the driver's object for b's memory (Rig_edge.location), such as an MTLBuffer, which b starts offset bytes into. It is 0n for an empty buffer that create made on a driver's device.
Warning. The object lives only while b is reachable. Work that uses it must keep b reachable until the work is done, for example with the keep of its submission (Submission.make), and must not free it.
Raises Invalid_argument if b is dead, or is host's or an io device's memory, which no driver object names.