Module Rig

A device holds memory and, unless it is an io device, runs work on it. The host is this process's device. A driver's device, such as a GPU, runs the work its driver places on the device's queues, and a program opens it with the opener the driver supplies (open_). An io device, such as a file, holds memory that the host reads and writes through its library's calls (open_io). A program uses the devices of another machine through that machine's host (Machines).

Most programs need only open_, Buffer and Buffer.copy. Image, Submission, submit, Local and Claim are for libraries that run compiled code, keep values per device or write results over their inputs.

Work on a GPU runs asynchronously. The call that submits it returns before the work has run. So that the host can tell what has run, rig numbers the work submitted to a device with consecutive integers, in the order the device accepts it. These numbers are the values of the device's timeline. A value v is reached once the work of every value up to v has completed. A point is a device together with one of its values (Point). Work or host access follows other work when it starts after that work has completed, and so sees its writes. It follows a point when it starts after the point is reached.

An allocation is the memory that one call, such as Buffer.create, makes on a device. A buffer is a range of bytes in an allocation, and a borrow is a buffer on another device over the same allocation, without a copy (Buffer.borrow). Each allocation records, for each device, the point of its last write and the point of its last use. These points are its stamps.

rig orders work and host access by the stamps of the memory they use (Order). Two pieces of work need an order only when they share memory, so the order lives with the memory and no call names an event or a fence. submit returns the point of the work it runs, and a library passes points to later work only to order it after work that shares no memory with it. An allocation returns to its device some time after no buffer over it, and no array made of it, is reachable (Reclamation).

A function may write its result over the memory of an argument it no longer needs, which is called donation. A claim records that something reads an allocation, so that a donation writes only memory that nothing else reads (Claims). A donor writes over memory that it holds exclusive, by the only claim on it. It first consumes that memory, so every buffer made over the memory until then is dead from then on (Claim.consume).

A device is lost when it can run no more work, for good, such as after a fault of its driver (Loss). Unless a function's doc says otherwise:

• a call that takes a buffer raises Invalid_argument if the buffer is dead;
• a call that uses a device or a buffer's memory raises Lost where Memory after a loss says;
• a call that allocates raises Out_of_memory if its device cannot provide the bytes (Reclamation);
• a call may be made from any domain, concurrently with any other, and one that may take a lock raises Invalid_argument, having done nothing, when it is made inside a lock (Domains).

This copies five bytes to a device and back. The blit back reads dev, so it follows the blit that wrote it.

  let d = Result.get_ok (Rig.open_ opener) in
  let dev = Rig.Buffer.create d 5 in
  Rig.Buffer.blit_from_string "hello" 0 dev 0 5;
  let b = Bytes.create 5 in
  Rig.Buffer.blit_to_bytes dev 0 b 0 5

The sections below state every rule. Timelines, Order and Domains say how work is ordered, Loss what a lost device leaves, and Reclamation when memory returns. The module reference comes after them: Buffer for memory and host access, Running work for compiled code, Local for values a library keeps per device, and Claim for writing over an argument. A driver implements Rig_edge.

Devices

A machine has at most one open device per name.

Opening and closing

An open returns Error why when the device cannot be opened, and raises Invalid_argument when it is misused. An open of a lost device's name first runs the device's owed stop (Stops and forks). Every open is Error after the process has failed (fail), while the name's device is lost and its driver's stop (Rig_edge.Driver.stop) has not returned, when the machine's host (Machines) is lost or closed, and while 65,535 devices hold every device slot. For a lost or closed host, why is what that host's Lost prints.

A device holds a slot of the process's 65,535, a bound that comes from a point fitting an OCaml int (Point). It holds its slot while it is open, while memory of it or a value its submissions keep remains to return, and while it is lost and has not stopped (Stops and forks). A device that never stops, the process's first failure, and a device whose timeline passed 2⁴⁶ hold theirs for the rest of the process. A device that opens takes a free slot, and its values go on past every value the slot's earlier devices were given, so the number of opens a process makes has no bound. Each open still costs what its driver keeps for the process, as the driver's documentation states. The GPU drivers keep a few dozen bytes of C state per open. A device whose queue waits on another device's word through a mapping keeps one such mapping for each slot it waits on, of the slot's device it waited on last. One of an earlier device of that slot goes back at the first of three calls after the device's queue has passed the waits that read it. These are the device's next submit whose queue waits on a later device of a slot it waited on, a call that moves its collected memory (Reclamation), and its stop. A timeline word of a device's own goes back at the first call that moves collected memory after the device has left its slot and no other device maps it. A device lost before it reached every value it was given leaves its slot too, and keeps its record for the rest of the process, which its dropped points raise Lost with.

An exception raised inside an open, by the opener's make or asynchronously (Errors and failure), is raised again. The name is then unopened, or open with the device the open made, which the next open returns. A handle that make returned and that no device holds is stopped once (Rig_edge.Driver.stop).

Properties

A device's properties are set when it opens, and keep their values once it is lost. A device's properties, timeline and reason live as long as the program holds the device. Its points name it until a later device takes its slot (Point.device), and answer Point.wait and Point.reached for good. Its driver's stop releases what the device needs only to run work, such as its queues, and once its work has ended rig gives back its mappings of other devices' words and memory.

Machines

A machine is a host and the devices it drives. A library that uses another computer opens that machine's host with open_host, then the machine's devices with open_'s machine. A library must give each machine it connects to a name of its own for the life of the process, and treat a second connection to one address as another machine. Rig_remote runs jobs over the devices of several machines, and states how a job fails.

Timelines

submit assigns the values of a device's timeline, and so do the copies and fills of Copying and Image.load for the work the device runs, and Buffer.submit_copy and Buffer.submit_fill on host for a copy or fill the host makes, in the order the device accepts the work, whichever domain submits it. The device's submitted value is the last value assigned on its timeline (submitted). A device reports the values it has reached (signaled), and a lost device may report more (Loss).

A driver may hold submitted work until it is committed, released by the driver to the device, and a device runs only committed work. Every wait for a point, on the host or in another device's queue, first commits the work it waits for. A driver also commits on its own at least once every k values, where k is a constant of the driver's that its interface states, such as 256 for Metal (The C functions). rig does not read k. A device so reports each value at most k values late while work is submitted, and a program never commits by hand. signaled commits nothing, so the last values submitted may stay unreported until a wait or Point.reached commits them.

Order

Follows is transitive. These rules say which work and host code follow which:

1. Device order. On one device, the work of value v follows the work of every value below v, on every queue.
2. Parts. Within a submission, a part follows each earlier part that may share memory with it, where one of the two writes. Submission.make says which parts may share memory and which queue each runs on. A Submission.words part, whose effect rig cannot see, also follows every earlier part, and a later part on its queue follows it. Parts that need no order may run concurrently.
3. Memory. Work that reads an allocation follows every device's last write of it. Work that writes it follows every use of it, by any device.
4. Waits. The work of submit follows the points of its waits.
5. Host access. Host code after Point.wait p, or after Point.reached p answers true, follows p's work. Buffer.read's function follows every device's last write of the buffer's memory, and Buffer.write's every use of it, as host code after Buffer.wait b Read or b Read_write does. Buffer.submit_copy, Buffer.copy, Buffer.blit_to_bytes, Buffer.blit_from_string, Buffer.submit_fill and Buffer.fill order their work the same way, as work that reads the buffer they copy from and writes the buffer they write.
6. Host writes. Work submitted after a host write to memory that its device addresses sees the write. For another GPU's memory that the device addresses through a borrow, this rests on peer access between GPUs, whose status each driver's interface states.

Rule 3 works through each allocation's stamps. Once submit has assigned its value v on d, it hands the work to d's driver and adds (d, v) to the stamps of every allocation the work uses, also when the driver fails to take the work and d is lost. A copy that a device runs does the same. Work on host is done when its submit returns, so it adds no point. A later access reads the stamps of the memory it uses and follows the points they name, and a submit may wait for one on the host (Submitting).

Work through an Submission.unordered ref is the one exception to rule 3. It follows no other work on the allocation beyond its device's order, and no later work or host access follows it. The program orders it against other work on the memory, such as with atomics on flags that the work of two devices spins on.

An access follows the points that the stamps hold when it reads them, so the stamps order two calls only where the program orders the calls themselves. If two domains concurrently submit work on two devices that uses one allocation, and either work writes it, nothing makes either follow the other. Work and host access that start after both submits have returned follow both. Where neither work writes a whole word (Words) that holds bytes the other writes, bytes that both works write hold either work's values or a mix of them; otherwise the values of such a word are unspecified. A program that needs one work to follow the other orders the two, such as by starting the second submit after the first has returned. In the same way, a wait orders the host after the work submitted before it returns, and after nothing that another domain submits later. On one device, device order orders all work.

Words

rig orders memory in words. A word is the word bytes that start at a multiple of word from the first byte of an allocation, or of the host's address space for host memory. Two buffers share memory where they hold bytes of one word (Buffer.overlaps), and rule 2 and the claims (Claims) order them as one. A copy refuses only buffers that share a byte (Buffer.share_byte), since a copy writes bytes.

A non-empty allocation that Buffer.create makes holds whole words: its bytes past its length, up to the next multiple of word, hold unspecified values. Work and host code that access a buffer of such memory may read and write each whole word that holds a byte of the buffer, as an atomic on a word of elements narrower than a byte does. The bytes outside the buffer that such an access reaches are its own allocation's, in words the buffer shares, so rig orders them with it. On a device whose alignment is a word or more, as every GPU's and the host's is, the allocation starts on a word, so its words lie at addresses that are multiples of word.

Domains

Calls on several domains run concurrently, and the stamps order them only as far as Order states. Host access to one buffer from several domains needs synchronization of the program's own, as for any shared bigarray.

While a function waits it holds no lock of rig's, with two exceptions:

• open_ holds its name's lock while the opener runs, so opens of one name run one at a time while opens of other names go on.
• While a driver's room, submit or commit blocks, where its driver declares may_block, other submits and commits on that device wait for it. On host and a memory_device, whose submit runs its work before it returns, other submits on the device wait for that work before they take a value. Waits for the device's earlier values return meanwhile.

A wait releases the domain lock while it blocks, and returns to OCaml at least every 200 ms, where a pending Sys.Break from Ctrl-C raises. An interrupted wait loses no device and assigns no value.

rig also holds locks of its own, briefly, inside most calls, and a driver or an io library holds its own inside its calls and inside the functions of its record that compiled code calls (capability). A signal handler, finaliser or Gc.Memprof callback runs where the code it interrupts allocates or polls, which may be inside such a lock on the same thread. A call is made inside a lock when such a callback makes it while its thread holds a lock of rig's, of a driver's or of an io library's.

A call to rig that may take a lock, made inside a lock, raises Invalid_argument as it starts, having done nothing, but for submit, which frees its run. It raises even where it needs none of the locks its thread holds, since the thread's count of its locks does not say which they are. A function that takes no lock may be called from a callback at any point, and never raises it: name, equal, pp, arch, computes, runs_on_host, alignment, cache_bytes, capability, host_of, reaches, submitted, lost, failure, budget, allocated, Point.device, Point.value, Point.reached, Point.pp, the properties and low-level values of Buffer, Buffer.view, Buffer.reached, Buffer.of_bigarray, Image.device, Image.name, Image.image, Image.params, Image.threads, Image.entry, Submission's refs and parts, Submission.block, Submission.address, Submission.Run, Claim, Profile but Profile.take, Local.new_key, and Local.get of a value already made for the device. Nor do signaled and fail. Outside a lock, a call from a callback runs as any call does. If the callback lets the exception escape, it ends the interrupted call as an asynchronous exception (Errors and failure). The callback must not block either, in a wait of rig's or for another thread: other calls, and every round of reclamation, wait meanwhile for the lock its thread holds. A program that calls rig from such a callback handles that exception, or has the callback record the work for the program's own code to do, such as in an Atomic.t.

Loss

A device becomes lost when:

• its driver reports a fault, or fails to take or commit work;
• another exception escapes its driver while the driver changes the device's memory or code (Errors and failure);
• its progress stands still past its hang bound (Hangs);
• its queue waits for a value that a lost device never reached;
• close ends it;
• fail fails the process;
• in a child of fork, it is a driver's device the child inherited (Stops and forks).

A loss is final. Opening the device's name again makes a new device, unequal to the lost one, with a timeline of its own, and a lost device's properties keep their values. A loss spreads to no other device except along queue waits, as below. A fault of something that several devices share, such as CUDA, whose fault fails every later CUDA call of the process, loses each device it reaches through that device's own driver.

rig treats as completed the values a device had reached when its loss was recorded. The work of later values may have run in full, in part or not at all, and no program can tell which. Values the lost device reports past those are dropped. They are never reached, and they count only for the return of memory (Reclamation). The loss spreads along every chain of queue waits for values the lost device never reached.

Memory after a loss

A use of memory depends on the points whose work must have completed for the use to be valid.

• A read depends on the points of every device's last write of the memory. This holds for Buffer.wait b Read, Buffer.read, Buffer.blit_to_bytes, the source of a Buffer.copy, a submit's read, and a claim that rig_buffer_claim takes for reading (Low level).
• A use through an Submission.unordered ref depends on no point.
• Every other use depends on every point of the stamps, such as a write, a borrow, Buffer.wait b Read_write, Buffer.write, Buffer.export, and a claim that Claim.with_ takes. A Claim.with_ claim names no access, so it depends on what a write would.

After a loss, these raise Lost:

• every use of the lost device, of its memory through any buffer, and of a borrow on it;
• every use of other memory that depends on a point the lost device never reached, since that point's work may never have run.

A point the lost device never reached stays so. Memory with a last write at such a point raises Lost at every use through rig's calls but an Submission.unordered one, for the rest of the process. Memory that the lost device only read raises it only for the uses that depend on every point, and only until the device has stopped (Stops and forks): from then on its unreached reads count as done, since they no longer run and changed no memory. An array Buffer.export made over such memory before the loss reads and writes it with no check of loss, and its bytes may be the old ones, the lost work's, or a mix.

These rules hold for work that writes only the memory its submission names. A kernel that writes outside it may have changed any memory its device addresses, before or after a loss.

Hangs

A driver whose work no kernel or vendor timeout bounds declares a hang bound, hang_ms in Rig_edge.facts. A wait checks the bound's clock while it watches the device. The clock starts when a wait first finds committed work of the device unreached. It starts again when a wait finds that the device has reached a later value, and when a wait finds the device idle, with all its committed work reached or its queue waiting for another device's unreached point. A device whose clock passes the bound is lost with the reason "no progress for n ms". Only a wait checks the clock, so no device is lost to a hang while nothing waits for it.

Stops and forks

A lost device has stopped once its driver's stop (Rig_edge.Driver.stop) has returned and the device reports its last submitted value. Only then do its memory, once collected, and the values its submissions keep (Submission.make) return. Its mappings of other memory (Buffer.borrow) go back when its driver's stop returns if it has stopped by then, and otherwise at the first free_cache of it, Buffer.create or Buffer.borrow on it, or round of reclamation (Reclamation) after it has stopped. A device that never reports that value keeps them for the life of the process. A device whose driver keeps its word of its own (Rig_edge.facts) reports a value only when rig reads that word: at its stop, at signaled, and at the calls that move its collected memory. Until such a read, rig counts it as not stopped.

From its loss on, a device's stop is owed until a call runs it. Only close, after waiting for the device's driver calls in flight, and an open of the device's name (Opening and closing), once none is in flight, run it. No other call does: neither the call that lost the device, nor a use of it, nor fail, nor reclamation. A lost device that is never closed keeps its stop owed, and with it its memory, its slot and the values its submissions keep. Until its stop, the work its driver took may go on running: a GPU driven over PCI, with no kernel driver to end its work, may keep running it, reading and writing the memory it addresses. So a program that has no more use for a lost device closes it, and the device's memory returns once it has stopped. A GPU driven over PCI whose memory another GPU maps still, or may still write, stays taken past its stop, its own work ended: an open of its name is Error until that GPU's mapping of the memory has gone and it can no longer write it, so that no open resets the GPU and gives that memory out meanwhile.

A call that runs a stop takes the time of the driver's stop, and raises again any exception other than a fault that escapes it. Since no call runs a stop but a close and an open, an exception that ends the call that lost a device, such as Ctrl-C's, never lands in its stop.

In a child of fork, every driver's device inherited from the parent is lost, with the reason "forked" unless it was lost before, and its points count as never reached. The child never calls that device's driver, and a submission on it made before the fork keeps nothing: once the submission is collected, the child's next call that moves collected memory (Reclamation), on any device, lets go of the value it keeps, without waiting for its work. A value kept by a submission on host returns as in any process, so one kept after work a thread of the parent was submitting on host at the fork stays until the child's next submit on host. rig leaves a live io device as it was, and its library decides what a fork does to it, as a child of fork keeps the open files of its parent. An io device whose stop has not returned at the fork, owed or under way, stays lost in the child, which runs no stop of it and frees none of its memory, since that stop is the parent's. The host's staging memory that a thread of the parent was copying through at the fork (Copying) stays in the child for good. A device the child opens is its own.

Errors and failure

An exception raised inside a call to rig leaves rig's own state consistent: its devices, names, timelines, stamps and memory. Every later call behaves as its doc states as far as that state decides it. No later call waits on the call the exception ended, and no name stays opening.

An asynchronous exception is one the runtime raises where the call allocates OCaml values or polls, such as Sys.Break, Stdlib.Out_of_memory, or an exception from a signal handler, a finaliser or a Gc.Memprof callback, or where a function of the call starts, as Stack_overflow is. It may leak the memory of the call it interrupts, but none that a driver's call made for it, nor memory an allocation took from the cache: an exception once the driver's call returned leaves it to return as it would have, once unreachable or at the device's stop, and loses no device. One that interrupts a driver's stop (Stops and forks) may leave that device never stopped, holding its memory and its slot for the rest of the process.

rig blocks every signal on the calling thread, but those a fault raises (SIGSEGV, SIGBUS, SIGFPE, SIGILL, SIGTRAP, SIGABRT and SIGSYS), across each call it makes to a driver or an io library but the reads of a word or an address and the check of a peer, which wait for nothing, and the driver unblocks them only inside its waits. A signal that comes while such a call runs outside its waits is delivered at the call's next wait or once the call has returned, so its handler, a Ctrl-C's Sys.Break included, never ends the call there. One that comes during a driver's wait runs its handler before the wait returns, and its exception ends the call as an asynchronous one. Reclamation (Reclamation) that gives memory back to a driver one region at a time blocks signals from its first region on, rig's own code between the regions included, and gives the thread its mask back after every 64 regions and at its end: a signal waits for at most 64 regions, and a finaliser or Gc.Memprof callback that runs at a poll point among them runs with signals blocked. A function of a driver's library that a program calls directly, such as an opener or a function of a capability, blocks signals in the same way, but for one whose doc says it runs the program's code. A driver's finalisers and exit handlers take no signal, even in their waits. A thread that rig starts blocks every signal for its life. Blocking applies to the calling thread only: a signal sent to the process while this thread blocks it goes to another thread that does not, and a program with several threads chooses the one that takes it with Thread.sigmask. On Windows, where OCaml has no signal masks, rig blocks no signal: one runs its handler at the next allocation or poll point of any thread that runs OCaml code, a driver's call, finaliser or exit handler included, and the handler's exception, a Ctrl-C's Sys.Break included, ends that call there as an asynchronous one.

A driver changes the memory and code that a device's work uses when it allocates, maps or gives back memory, and when it loads, looks up or unloads code. An exception other than a fault that escapes the driver during such a change may leave it half done. rig then loses the device, with the exception's text as the reason (Lost), and raises the exception again. Calls that allocate, borrow, load or ask for a function (Image.resolve) make such changes, and so do reclamation (Reclamation), on whichever device's memory returns, and a submit that gives back its device's mappings of other devices' words (Opening and closing). An asynchronous exception from any of these calls may therefore lose a device. Every other exception leaves the device serving later calls, such as one that interrupts a wait (Domains), one from an io device's call, the Invalid_argument of a function the device cannot run (Image.resolve), and that of a call made inside a lock (Domains).

Reclamation

Memory returns on its own once collected, or early through Buffer.free. After its use, the life of an allocation goes like this:

1. Collection. Once every buffer over the allocation, and every array Buffer.read, Buffer.write or Buffer.export made of it, is unreachable, the OCaml collector collects it, on whichever domain the collector runs. A prepared submission keeps the buffers it uses reachable (Submission). Buffer.free skips this step.
2. Waiting. It waits until every point of its stamps, and of the work that used it through Submission.unordered refs, which rig records beside the stamps, is reached or dropped.
3. Reuse. It then enters its device's cache, the device's returned memory, which Buffer.create reuses with empty stamps. A new buffer over it waits for no work of its earlier use.
4. Release. Cached memory returns to the driver when the cache is freed (free_cache), when the budget is lowered (set_budget), or when an allocation needs the room. Staging memory that no copy holds (Copying) returns at these times too, once the work that used it is done. An allocation that finishes waiting while its device holds more than its budget skips the cache, as does an io device's memory, and returns to the driver. A lost device's memory returns as Stops and forks states.

Memory that an asynchronous exception leaked (Errors and failure) may never take these steps.

Collected memory moves to the cache or the driver only inside rig's calls, on any domain. Buffer.create, Buffer.of_string, Buffer.of_io, Buffer.submit_copy and Buffer.copy first move the collected memory whose points are reached or dropped on the devices they use, and let go of the values that collected submissions on those devices keep, once the work submitted on the device before the first such call after the collection is done, also when they then raise Lost. free_cache moves memory on its device, reclamation on every device, and a lost device's stop on its own. Each of these moves first releases the holds collected on any device (Claim.hold), and moves the memory and lets go of the kept values collected on any device whose stop returned, such as a closed one. Memory collected while a domain blocks returns through other domains' calls. A device reports its values up to k values late (Timelines), and memory and kept values return by what it reports.

A device's budget is the most bytes it holds at one time of the memory that counts in it: its live buffers, its loaded images' code and its cache (budget), each allocation by its whole words (Words). Pinned memory counts in the host's budget. A host buffer counts the bytes it takes of the C library's memory: its whole words, and up to 112 more where it starts on 128 bytes (Buffer.create). On Windows, where free releases no allocation aligned to a page, one that starts on a page counts a page less 16 bytes more. An allocation's bytes also pace the OCaml collector, so a device nearing its budget collects unreachable buffers more often. Live buffers are freed only by Buffer.free, so a program that keeps every buffer reachable and frees none runs out of memory however large its budget.

• A dropped host buffer holds its memory until a collection frees it: a minor one for a buffer the minor heap still holds, and a major cycle otherwise. A buffer starts in the minor heap, unless it holds more than custom_minor_max_size (Gc.control) bytes and is as large as the bound below, or that parameter is 0. Once the bytes of host buffers that reached the major heap since the last cycle reach a bound, they complete one cycle's work of the collector. A buffer that starts young of more than custom_minor_max_size bytes counts an eighth of its bytes at birth as well, so buffers that die young still bring a cycle per eight bounds of them. The bound is custom_major_ratio / 150 of the program's memory, the allocating domain's major heap and the host buffers live at the last cycle, or half the host's last-level cache if that is more, and at least 4 MiB. In a small heap a cycle also does the runtime's minimum work (small_heap_limit of OCaml 5.5's runtime). A dropped buffer returns with the sweep after the marking that finds it unreachable, so its memory returns within about two cycles. Above both floors the bound is proportional to custom_major_ratio, so raising the ratio holds more dropped memory between cycles and runs fewer of them. rig sets none of the runtime's parameters. The bytes of buffers the minor heap holds bring a minor collection once they reach the same bound.
• A dropped buffer of another device holds its memory until a collection frees it, as a host buffer does. A budget's room is the bytes it has left. Memory that counts in the host's budget, a driver's device's Pinned memory, brings a cycle as host buffers do, floors included, beside the bytes live in the host's budget at the last cycle; or sooner, once the dropped bytes reach custom_major_ratio / 150 of the host's room. The memory of a memory_device, of a driver's device whose driver declares host_addresses (Rig_edge.facts), and a driver's device's Mapped memory brings a cycle as host buffers do, floors included, with the device's allocated bytes standing for the host buffers live; or sooner, once the dropped bytes reach custom_major_ratio / 150 of the device's room, so a device nearing its budget collects in time. A device's other memory brings a cycle at that share of its room alone. Either amount is at least a page, and for a buffer of at most custom_minor_max_size bytes at least the minor heap's bytes.
• The host keeps collected buffers of 64 KiB or more for reuse. It keeps up to 32 MiB of them, and beyond that only while the bytes of its kept and live buffers together stay within the peak of its live buffers over the collector's current and previous major cycles. Pages of a host buffer that reuses none fault in at their first write, in the call that writes them.
• On a driver's or an io device, an allocation larger than the budget it counts in raises Out_of_memory immediately, without freeing the cache.
• An allocation that its budget or its driver refuses first reclaims. rig frees the cached memory charged to that budget, waits for the work that holds back other collected memory, from the second round on returns staging memory that no copy holds, and tries again. The second round also runs the collector before it tries. The third runs it again only if, since the second's run began, a finaliser ran or rig let go of a value that may hold memory, such as a collected submission's keep (Submission.make): only those make memory unreachable that the second run missed. The allocation raises Out_of_memory after three such rounds.
• When a driver has no room for a mapping that a borrow (Buffer.borrow) or staging memory needs, such as under a limit on locked memory or on mappings, rig returns memory once and asks again. The process's devices may share such a limit, so rig runs the collector, waits for each device of the machine that has unfinished work, and returns on each the mappings of borrows collected or ended, the staging memory no copy holds and the cached Pinned memory. It frees no other cached memory. It raises Out_of_memory if the driver still has no room.
• A growth of a device's queues (submit) that its driver has no memory for reclaims as an allocation does, both the device's budget and the host's, as the driver may grow them in memory that counts in either.

Buffers

Running work

A device runs code from an Image loaded on it. A Submission describes work for one device once, and submit runs it with one submit's buffers and returns the point of that work on the device's timeline.

A library runs work like this:

1. It loads an image (Image.load), resolves its functions (Image.resolve), and makes a submission of parts that launch them (Submission.make).
2. It takes a run of the submission (Submission.take), which holds the launches' grids and parameters, and stores them.
3. It calls submit with the run and this submit's buffers, which frees the run. Each later submit takes a run again, stores its parameters and submits it.
4. Once the submission is unreachable, its memory returns after the work of its last submit is done.

Objects has an example.

Images

Host code

host runs code that the process links from relocatable ELF objects, on its own cores. Image.load on host links an object into executable memory once (Linking), and the image's functions are the named symbols of its executable sections. The host's queue runs only launches.

Launches

A launch runs on the thread that submits it and on the workers of the host's pool, which the host's other parallel code, such as nx's CPU kernels, shares, and it is done when submit returns.

A host function's C type is

void f(const void *params, int64_t first, int64_t end);

params points to the launch's parameters, in which each ref holds the host address it names (Submission.ref). The launch's threads are numbered g * t + i, where g is a group's index in the grid with x fastest, t the threads of a group, and i a thread's index in its group with x fastest. Each call of f runs the threads first to end - 1, which are whole groups, and the calls together run each thread once. The calls run in any order and in parallel, on at most as many threads as the host has performance cores (Performance cores), so a call must not write what another reads. A launch with shared memory, with an axis of zero groups or threads, or with more than 2⁶² threads never runs, and submit raises.

Performance cores

The performance cores are the host's cores that run compute-bound work at more than half the speed of its fastest, bounded by the cores the process may occupy. Their count is taken at its first use and stays the same for the life of the process. They are:

• On Linux, the CPUs in the affinity mask of the first thread to ask whose reported capacity is more than half the largest, or all of them where no capacity is reported, and at most the process's cgroup v2 quotas allow.
• On macOS, the performance cores where the system reports them.
• Otherwise, the cores the process may occupy.

Objects

A C compiler makes such an object, for instance with

clang -c -O2 -fPIC --target=x86_64-none-unknown-elf f.c

for an x86_64 host. For an arm64 host the flags are --target=aarch64-none-unknown-elf -ffixed-x18, since macOS and Windows reserve the register x18.

For instance, this function adds the floats of two arrays into a third:

#include <stdint.h>
struct add { const float *x, *y; float *z; int64_t n; };
void add(const struct add *p, int64_t first, int64_t end) {
  for (int64_t i = first; i < end && i < p->n; i++)
    p->z[i] = p->x[i] + p->y[i];
}

Loaded once, it runs over n floats in 64 groups:

let image = Result.get_ok (Rig.Image.load Rig.host obj) in
let add = Option.get (Rig.Image.resolve image "add") in
let refs = Submission.[| read 0 ~at:0; read 1 ~at:8; write 2 ~at:16 |] in
let s = Submission.(make host [| launch ~refs add ~params:32 |]) in
let run = Submission.take s and k = Submission.block s 0 in
Submission.Run.groups run k 64 1 1;
Submission.Run.threads run k ((n + 63) / 64) 1 1;
Submission.Run.int64 run k 24 (Int64.of_int n);
ignore (submit run ~buffers:[| x; y; z |])

Linking

Image.load on host links an object that meets these rules, and is Error for any other:

• The host is x86_64 or arm64, and the object is a well-formed 64-bit, little-endian, relocatable ELF object (ET_REL) for the host's machine (EM_X86_64 or EM_AARCH64).
• No allocated section is writable and not empty, as a .data or .bss in use would be. Code that needs a variable gets it through its parameters.
• Each relocation holds its addend in its entry (SHT_RELA), is of a type listed under Relocations, uses no symbol defined outside the image, such as a common or thread-local variable, patches inside the image, and has a value that fits its field.
• Each symbol the object refers to and does not define is defined by the process (Symbols).
• The image and its stubs fit in max_int bytes.

The link lays out the object's allocated sections (SHF_ALLOC), its code and read-only data, in one image, each at an address that is a multiple of its alignment (sh_addralign). It writes the image once, applies its relocations, then makes the memory executable. The memory is never writable and executable at the same time (Platform support).

Relocations

The relocations are those a compiler emits for position-independent code of the small code model, with -fPIC, that calls functions and reads its own data. For a relocation at the place P of the image, of the symbol whose address is S, with the addend A, the link writes:

• R_X86_64_PC32: S + A - P.
• R_X86_64_PLT32: S + A - P, for a call or jump, through a stub when S is out of its reach.
• R_AARCH64_ADR_PREL_PG_HI21: Page(S + A) - Page(P).
• R_AARCH64_ADD_ABS_LO12_NC, and R_AARCH64_LDST8_ABS_LO12_NC to R_AARCH64_LDST128_ABS_LO12_NC for loads and stores of 8 to 128 bits: the low 12 bits of S + A.
• R_AARCH64_CALL26 and R_AARCH64_JUMP26: S + A - P, through a stub when S is out of reach.

A stub lies after the image for each symbol the object refers to and does not define, with a word that holds the symbol's address. An x86_64 stub jumps through the word and uses no register. An arm64 stub loads the word into x17, which a veneer may use.

References.

• The x86-64 psABI, chapter 4.4: relocation types and their formulas.
• The ELF for the Arm 64-bit Architecture, section 5.7: relocation types, their formulas and fields, and the registers a veneer may use.

Symbols

A symbol the object refers to is its definition in the object, or else its definition in the libraries the process loaded into its global scope, such as the C and math libraries. A function of another image is called through its address, which an entry ref places in the launch's parameters (Submission.entry).

A compiler also calls functions the source does not name, such as __extendhfsf2 where the machine has no instruction for a 16-bit float conversion, or memcpy for a large copy. The object defines these, or they come from the process and follow the process's convention. On x86_64 Windows that convention differs from that of an object for no system, so code for Windows defines every function the compiler calls, or is compiled for Windows' convention (Calling conventions).

Calling conventions

Linked code follows its object's convention. A compiler that targets ELF without an operating system uses System V AMD64 on x86_64 and AAPCS64 on arm64. x86_64 Windows has its own convention, but the host still calls a function by System V. A function of the process's libraries follows the platform's convention, so on x86_64 Windows linked code declares those it calls __attribute__((ms_abi)), or is compiled for Windows' convention, as with --target=x86_64-w64-windows-gnu-elf -mno-stack-arg-probe -mno-red-zone, and declares the functions the host calls __attribute__((sysv_abi)). Such code's long has 32 bits. Code for an ELF target probes no stack, and Windows commits a thread's stack one guard page at a time, so on Windows a thread's stack is committed, but for its last 64 KiB, before the thread first runs linked code. A stack frame of any size within it then runs, as elsewhere.

Platform support

The host links code on x86_64 and arm64 hosts running Linux, macOS or Windows. On arm64 macOS the code's memory is mapped with MAP_JIT, whose write protection only the linking thread lifts while it writes. Elsewhere the memory is mapped writable, then made executable and read-only once written.

Executable memory on macOS

On macOS, a program built with the hardened runtime maps executable memory only with the com.apple.security.cs.allow-jit entitlement (arm64) or com.apple.security.cs.allow-unsigned-executable-memory (x86_64). Without it a load is an Error. With com.apple.security.cs.single-jit, a process maps executable memory once, so its loads after the first are Errors.

With com.apple.security.cs.jit-write-allowlist, the system writes code only through functions that the executable, or a library it loaded at start-up, lists. rig's C stubs list theirs, so loads work where the stubs are linked into the executable: in native programs, and in bytecode ones built with -custom or -output-complete-exe. In a bytecode program that loads the stubs at run time, as ocamlrun does, the system ends the process at its first load. Link the stubs into its executable, or give it the com.apple.security.cs.jit-write-allowlist-freeze-late entitlement and call pthread_jit_write_freeze_callbacks_np once the stubs are loaded, before its first load.

Submissions

Valid submissions

A submission is valid when its device runs submitted work and has a queue for each of its parts, as Rig.Submission places them; its images are live; its parts' buffers and its fixed memory are live, held exclusive by no Claim.with_, and on the device, but as Submission.copy admits; no fixed buffer it writes admits only Read; every slot below the highest is named, and none by both an Submission.unordered ref and an ordered use; and its launches and fills fit a run: 2{^31} - 1 bytes, each counting 16 bytes plus its parameters rounded up to a multiple of 16. Submission.make raises for each.

A submit is valid when it passes one buffer per slot; no buffer its work uses is dead or held exclusive (Claim); each buffer is on the device (Buffer.borrow), but as Submission.copy admits; it writes through no Read buffer (Buffer.access); each copy's two buffers have one length and share no byte; no ref names handle-only memory (Low level); and its driver can run the work (The C functions). A driver may bound the words one submission places on its queues, its own packets' included, where its queues cannot grow past a size, as its interface states. submit raises for each.

Submitting

A submit's work is ordered by the rules of Order. The device's queue waits for another device's point where its driver supports that (Rig_edge.waits). Otherwise submit waits for the point on the host. A submit for which the driver's room finds no room on the device's queues waits until the device has reached one more value, and asks again, holding no lock while it waits. A driver may instead block in its submit until the work fits, as Metal's does, and other submits and commits on the device then wait for it (Domains).

Values per device

A library keeps values per device under a key, such as the image it loaded on each device and the workspace its code uses there. A key's value on a device is what the key's init makes at the first Local.get on that device (Local.new_key), and it lives as long as the device. Once the device's driver's stop (Rig_edge.Driver.stop) has returned, rig drops the value, so the memory it holds can return, and close returns after the drop. A reopened name is a new device, with values of its own.

Claims

Profiles