Module Munin_tui

The dashboard shows a Munin.Store.t in a terminal, reads it again once a second, and never writes it. It has two screens. Runs lists the store's runs, to find a run and compare runs by their params and metrics. Run monitors one run, the focus: its state, its metrics drawn over its steps, and its machine. A marked run is drawn in every chart of Run beside the focus, so the focus reads against earlier runs of a sweep. Run's charts follow a metric cursor, and Runs shows the cursor's metric, the focus metric, in a column. Three views open over the screens: Zoom, one metric of Run at full size with a step cursor; Details, every fact about one run; and Help.

Each screen's last line names its keys (Keys). The screens form a stack whose bottom is Runs. Enter opens the run under the cursor in Run, and Esc goes back one screen. A dashboard that starts on Run has Runs below it, so Esc from either start ends on Runs, and only q and Ctrl+C quit.

The screens draw no boxes. Two weights rank what they show: normal text for what the user reads, dim for what frames it. They state each fact in words or glyphs, and colour repeats what they say, so a terminal without colour, or under NO_COLOR, shows every fact. Text takes the terminal's own foreground and background, and other colours are the 16 colours of the terminal's palette, except the cursor row's tint at 256 colours and more; with fewer, the cursor row is reverse video.

The dashboard reads a run as its library does (Munin.Store_format). Unless a section says otherwise, a value shown is the run's as last read, and a time is the dashboard's clock against a time the store recorded.

Runs

Runs has a row per run of the store under a header row per experiment, the experiments in the order of their newest run, and a cursor on one row. Its first line counts the runs, the live ones, the failed ones, the crashed ones and those with a metric sample that is not finite, each count the rows that show its glyph; with a filter, the rows it keeps. A value that every row shows equal, such as the experiment of a filtered sweep or the host, is stated once on that line, and leaves it first when it is too long. The store's path ends the line, cut at its start. An experiment's header row counts its runs, its live ones and its failed or crashed ones.

The columns, left to right, are the run's state word (States), its name or else its id, its parent, its start time, the time from its start to its end or its last write, its latest step with its progress while short of its steps (Numbers), the focus metric's value, the first sample of the run's metrics that is not finite, as × NaN at 4,250, its params, its host and its commit. The value is the run's best sample by the metric's goal (Munin.Metric.best), or its latest without a goal, and a * follows the best value of each experiment among the rows shown. The parent, the host, each param and the commit are columns only when their values differ among the rows shown, a param column with — for a run without it. A param whose values differ within an experiment comes before one that differs only between experiments, and among them the param with the most values first. Columns that do not fit are left out whole, in this order: the commit, the params that differ only between experiments, the host, the time from its start, the start time, the params past the first, the parent. The column the rows are ordered by carries ▾, or ▴ reversed.

The focus metric is chosen by the first listing whose rows have a metric: the metric with a goal that the most shown rows have, the first by name among equals. After that only left and right move it, through the metrics that any shown row has, by name, and a row without the focus metric shows — in its column.

In the lines the rows leave, once they are 7 or more, the cursor run's pane states the run and its params, its focus metric's latest and best values, and draws the metric over its steps; at 140 columns or more a list of its metrics' latest and best values sits beside the chart. With fewer lines, the third line from the bottom states the cursor run.

A filter, typed after /, keeps the rows of which each word of the filter is part of the name, the id, the experiment, the parent, a tag, the state word or a param written key=value, ignoring case. The rows are filtered as each key is typed. Enter keeps the filter and Esc clears it.

The values come from each run's statistics (Munin.History.stats_of), which are an ended run's hint, and for a run without an end a history the dashboard follows while Runs is shown.

Run

Run's first line states the focus in words (States): its state word, its name, its progress, its rate, the time it has left, its active time, and the age of its last arrival or its last write (States). Parts leave the line whole when it is too long, the rate and the time left first. The second line, shown only when something needs attention, starts with ! and names a sample that is not finite in the focus or a marked run, a stalled or crashed run, a run whose lock is held and from which no metric sample has arrived in the 30 s since the dashboard began watching it, or damaged lines in a log.

Then comes one chart, the cursor's metric (Charts), under a title with its latest and best values and, with marked runs, a legend that names each marked run and the params in which it differs from the focus. Under the chart each marked run has a pair of difference rows: the focus's sample minus the marked run's at each step both logged with finite values, the marked run's latest sample at a step taken, averaged per dot column, around a dim zero rule; at the right the difference at the latest such step, and whether the focus is worse or better there by the metric's goal, with that step. A marked run that stops before the focus's latest step, by an end or a sample that is not finite, says where after its row's end. A metric is a key of the run's metrics with a sample (Munin.History.metrics).

Under them each metric has a strip: the metric drawn over the chart's x domain in the chart's own columns, one or two rows high, its name in a column at the left and its latest value at the right. Strips are grouped by the prefix of their keys, up to the last /, which the first strip of a group prints. They are two rows high when all fit so, with a blank row between groups when that fits, and otherwise a window of them, which the metric cursor moves, shows as many as fit and counts them. The chart keeps at least three fifths of the lines that the first lines, the difference rows and the machine leave, four fifths below 30 lines.

The machine's readings come last and follow the USE method, with a row per resource that gives its utilisation, its saturation and its errors. The keys are those of Munin_sys. munin logs no errors, which the rows state once. A device's rows give its used of its memory, the device's whole memory, apart from the allocated bytes of the run against its budget. At 140 columns or more each row adds the reading drawn over the run's time. Below 30 lines one line gives the first reading of each resource. The readings of a run that is not live, running, writing or starting give the time of the last one.

Zoom, Details and Help

Zoom draws the cursor's metric at the terminal's size with a step cursor on the focus's samples, which left and right move one sample, and a table of each run's value at the cursor, its latest value, its best value and its step, and counts of samples, non-finite samples and orphans. Below 100 columns the table keeps each run's value at the cursor and its best value. Details lists, in a column at most 104 cells wide, the run's id, start and end, its source, its params, beside each the value a marked run gives it when that differs, its metrics' latest and best values, its notes and tags, the artifact versions it used, the files it logged, its resumes and its damaged lines. It scrolls. Help lists, over the screen it opens on, the keys of every screen, the keys of that screen, and the words and glyphs of the screens; it scrolls when it does not fit. Below 40 columns or 10 lines every screen gives its gist in a few centred lines, the focus's state word, name, progress and latest value, or the counts of Runs, and the size it needs.

Charts

A chart draws the focus's samples of a metric as a line in braille dots, over a domain of x and of y, and each marked run's as a level line.

x. A chart's x is the step, or with x the metric's step metric, or the time. A sample on the step metric's axis takes the value of the step metric's sample at the same step (Reading a log), and a sample without one is not drawn; the title counts them. The time is the run's active time, the time from its start less the time between a writer's last record and the next writer's resume. The x domain runs from 0 to the focus's steps on the step axis of a run that declared them and has no sample past them, and otherwise to the least of 1, 2 or 5 times a power of ten at or above 1.1 times the largest x, so that it changes a few times over a run and holds every sample, or to the largest float where no such float is.

Reduction. The samples of a dot column, the samples whose x falls in that column of the plot, are reduced to the first, the least, the greatest and the last of its finite samples, joined in order. The line through them inks the dots that the line through every sample inks.

Non-finite samples. A sample that is NaN or infinite takes no part in a domain or in smoothing, and the line breaks across it. The row above the plot marks each dot column that holds one of the focus's with × for NaN, ↑ for an infinity and ↓ for a negative one, × winning. Two marked columns with no finite sample between them are joined by a dotted rule, so that a metric that stays NaN reads as one span however often it logs. A marked run's are named on the attention line.

Orphans and resumes. A sample that a resume orphans is never drawn (Munin.History.orphans). Each resume is a dotted vertical line at its step. Where a run's steps decrease without a resume, its line breaks, and smoothing starts again.

y. The y domain is the extent of the finite values drawn in the x window, the focus's and the marked runs'. Nothing is trimmed, so a spike sets the domain; a narrower x window (+) refits y to the samples in it. With L a chart whose finite values in the window are all above 0 draws y on a log scale, and another says linear: values <= 0. The y axis is labelled at both ends of the domain, the least and the greatest value drawn, and between them at multiples of 1, 2 or 5 times a power of ten about a third of the plot apart, none closer than two lines to another; on a log scale at the powers of ten. A dot follows the focus's latest sample when it is finite.

Smoothing, off by default, draws a smoothed line over the raw one, which stays drawn, dimmed. The smoothed value at a step x_i is the debiased mean of the finite samples at and before it, s_i = y_1 + Σ_j w_j (y_j - y_1) / Σ_j w_j, with w_j = (1 - exp (-(x_j - x_(j-1)) / τ)) * exp (-(x_i - x_j) / τ), where x_0 = x_1 - (x_2 - x_1), or x_1 - 1 for a lone sample. The samples that share a step count as one sample, their mean, at that step. s cycles τ through a hundredth, a thirtieth and a tenth of the shown x span, rounded to 1, 2 or 5 times a power of ten, and the legend prints it. A latest and a best value are always the raw samples'.

Marked runs. A marked run's level line joins, per dot column, the mean of its finite values in that column, smoothed when smoothing is on. It is drawn before the focus so that the focus loses no dot, and carries its letter, a or b, at its end, beside it when the focus or the other letter holds that cell, and in cells along it that the focus does not ink. Where the two lines share a cell the focus's dots replace the marked run's, so a level line shows the level of a marked run and promises none of its extremes; Run's difference rows show the gap the overlay hides. Its points are means over a dot column, so a narrower chart averages more samples per point; a marked run's values are read from Zoom's table. At most two runs are marked. A third level line shares the cells of the other two at braille resolution, and without colour no reader can tell it from them.

States

A run's state word comes from its status as read (Munin.Run.status) and from when its samples arrived:

• starting: a process holds the run's lock and the run has no metric sample;
• running: a process holds the lock and no metric sample has arrived since the dashboard began watching the run;
• live: a process holds the lock and a metric sample has arrived, within three usual gaps once they are known;
• stalled: a process holds the lock and no metric sample has arrived for longer than three usual gaps;
• writing: the lock is not probed and a metric sample has arrived, within three usual gaps once they are known;
• silent: the lock is not probed, and no metric sample has arrived since watching began, or for longer than three usual gaps;
• crashed, finished, failed and killed: the run's status.

The dashboard times each arrival on its own clock. A poll that finds new metric samples of a run is an arrival at that poll's time, and a run's samples that the first read finds are none. The usual gap is the median of the gaps between the run's last 32 arrivals, known once two arrivals have been timed, and three of them are never less than 10 s. No time the run recorded judges its state. Run's first line gives the age of the last arrival, or before one, the age of the last write (Munin.Run.updated_at) labelled as such and how long the dashboard has watched. A run that is writing or silent may have exited, which its line says by lock not probed.

Numbers

• A value prints with four significant digits, its trailing zeros kept, as 0.4077, 2196 or 6.200, so that values line up digit for digit; in scientific notation below 1e-4 and from 1e6 in magnitude, as 9.908e-05, so that no value that is not zero prints as 0. NaN prints NaN, and the infinities inf and -inf. A param prints as the program gave it, in the fewest digits that read back as its value.
• A step or a count prints with separators, as 12,400.
• A duration prints in at most two units, as 45s, 1h37m or 3d4h.
• Memory prints in GiB and a rate of bytes in MB/s.
• The latest step is the largest step of the latest samples of the run's metrics. Progress is the latest step over the run's steps (Munin.Run.steps), and absent without them; its percentage is rounded down, so it reads 100% only once the steps are reached.
• The rate is the steps per second over the last 10 minutes, or over the last tenth of the active time when that is longer, once that window holds 60 s of samples. The time left is the steps left over the rate, shown as ~59m left while the run is live or writing.
• Times of day are local, as 10:41 today and Oct 31 16:20 before.

Keys

A key means one thing on every screen, and a key whose object the screen lacks does nothing. Keys are matched with their case.

up down, k j     move the cursor: run, metric, line; scroll Help
left right, h l  move sideways: focus metric, step cursor
g G              first, last
Enter            open: the run in Run, the metric in Zoom
Esc              back one screen; in the filter, clear it; on Runs,
                 clear a kept filter; close Help
q, Ctrl+C        quit, except in the filter, where q is typed
?                Help
/                filter the rows of Runs, the metrics of Run
Space            mark or unmark the run: the cursor's on Runs, the
                 focus elsewhere
[ ]              focus the previous, next run of Runs's order
o O              order Runs by the next column, reverse the order
s                smoothing: off, then three time constants
x                x axis: step, step metric, time
L                log y
+ - 0            narrow, widen, reset the x window, at the latest step
< >              pan the x window
i                Details of the run
y                copy the run's id to the clipboard

y sends the terminal its sequence to set the clipboard (OSC 52). A terminal may ignore it, which the dashboard cannot see.

The dashboard leaves the mouse to the terminal, so the terminal selects text as it does for any program.

Running