The basic modes
6 modes in the model
In the data layer (model), the following three new essential BiDi properties are tracked for each paragraph:
- Implicit or explicit
- Direction autodetection enabled or disabled
- LTR or RTL
Other properties might also be introduced, e.g. see later the ncurses compatibility mode for mirrored box drawing characters.
The core idea behind implicit and explicit modes is the same as in ECMA TR/53 (discussed later), however, many details are different. My explicit mode (level 1) is a small subset of TR/53’s explicit mode. My implicit mode level 1 corresponds to TR/53’s implicit mode. My implicit mode level 2 replaces the features that I didn’t like and didn’t adopt from TR/53’s explicit mode.
Unless otherwise specified, in this document implicit and explicit modes refer to my definition.
The combination of the BiDi properties results in the following 6 possible main modes for each paragraph, as it’s stored in the model:
- Implicit, LTR
- Implicit, RTL
- Implicit, autodetected direction with LTR fallback
- Implicit, autodetected direction with RTL fallback
- Explicit, LTR
- Explicit, RTL
The autodetection flag is ignored in explicit mode.
Based on these BiDi properties and possibly the paragraph’s contents too, each paragraph is then transformed to its view in one of 4 possible ways.
4 ways of transformation
For each paragraph, there are 4 possible ways the cells of its lines could be arranged.
One of the two flags is the implicit vs. explicit property, as stored in the model.
The other flag is the direction. When in explicit mode OR autodetection disabled, the model’s corresponding flag is used directly. When in implicit mode AND autodetection enabled, it’s autodetected (details soon) based on the paragraph’s contents, still possibly falling back to the model’s corresponding flag.
This results in the following four possible ways of laying out the cells:
- Implicit LTR
- Implicit RTL
- Explicit LTR
- Explicit RTL
Implicit mode (level 1)
In implicit mode, doing BiDi and shaping are the sole responsibilities of the terminal emulator, to do it for applications that don’t do it themselves. The transferred data stream contains the text in its logical order.
This mode is designed for BiDi-unaware apps, typically simple utilities that just print their data to the standard output/error without sophisticated screen handling.
The most basic expectation from those who ask for BiDi support is for
simple commands such as a cat file to show its Arabic, Hebrew etc.
contents in a readable way. This is what PuTTY, Konsole, mlterm,
Terminal.app and probably a few other emulators address to some extent,
and this is what both ECMA’s and my implicit mode addresses too.
For these kinds of utilities there’s no reasonable way to inject special instructions to enter (and later leave) implicit mode. Hence, contrary to ECMA TR/53 and ECMA 48, this implicit mode must be the default.
On level 1, BiDi control characters are discarded. Strings that rely on them won’t show up properly. In a future revision, implicit mode level 2 might be designed that remembers them to some extent (somewhat similarly to combining accents) and takes into account when running the BiDi algorithm.
Autodetecting the direction
In some cases having a strict predefined paragraph direction provides
the better user experience. For example, when printing lists (e.g. the
output of an ls command), it’s generally preferred if foreign
direction items are still aligned as normal direction ones. In some
other cases it’s better to have it autodetected, typically when printing
larger amount of foreign direction text. (PuTTY does autodetection,
Konsole and Terminal.app don’t. Mlterm offers a command line switch.)
Let our users and utilities choose.
For implicit paragraphs that have autodetection enabled, their base direction is autodetected (guessed).
UAX #9 doesn’t specify an exact algorithm for autodetecting the paragraph direction. An implementation might decide on the first strong character (as it’s done inside an FSI…PDI block), this provides a reasonable experience if the user is typing some text and doesn’t want the overall direction to jump back and forth. Most terminal emulators will probably just go with whatever is offered by the BiDi library of their choice, and that’s fine. (It’s not necessarily a problem if the exact autodetection method slightly varies across terminal emulators).
UAX #9 bullet point HL1 says:
“As another example, when the paragraph contains no strong characters, its direction could be determined by the levels of the paragraphs before and after.”
In terminal emulators the paragraphs are often unrelated to each other (e.g. output from different utilities), hence I discourage looking at the surrounding paragraphs, I encourage handling each paragraph independently. (Should there be a demand for looking at surrounding paragraphs, a new escape sequence could be introduced in the future to control this behavior, or to set up hard break points, e.g. to look at the entire output of each command.)
If the autodetecting algorithm fails to make a guess (e.g. the paragraph
is empty, or consists of neutral characters (e.g. whitespaces) only), we
fall back to the paragraph’s direction as specified in the model. (Even
for empty paragraphs a decision has to be made, so that empty input
lines can show the cursor at the expected edge, and mouse clicks can
properly be transformed. Therefore, as opposed to HTML having a single
dir=auto mode only, we have separate autodetection modes for both
fallback directions.)
Finally the paragraph is handled according to implicit LTR or implicit RTL mode, whichever direction was guessed or fallen back to.
Implicit LTR mode
The terminal emulator asks the BiDi algorithm to lay out the entire paragraph, using LTR paragraph direction. This results in a mapping from logical to visual positions. The cells are laid out according to this mapping, aligned to the left edge of the emulator.
Glyphs are mirrored (or rather: counterpart glyphs are used) according to the BiDi algorithm.
Arabic shaping is performed on the paragraph, before splitting to lines.
The I-beam cursor is shown at the side corresponding to the resolved directionality of the character underneath.
Implicit RTL mode
Like implicit LTR mode, except that RTL paragraph direction is used, and the result is aligned to the right edge of the terminal.
Explicit mode (level 1)
In explicit mode, doing BiDi is the sole responsibility of the application, the terminal emulator doesn’t do any (it may still do overall RTL direction, though). The transferred data stream contains the text in its visual order (either pure LTR or pure RTL).
The emulator doesn’t perform shaping either, this is also the sole responsibility of the application, using “presentation form” characters. (Level 2 might add some improved shaping control in a future revision of this document.)
This mode is primarily for sophisticated BiDi-aware apps.
However, this mode is also useful for damage control in apps that don’t do any BiDi at all, but have a complex UI that could fall apart in implicit mode when encountering foreign direction text. By switching to explicit mode these apps still won’t do BiDi (in fact, foreign direction text snippets will be reversed, i.e. much harder to read), but at least the overall UI will remain intact.
The autodetection flag is ignored, as it doesn’t make sense with visual order.
Explicit LTR mode
This is exactly what most terminal emulators (the ones that don’t do any RTL or BiDi at all) do now. Everything goes from left to right.
Explicit RTL mode
Cells are laid out in the exact reverse order than in explicit LTR. The model’s first column is displayed in the rightmost cell, etc.
Mirrorable glyphs are mirrored.
The I-Beam shaped cursor is displayed on the right side of its cell.