Explicit mode, level 2
Explicit mode gotchas
In the current state of this recommendation, BiDi control characters should be ignored in explicit mode. After all, the data is supposed to be the outcome of the BiDi algorithm which never contains them.
When selecting, copy-pasting, or searching for text, the terminal emulator might attempt to run some inverse BiDi algorithm and inverse shaping algorithm on its contents, and then perform the user action on its result for better user experience. Otherwise, text fragments of foreign direction will be reversed, searching won’t work in Arabic and copy-pasting will result in undesired shaped characters.
In a future revision to this document, BiDi control characters might be (mis)used somehow, or some other means invented to help terminal emulators provide a better copy-paste or search functionality, or for improved shaping and “unshaping” of Arabic letters. See level 2 ideas below.
Note that copy-pasting done by terminal emulators is already somewhat broken in text editors: chopped parts of the text aren’t copied. Similarly, the emulator’s search functionality is also limited to the onscreen bits. The cleanest solution is to encourage all the responsibility being at one place, and that one place is the application in case of explicit mode. Just as such applications provide their own search functionality, probably they should handle mouse selection and copy-paste themselves too.
When mirrorable characters appear within a foreign direction text snippet (e.g. the paragraph direction is LTR, and it has a piece of embedded RTL text with parentheses), these parentheses should be mirrored but the terminal emulator won’t know this (on in case of RTL paragraph direction with embedded LTR: the emulator will mirror them once but they should be mirrored back). The obvious approach is to require the application emitting them to replace them by their counterpart. (It’s not ideal for copy-pasting, but copy-pasting is “broken” anyway in explicit mode in a couple of ways, and if a terminal emulator attempts to fix it by running inverse BiDi and inverse shaping then it might as well attempt to fix the parentheses.)
A remaining minor problem with this approach is that not all mirrorable glyphs have a counterpart, and those that don’t can’t be mirrored with this approach. Maybe we can live with this limitation. Or explicit BiDi controls could be (mis)used to denote that they need to be mirrored (exact details to be figured out). Yet another approach could be to introduce a new attribute to mirror mirrorable glyphs (see SAPV in ECMA 48), or even all the glyphs, pretty much unrelated to the whole BiDi story. (Obviously if a glyph is mirrored for two reasons: because of BiDi and because of this attribute too, they’d cancel out each other.) [We need to make a decision here: which approach to take? It’s pretty low prio future improvement.]
Explicit mode level 2
Whenever we have notable applications that do BiDi over the emulator’s explicit mode, and the need arises to fix the remaining issues (better copy-paste and search functionality inside the terminal emulator, nicer shaping), we might consider designing explicit mode level 2.
Here’s an idea for future consideration to fix copy-pasting and searching in the terminal:
In order for these to work as expected, the application would not only have to transmit the visual order as the primary information, but also the logical one as secondary, additional info. For backwards compatibility, emulators not supporting these additional info must still display the text correctly.
For each character we’d remember a “logical order” integer, defaulting to 0, capped at two times the maximum width supported by the emulator. A new escape sequence would set a certain value for all subsequently printed characters’ logical order. In each paragraph in explicit mode, the logical order would be reconstructible from the visual order using the following simple algorithm. If the explicit paragraph spans across multiple lines, characters in earlier lines always precede in the logical order the ones in subsequent lines. Within the same line, a character with a lower value precedes a character with a higher value. For characters in the same line having the same even value, their logical order is their visual order, according to the paragraph direction. For characters in the same line having the same odd value, their logical order is the reverse of their visual order, according to the paragraph direction. (Or, as with the levels of the BiDi algorithm, should even numbers mean LTR and odd numbers RTL?)
The directionality of each character would also be specified by the value’s parity. For RTL characters the codepoint would be mirrored back when copied to the clipboard. The directionality could also be used to tell whether the mouse event (click or release) occurred over the start or end half of the glyph, and based on this whether the given character should be included in or excluded from the selection.
Note that I’ve ruled out the possibility of introducing the SIMD, SDS and SRS sequences of ECMA TR/53 and ECMA 48, and reconstructing the logical order based on them. See the section “ECMA TR/53 Review” for rationale.
As for Arabic shaping, we should evaluate whether ECMA 48’s SAPV, values 5–8, 13–14 and 21–22 are inferior or superior to emitting the presentation form characters, whether apps could reasonably easily use them, and thus whether we should encourage implementing support for these escape sequences. (I have the guts feeling that they are inferior to presentation form characters, and that there’s no library support that could make it easy for apps to emit the correct ones.)