Arbitrary keymaps vs. terminals

Hi,

I have spent literally my whole day on this, with no result, so it really bugs me now :slight_smile:

I just started learning kakoune and while i was reading :doc keys i have run into <c-d> and <c-u> for buffer scrolling. As i started learning with disabled arrow keys to make myself use hjkl i have though to myself it is quite lame to use different keys for moving if you could use <c-j> and <c-k>. You do not have to leave home-row, and also do not need to switch LControl/RControl while scrolling.

So I have added the following to my kakrc:
map global normal <c-j> <c-d>
map global normal <c-k> <c-u>
map global insert <c-j> '<esc><c-d>i'
map global insert <c-k> '<esc><c-u>i'

And I am doomed since then, as it does not work.

I have asked it on #kakoune and @SolitudeSF replied: <c-j> is enter in terminal.
Then I have found http://www.leonerd.org.uk/hacks/fixterms/ and @mawww answered it is already supported by kakoune.

After a day of googling and reading around I know a lot more about commands like showkey or ASCII keymaps. I even found that leonerd’s libtermkey is somewhat competing with the solution used by xterm for the very same issue: XTerm – “Other” Modified Keys

Now i think it is not related to my basic issue, as these try to solve the multi-modifier-key problem as i understand. (And also some other cases when there is no difference in keycode for different keystrokes like TAB and <c-i>). It is a shame this problem have not solved in unix world in ages, but does not fix mine.

In my case <c-j> do differ from Enter, but my terminal send the same <ret> action to kakoune. What can i do about that? I have not found anything on how to change that.

Out of curiosity: I have tried this with a couple of terminals and all do the same. Is there anybody using this feature or it is some legacy that can not be changed or what?

You have stumbled on a weird legacy quirk added to the original versions of Unix.

In the beginning was the typewriter, which had a knob to feed paper through the device, and a big metal lever to return the “carriage” back to the left-hand edge of the paper, so the user could resume typing. As technology advanced, somebody figured out how to put the keyboard and the print head at opposite ends of a telephone connection, creating the “tele-typewriter”, or “teletype” or just “TTY” for short.

To keep things mechanically simple, because there were two separate mechanisms involved (move the carriage horizontally, feed paper vertically) the TTY protocol had two separate control codes to trigger them: “carriage return” and “line feed” respectively, usually abbreviated to CR and LF. Early-model TTYs, like the ASR-33 had separate CR and LF keys.

Once computers came along, we needed some way to communicate with them, and since there were so many TTYs lying around, it was natural to plug one in and teach the computer how to use it. However, early computing pioneers were annoyed by having to use both CR and LF to signal end-of-line. What if you just get CR without LF? What if you get LF without CR? What if you get them the wrong way around? What if you get a CR and then nothing, how long do you wait?

Using a single control code for “end of line” would make things much simpler. It would make your computer incompatible with all the existing TTYs, but luckily computers are smart, and you can teach them to automatically translate the computer’s internal scheme when talking with an TTY. Unix’s creators decided that on their system, a bare LF would represent the end of a line (which is why LF on Unix systems is often called “newline” or “NL”), and taught their operating system to do the conversion for every TTY. Thus, even on my modern Linux system:

$ stty -a
speed 38400 baud; rows 62; columns 239; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z;
rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl
ixon -ixoff -iuclc -ixany -imaxbel iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0
vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop
-echoprt echoctl echoke -flusho -extproc

stty -a prints all the TTY settings for a terminal window I happened to have open. In particular, “speed 38400 baud” means it think it’s limited to ~3.8KB/s, “icrnl” at the end of the sixth line means that incoming CR characters are automatically translated to LF, and “onlcr” in the middle of the eighth line means that outgoing LF characters are automatically translated to CRLF.

You might be thinking “yes, that’s it, that’s why <c-j> is always received as <ret>!” but unfortunately things are a bit more complex than that.

“intr = ^C” means that when the terminal sends a Ctrl-C, the kernel eats it and sends SIGINT to the foreground process. “quit = ^” means that the kernel converts Ctrl-Backslash into SIGQUIT. “susp = ^Z” means that the kernel converts Ctrl-Z into SIGTSTP, and so forth. All these keys are very useful and important for command-line tools like grep and cat, but they get in the way for full-screen, interactive tools like Kakoune. Therefore, the kernel provides “raw” mode, where all these conversions are disabled and every key the terminal sends is received directly by the application.

In raw mode, <c-j> sends LF, and Enter send CR, so they can definitely be distinguished. However typing on the keyboard is not the only way to send text through the terminal: if you paste text with Ctrl-Shift-V or Shift-Insert, and that text includes a line-break, the line-break will still be a LF regardless of whether the terminal happens to be in raw mode or not. Therefore, Kakoune includes code that treats both LF and CR as <ret>:

4 Likes

Thanks for the thorough explanation.

If I have understood it correctly, Kakoune switches the mode to raw mode. From the code you mention I can see that some keys retain it’s functionality. For example, <c-z> is still suspend, which is indeed pretty useful. But why are other keys translated? What is the benefit of having <c-i> mapped to <tab>? Isn’t it better to have two different shortcuts that I can map to different things? Same applies for <c-h>, <c-m>, <c-j> and escape characters.

EDIT: Does Kakoune support kitty in full mode? Theoretically, we could solve the problem for those using kitty, right?

<c-z> in Kakoune still sends it to the background, like in cooked (non-raw) mode, because Kakoune very carefully recreates that behaviour manually, it’s not using the behaviour built into the terminal.

Unfortunately, the Tab key sends Ctrl-I, both in raw and cooked mode. There’s no way to tell them apart, so Kakoune has to guess which one the user pressed. For simplicity, it always guesses <tab>. LeoNerd’s “fixterms” system and xterm’s “modifyOtherKeys” system might provide otherwise, but neither one is widely available yet.

We could solve the problem for people using Kitty, and also for people using genuine xterm, and for people using mintty, but each of those terminals does things differently, and none of them are enormously popular. Last year when a bunch of terminal-emulator developers set up the Terminal Working Group to try and discuss cross-emulator compatibility issues, solving this problem was literally issue one, but it seems discussion’s died off and not much progress has been made.

Thank You, @Screwtapello!

For a moment I have though raw mode will solve my issue, but copy-pasting is a much higher priority, than handling <c-j>. I will use the defaults.

I still do not see why keeping this TTY thing is good for us. How come, that nobody wants to replace it with something else?

My hope was https://github.com/withoutboats/notty but it is dead officially.

1 Like

Yep, I figured out looking at the code that Kakoune had no way of distinguishing with the current state of the art.

Adapting kakoune to use kitty seems like a nice idea to try this summer.

iTerm2 supports LeoNerd’s “fixterms”, but I didn’t turn it on because last time I checked it with Kakoune not all keys were recognized.

Basically it’s because pressing ctrl clears 7th bit of binary representation:

Pressing Control+i (lowercase)

105	0x69	11 01001	i

would mean sending “)”, which is not very useful.

41	0x29	01 01001	)

So most terminals interpret this as Control+I (uppercase), which sends HT

9	0x09	00 01001	HT

https://bestasciitable.com

I am keen to improve CSI u (aka LeoNerd’s fixterms) support in Kakoune, do you remember which keys were not working ? It seems <c-j> is not sent using CSI u by iterm2, it can be worked around by setting it up manually (adding a binding in iterm2 that sends the correct escape sequence), do you know of other keys that are failing ?

1 Like

Is it because you find it’s the best alternative of the proposed new protocols?
I’m curious to know what the future of terminal emulation might be.

It seems to be the proposal that has most traction, it is really easy to implement when you have a CSI parser in your code (which is already necessary to handle some other kind of keys, it takes 2 lines in Kakoune codebase to support it), it can be extended to support more features (key down/key up for example)…

Confirm, <c-j> sent as enter.

Surprisingly, I didn’t spot any problem so far with CSI u on in iTerm2. I can’t remember what was a culprit key, maybe it was <c-j>. However I was able to map <c-i> both in normal and insert mode and it didn’t clash with Tab.

I’ll keep CSI u mode on, and report here if bad keys are found.

I’ve set following for <c-j> and it works. I mean I’m able to map <c-j> in Kakoune now! It’s really cool!

Screenshot update: F1-F4 didn’t work with CSI u, so I had to set them manually:

1 Like

I’m so jealous! :heart_eyes:

No need! You can use alacritty and customize keys you need to send CSI u sequence.

1 Like

In Vim I am able to map <c-j> and <ret> to different values. Why isn’t that the case for Kakoune?

As I wrote above:

Possibly LF could only be processed as a line-break in insert mode, leaving it available for mappings in other modes, but that would be more complex to implement. How much more complex, I don’t know, but complex enough that nobody’s bothered to try yet.

Interesting. In Vim I never paste from insert mode, but if I did, then there’s a chance it might do a <c-j> instead of a new line?

That makes sense. Thanks for the explanation.