Kakoune-repl-buffer, a new way to work with repls

Plugins
1

A while ago, somebody popped into the Kakoune IRC channel inquiring about an equivalent to Vim’s :!, which just runs a shell command in the foreground. Because of Kakoune’s client-server architecture, it can’t support that feature exactly, but I can see the appeal of running ls or cal without having to open another terminal.

I’ve also been thinking about Kakoune’s REPL features, :repl-new that launches a command in a new terminal, and :repl-send-text that sends the current selection to the active repl process. These tools can be very useful, but they’re somewhat limited - the tmux and kitty backends only work inside those respective tools, and the X11 backend depends on external tools (xdotool, xsel) and implicit assumptions (whether the terminal respects title-bar text, whether programs can change keyboard focus on their own, and whether <s-insert> means “paste”) so it can be quite brittle.

I wondered how difficult it would be to teach Kakoune to run a REPL inside Kakoune, with commands to send data to the REPL process, and the output being plumbed into a FIFO buffer so you could scroll through it and copy/paste the results into other buffers, etc. I learned a lot about POSIX FIFO and pipe semantics, and had to invent some debugging tools, but it turned out OK:

I made an Asciinema demonstrating the features I’ve implemented so far. I apologise for the length (nearly 15 minutes!), but I couldn’t think of a better way to explain what’s already there and how to use it:

https://asciinema.org/a/X6QMxb0m1PvtzBdJ204A3ELdc

I’d really like to write helpful documentation with examples, and polish the interface to make it useful and comfortable. However, I didn’t write it because I needed it, I just wanted to see if it was possible, so I don’t really know what kind of examples would be helpful, or what kind of interface would be comfortable.

Would anybody here find this plugin useful? What do you want to be able to do? How would you want it to work? What examples would you look for in the documentation?

Bonus round: scripting tricks

If you’re not interested in using this plugin, but you might be interested in writing your own, here’s some cool tricks I found along the way.

  • the repl-buffer-input helper script
    • It’s natural for Kakoune to send inputs by opening a FIFO, writing, and closing, but a traditional Unix pipeline shuts everything down when the upstream pipe is first closed.
    • This script watches a FIFO for multiple writers and combines them all into one output; to actually shut down the pipeline you must delete the FIFO from disk (and wait a few seconds for the script to notice).
    • It’d be easy enough to do this in pure C, and drop the Python dependency, but then you’d need to compile it on every system.
  • Writing a Kakoune command to send a newline charater is harder than it sounds.
13 Likes
2

This looks like a solid base for building some neat plugins/extensions, similar to connect.kak.
I’ve had a few attempts at making something similar myself but ran into a number of issues that made building a generic plugin feel like more effort than it was worth.

  1. REPLs using a variety of different delimiters for multi-line input & different meta commands meant all the glue had to be pretty extensible. Also might need to support things like password input reasonably for something like a database REPL CLI. And then there were things like SBCL that had interactive debugging by default.
  2. I wanted to be able to capture stderr and pipe it through something (I was using perl scripts as it has concise syntax for this sort of thing) that would let me have REPL errors show up in Kakoune as a message rather than as text in the output buffer but there wasn’t an easy way to determine where each error stopped and started.
  3. I was experimenting with using replace-range to put the REPL output in the same buffer as the commands but this added a whole lot of extra complexity. Writing in one buffer and having the output in another buffer wasn’t really what I wanted.
  4. You miss out on a lot of the nice features of many REPLs like tab completion. Made me realize that what I really wanted to be interfacing to wasn’t a REPL but something like the SWANK server for slime in Emacs. Good old repl-send-text + a normally accessible REPL in another panel felt better in a lot of ways since I still had the full REPL if I needed it.

Anyway I’ll probably end up having another crack at it one day using your plugin as the basis. When I was trying to implement something interactive in Kakoune it made me think we’re missing a number of potentially useful hooks like MainSelectionChange. You can of course approximate it with NormalKey hooks but it’s awkward. But there is also the issue of the regex hook filters not being well suited for something like “if selection includes line 8”.

What I use at the moment, if a REPL supports it, is I just run it with the output pointing at a FIFO and read by edit -fifo -scroll so I have easy access to the text if I need it. For example postgres CLIs --output flag.

1 Like
3

This comes down to how people actually use REPLs from Kakoune. I normally use REPLs interactively from a terminal, typing multiple lines, so I added :repl-buffer-prompt to make that possible… but if I’m just going to do that, it might be easier to run the REPL directly in a terminal, and what use is a plugin?

I can definitely sympathise with the use-case of writing a database query in one buffer and using :repl-buffer-send-text to repeatedly send it to a database query tool to test it, but is that enough?

I kind of expect people to build other tool-specific integrations on top of it. For example, rather than launching a database tool and having it prompt for a password interactively, I’d expect somebody to write a wrapper that prompted for the password with zenity --password or similar.

That sounds nice, but I think having error messages evaporate would be frustrating. I find myself going back to check error messages repeatedly (to double-check I’m looking in the right place, or that I’ve understood the problem correctly), so I like having error output stored in the buffer.

Apparently the Acme editor likes to put stdout into one buffer, and stderr into a different buffer, which would be possible but awkward from a UI perspective.

Yeah, that’s true. Running a REPL in an actual terminal is a much richer experience than just feeding bytes in and collecting bytes coming out. I guess the two models of “interacting directly with a tool” and “sending bytes, collect output” are very different, and trying to bridge them is counter-productive.

Rather than trying to make direct interaction more comfortable, perhaps it would be a better idea to optimise the “send selection” workflow - say, remembering the last block sent instead of having to re-select it each time, or detecting the specific output caused by sending a block, and pasting that back into the buffer where the block was sent from.

4

I actually needed this the moment you posted it. I’ve been using it since then for other things.

I use a repl to spawn (I only count things that I’ve done multiple times) gnuplot, shell or julia. Sometimes I’ve also spawned a connected shell that I use as a normal shell but from where I control kakoune. My usage varies depending on what I spawn.

Script Languages

When I’m writing a shell script I use my repl to test pieces of code. I suppose people writing python scripts do the same. In this cases I just do it to check that it works. That means that I already know exactly what I want to do, but I need to test it meanwhile.

Continous Testing Tools

When I use julia or gnuplot things are different. For example, if I’m solving a differential equation in julia I repeat: write code, execute multiple times changing parameters, adjust code, see the plot, etc. It’s different to scripting because my target is not creating a script (although I end up having a script that I save). Rather, it’s also studying an object, a differential equation for instance. Therefore, I use the repl as a modern calculator. I cannot simply run the script each time because julia would require a lot of time to compile everything.

I’m constantly sending code and seeing the results in the repl or in a plot spawned by the repl. But I need the repl because sometimes I need to write things there too. I mean I cannot just have the repl in the background. I just need to change things dynamically.

When I use gnuplot the experience si similar. In these cases I’m actually creating a script, but I’m not sure of what I want from the beginning. I need to test colors, positioning, etc.

Shell or connected shell

When I start a shell is just because I’m working on a project and I’m continuously using the terminal.
For example, I may select which files to open in my client in the terminal. I also execute things like git, mvn, br… But then I need to write a for loop. Or my project is failing somewhere and I need to execute the same long commands multiple times. It’s easier to do it in a scratch buffer and then send it.

What I usually don’t need is to get the output of the repl in a buffer, because I don’t need to modify it. Also, if I’m using the shell, I can simply pipe the output to :fifo, a connect command that brings the output to a buffer. Maybe I’d be interested in I could get it in both the terminal and a hidden buffer.

I would like to work in that direction too. What I do currently is to save my selection with Z if I will be sending the same lines continously. But I agree that a repl user mode with handy commands could be helpful for me. I don’t know which commands I need though.

I like that your plugin tries to run away from using things like clipboard integration. I also believe using fifos is the way to go, but I’d like to keep the repl in my terminal, and if I could have it in a hidden buffer too great.

Something like this:

  • Programs inside the terminal take input from fifoA and output to fifoB.
  • Terminal input is redirected to fifoA but external programs can also write to fifoA.
  • fifoB is redirected to terminal output but can also be queried by external programs.

It’s more important to think about the input than the output though.

Side note: it is also possible to use some terminal emulators to avoid using the clipboard. kitty’s @ send-text por example.

5

@Screwtapello
Great!
Two questions!

Is it possible to give this “REPL buffer”, an anchor(getting it to appear where we want and defining its size)?
Can we get a similar option for “docs buffer” too?

6

A Kakoune client can only display one buffer at a time; there’s not really a way to make a buffer appear in a particular place and size, only “this entire client”. That said, it’s possible to run a new Kakoune client in a new terminal with the :new command, and the positioning options depend on what kind of terminal you’re using.

:new calls :terminal to actually launch the terminal. If you’re running inside tmux, you can choose whether the new terminal splits the current pane horizontally or vertically, or launches a whole new full-screen terminal. If you’re running inside kitty, you can choose whether the new terminal is a top-level terminal or just a tab. Otherwise, Kakoune just runs the command in the termcmd option, and your window manager decides where it appears and what its size will be.

7

Previously, the only ways to send text to a REPL in a buffer were to use the repl-buffer-send-text command (usually to send text from a different buffer), or repl-buffer-prompt, which asked for input in Kakoune’s prompt line and sent that to the REPL… which was kind of like typing at the prompt, but not really. After last month’s dicussion about Plan 9 and the Acme editor, it occurred to me that I might be able to do something similar inside Kakoune - it’s a text editor, isn’t it? You can type into the buffer, can’t you? So why not send the text you type?

It turns out there were some minor bugs with the BufReadFifo hook in Kakoune, but those have been sorted out in the latest trunk versions, so if you build Kakoune from source you should be able to try out the latest version of kakoune-repl-buffer and type into the buffer:

asciicast

The way it works is:

  • as each chunk of output is received from the REPL, the plugin keeps track of where that chunk ended
  • when you hit <ret>, it checks to see if the cursor is after the end of the last chunk (so you can type annotations earlier in the buffer without causing problems)
  • if the cursor is after the end of the last chunk, all the text from the end of the chunk up to the cursor is sent to the REPL as input

So you can type as normal and hit <ret> to send a command; you can <esc> back to Normal mode and edit the command with the full power of Kakoune, you can even paste several lines, edit them, then send them.

The current limitations are:

  • Currently, after typing a command, a copy of the command is printed before it executes
    • Since the “normal” method of interaction involves sending text from another buffer, the plugin defaults to echoing sent text
    • I’m thinking of removing this behaviour entirely, since “type in buffer” doesn’t need it
  • Kakoune does not permit “undo” in FIFO buffers, until the FIFO is closed (in this case, when the REPL exits)
    • This can be annoying, if you paste the wrong thing at the prompt, you can’t undo the paste

I think this is going to be pretty cool, but I need to use it more to find the rough spots. If you’re interested in testing it out, I’d be interested to hear about any problems you encounter.

2 Likes