Command_fifo explanation?

1

Hello,

I’m having hard time understanding the command_fifo new feature, even after reading the :doc.

Can someone help me understand?

And. more broader, where can we find explanation on new feature?

2

The documentation is the place to look, but it could probably be fleshed out with more details about the things that are confusing people. I’ll see if I can explain it, and if any of this helps, point it out and maybe we can add it to the official docs to help other people.

Kakoune’s traditional shell integration looks like this:

nop %sh{
    echo "Run by Kakoune $kak_version" > /tmp/log.txt
}

Kakoune scans the text inside {} for kak_* tokens it recognises, adds them as environment variables, then feeds the script into the shell.

Sometimes the shell-script wants to feed information back to Kakoune. In this case the idiom is:

evaluate-commands %sh{
    case "$kak_selection" in
        *duck*)
            echo info "You have selected a duck"
            ;;
    esac
}

As before, Kakoune collects tokens, adds them to the environment, launches the shell, and collects standard output. However in this case the output is given to evaluate-commands which makes Kakoune execute it as a command.

So now we can move information from Kakoune to the shell (as environment variables) and move information from the shell to Kakoune (as stdout). This is pretty powerful, and there’s a lot you can do with this system (see all the Kakoune plugins written up until now!), but there are also a number of limitations:

  • The total size of environment variables is limited, and the limit is platform specific and difficult to measure, so reliably moving information from Kakoune to a shell-script is difficult
  • Kakoune only provides values explicitly named in the shell-script’s source, so if you want to choose a value at runtime (for example, building $kak_opt_myplugin_python from $kak_opt_myplugin_ and %opt{filetype}), you’re out of luck
  • Kakoune doesn’t process stdout until the shell-script has exited, so if you want to try something and then do different things depending on whether the test worked, you’re out of luck.

The environment variable thing is a fundamental limitation, but it’s sometimes possible to work around the others using a series of shell blocks, interspersing processing in shell with processing in Kakoune. However, that means all the shell script state has to be passed back to Kakoune, and then into the next shell script Given that Kakoune only has 26 variables (registers), that’s a pain, and if your shell script wants to do a Kakoune operation inside a for loop… there’s no way.

It would be nice to have a way for a shell block to talk to Kakoune, back and forth, in parallel. Unfortunately, this raises the ugly spectre of synchronisation - when two independently-running processes need to communicate, the sender must wait for the receiver to be ready, the receiver must wait for the sender to be ready, the receiver must wait for the sender to finish, and the sender must wait for the receiver to be ready before it sends again - which means you already need quite a lot of communication set up before you can communicate! In languages like C and C++ you have things like mutexes and atomic values and shared memory, and none of those things are available to shell-scripts.

However, it turns out that Unix kernels provide a synchronisation scheme that works perfectly with shell-scripts: the FIFO, or “named pipe”. It works like this:

  • You can create a FIFO using the mkfifo shell command (or the equivalent mkfifo(3) API)
  • If a writer shows up first, the kernel will pause it until a reader shows up
  • If a reader shows up first, the kernel will pause it until a writer shows up
  • once the reader and writer are both available, data flows until the writer is done, and the reader gets a regular “end of file” response

In Kakoune terms, it works like this:

  • Kakoune collects terms, sets up the environment, and launches the shell as normal; if the script mentions kak_command_fifo or kak_response_fifo it creates new temporary FIFOs just for this shell block
  • At any point, the shell script can write a command to $kak_command_fifo
    • The kernel pauses the script until Kakoune is ready to read
    • Eventually, Kakoune reads the command and executes it
  • If the script wants a response from Kakoune, or it wants to be sure the command it sent has completed before it sends another, it may ask Kakoune to write to $kak_response_fifo with :write or :echo -to-file
    • When the script attempts to read the result from $kak_response_fifo, the kernel pauses the script until Kakoune is ready to write
    • Once Kakoune has written a response, the script can read it and continue

This solution addresses our original problems:

  • instead of receiving bulky data through environment variables, a shell-script can request that data via the FIFOs

    echo "exec %; echo -to-file $kak_response_fifo %val{selection}" > "$kak_command_fifo"
cat "$kak_response_fifo" | awk '... blah blah...'

  • instead of naming every possible value up-front, a shell-script can construct a name dynamically and request it via the FIFOs

    echo "echo -to-file $kak_response_fifo %opt{myplugin_$kak_opt_filetype}" > "$kak_command_fifo"
per_filetype_value=$(cat "$kak_response_fifo")

  • instead of having multiple shell blocks to implement a multi-stage process, you can just write to $kak_command_fifo multiple times

    for word in 5.. 4.. 3.. 2.. 1.. BOOM; do
    echo info "$word" > "$kak_command_fifo"
    sleep 1
done

Does that make sense?

(Note: I have not tested any of the example code in this post, I will be very surprised if it works!)

EDIT: Turns out the kernel pauses a writer waiting for a reader, not just readers waiting for writers.

9 Likes
3

I came up with this (tested) example while working on kak-bundle:

nop -- %sh{
kkq() {  # quoter
  while [ $# -gt 0 ]; do
    printf " '"; printf '%s' "$1" | sed -e "s/'/''/g"; printf "'";
    shift
  done
} #quoter

kkff() { cat >"$kak_command_fifo"; }

echo 'edit -scratch *bundle-status*; exec %(%d)' | kkff
for i in $(seq 1 5); do
kkff <<EOF
reg dquote $(kkq "$(date)"); exec 'gg' '<a-x>"_d' '<a-O>h' 'P'
reg dquote $(kkq " '$i' ... "); exec 'geP'
EOF
sleep 1
done
}

I’ve tried to implement a status buffer for kak-bundle using command_fifo, but ran into some unreliability problems ([BUG][CRASH] command_fifo fast re-opening crashes kakoune · Issue #4410 · mawww/kakoune · GitHub , now fixed, but this feature needs more testing, especially when combining command_fifo with response_fifo). Also, the %sh{} still blocks user action, even if it permits updating the UI and receiving replies from kak (meaning, you can’t switch to another buffer manually, continue editing another source etc).

The main thing to be careful about is to always close the command_fifo as soon as (and not more frequently than) you want kak to process your commands.

1 Like
4

Thanks yes it did help a little bit, even if it’s a bit into into the mechanisme and I didn’t get all of it. The example where useful.

To see if I understand well, could you say that is pretty close to kcr?
If so, what would be the differences with @alexherbo2 's kcr?

5

I don’t know much about kcr myself, but as I understand it they’re both ways for shell-scripts to communicate bi-directionally with Kakoune. However:

command_fifo kcr
scoped to a single shell block scoped to a Kakoune session or a client
provides response_fifo to synchronise responses need to make your own FIFO for synchronised responses
only provides command execution provides many services
only works in a shell block can launch an interactive terminal connected to Kakoune
built-in to Kakoune a third-party extension

There’s probably a few use-cases where either command_fifo or kcr could work, but I suspect most of the time one or the other would have the clear advantage.

2 Likes
6

Here’s another thing a commnd fifo can do efficiently (without a double shell call): time other commands

decl -hidden str-list stkmach_time_cmd_error
# a trampoline, Saves `sh` errors.


decl str stkmach_elapsed_time
def stkmach-time-cmd -params .. %{
  set global stkmach_time_cmd_error
  eval %sh{
    save_t() {
      t=${EPOCHREALTIME:-$(date +%s.%N)}
      ms=${t#*.}; s=${t%%.*}
      s=$(( s - 1622494800 ))  # 2021-01-01
      ms=${ms%${ms#???}}
      t=$s$ms
    }
    save_t; t0=$t
    printf >"$kak_command_fifo" '%s; %s "\n"\n' \
      'try %{%arg{@}} catch %{set global stkmach_time_cmd_error fail %val{error}}' \
      "echo -to-file %\"$kak_response_fifo\" '
'"
    IFS= read dummy <"$kak_response_fifo"
    save_t
    printf 'set global stkmach_elapsed_time %s\n' $(( t - t0 ))
  }
  %opt{stkmach_time_cmd_error}
} -override
def stkmach-time-cmd-dbg -params .. %{
  stkmach-time-cmd %arg{@}
  echo -debug "stkmach: elapsed time: %opt{stkmach_elapsed_time}"
} -override