Suggestion: Lazy loading with a PluginEvent hook


#1

Kakoune’s extension system is designed to be modular rather than hyper-efficient, and while this allows plugins to be written without ceremony (“write a .kak file, stick it in this directory”), it does mean that Kakoune’s startup time suffers as it loads every detail of every plugin.

There have been a number of solutions proposed for this problem:

  • A standard filesystem structure for plugins so that Kakoune (or a third-party plugin manager) can be smarter about which code is loaded and when
    • Very similar to what Vim does, so it’s a proven approach
    • Requires documentation and enforcement of the structure
    • Makes it writing a plugin more difficult
  • A central configuration of which filetypes require which scripts loaded so that dependencies can automatically be loaded
    • Simpler than imposing a structure on every plugin
    • Only really addresses filetypes; there may be other things that benefit from lazy loading
    • It’s not clear where such a configuration would exist, or how it would be maintained
  • A module/dependency system where scripts explicitly state what other scripts they depend on
    • Scripts without dependencies are still simple
    • Works for all kinds of dependencies, not just filetypes
    • Would require changes to many existing scripts
    • Would require a bunch extra logic in Kakoune itself
  • Scripts can just avoid defining highlighters until a file of the relevant type is loaded
    • Existing scripts work unmodified
    • No extra logic required in Kakoune
    • Doesn’t work for scripts with dependencies, unless the script quickly sets the filetype option to different values to trigger their hooks, which is a bit of an ugly hack

I had another idea on IRC today, so I figured I should write it down to see what people think. This idea has two parts: the PluginEvent hook, and a convention for lazy loading.

The PluginEvent hook

Most of Kakoune’s hooks indicate that something has changed in Kakoune’s internals, and plugins can’t add or modify Kakoune’s internals, so they don’t need to add new hooks. On the other hand, sometimes plugins want to interact with each other, and for that purpose custom hooks would be great.

A non-lazy-loading related example: kakoune-cargo runs the cargo build-process in the toolsclient, because that seemed to be the Kakoune way, but the other day somebody contacted me and asked me if there was a way to automatically jump to the first error when compilation finished. When cargo runs in the toolsclient, I don’t want that, because I might be doing something else while the build is in progress and I don’t want to be interrupted, but I can see that in other circumstances that would be a useful behaviour. Unfortunately, it’s not easy to make configurable, because “what happens when compilation finishes” is a BufCloseFifo hook inside a printf command inside a %sh{} block, and not really the place for checking a boolean option, or reading a command to eval from a string option, or anything like that.

If Kakoune provided a PluginEvent hook, my plugin could say something like:

hook -once buffer BufCloseFifo %{
    plugin-event "cargo-complete:%arg{@}"
}

…and the end-user could put this configuration into their kakrc:

hook global PluginEvent cargo-complete:(check|build|test|doc).* %{ cargo-next-error }

Lazy loading

To make lazy loading possible, I propose the following convention:

  • A plugin that wants to be lazily loaded should put all its expensive setup operations into one or more files with the .kakp extension (the P is for Plugin), so they won’t be automatically found and loaded

  • The plugin should include a regular .kak file that adds hooks to source the .kakp files:

    declare-option str foo_plugin_source_path %sh{ dirname $kak_source }
    
    hook -once global PluginEvent foo_load_highlighting %{
        source %opt{foo_plugin_source_path}/highlighting.kakp
    }
    
  • The plugin can trigger PluginEvent from any other hooks it likes:

    hook global WinSetOption filetype=foo %{
        plugin-event foo_load_highlighting
        add-highlighter window/foo ref foo
    }
    
  • Any other plugin that wants to use a lazily-loaded resource can trigger PluginEvent in the same way.

As a result, “the list of PluginEvent patterns hooked” becomes part of a plugin’s API, in the same way that “the list of options declared” and “the list of commands defined” is today.

Of course, a plugin could just do the expensive operations in the PluginEvent hook itself, but if they’re expensive enough that you need to make them lazy, you probably don’t want Kakoune to have to read through them at startup even if it doesn’t execute them.

Conclusion

Such a system would allow Kakoune to have lazy loading without any loading-specific functionality (just one extra hook and a command to trigger it, but those aren’t specific to lazy loading). Existing plugins keep working unmodified, simple plugins stay simple, and this system can be used for all kinds of dependencies, not just highlighters.

Of course, a dependency system is inherently complex, and a simple solution just means the complexity is pushed elsewhere. In this case, I believe the complexity goes to:

  • plugin authors have to document and support what PluginEvent hook patterns they respond to
    • but they already have to document and support options and commands, so this isn’t a big deal
  • plugin authors have to decide when to shift from eager to lazy loading, instead of getting it for free
    • but we’ve gotten this far without lazy loading at all, and Pareto says only a few plugins will benefit from it
  • another global namespace for plugins to trip over each other in
    • but we already have this problem for options and commands

#2

This is probably my single biggest concern at the moment (not with your proposal, but with Kakoune and plugins). I think namespaces and modules are going to be a must have, and can include lazy loading as a part of that idea.

Basically, I think your idea is a good one – but I think solving this requires thinking at a slightly higher level and in doing so we will solve multiple other problems. Modules give namespaces, lazy loading solutions, doubly loading solutions, possible bundling in .zip and more I can’t think of at the moment.

I can’t decide if I think a step forward like this cements us deeper into our current design, or just makes our day to day experience better without pushing modules further down the road.


#3

I think what you’re talking about is the quality of scalability, where a technology can be used on different-sized problems. Most technologies aren’t very scalable, a few can scale up and even fewer can scale down. For example, storing data in CSV files is very easy to do, but as you have more data and need to access it more often you spend more time searching for data than using it. SQLite scales up much better than CSV files, PostgreSQL scales up even further, and whatever Google uses to store their search index scales ridiculously far.

But a Google-scale index doesn’t scale down very well, what with needing a world-wide network of expensive data-centres. Even PostgreSQL usually needs user accounts and dedicated data storage and maybe some network ports for communications, making it too complex for some use-cases.

My point is, each of those data-storage solutions has a particular range of use-cases they’re designed for, and if my use-case is too big or too small for one of them, there’s some other solution I can turn to. It’s OK that not every tool can meet every use-case.

Kakoune’s current single-flat-namespace system definitely limits its scalability. Super-extensible editors like IDEA and Eclipse surely benefit from Java’s namespacing system, so different plugins don’t stomp on each other, and Kakoune will never get a plugin ecosystem as powerful as theirs without such a thing. On the other hand, Vim and (as I understand it) Emacs also have flat namespaces, and their plugin ecosystems are much larger than Kakoune’s, so Kakoune probably has some more room to grow.

My impression so far has been that Kakoune is quite aggressively minimalist — even if statements require shelling out! — so I think a full-fledged module system would be “not Kakoune-like” even it were very powerful and flexible. But really, it’s up to mawww to decide what scale Kakoune is aimed at.


#4

I think this is a pretty nice idea, the only thing I dont like is the name “PluginEvent”, I’d prefer a more generic one, such as “UserEvent”. There is nothing plugin specific about this functionality, its just a hook that can be triggered by a command such as raise-event '...'.

From there, plugins could by convention respond to those events. Either to implement lazy loading as suggested, or just to provide event-based functionality, such as the cargo-complete example.

I am not sure this is the best solution we got for lazy loading though, but I think the mechanism is general enough not to care for now, UserEvents are sufficiently useful for non-lazy-loading use cases to justify themselves.


#5

Yep, perfect is the enemy of good (and done).


#6

Vim has a series of namespaces for different variables, which are then used as a way to protect scripts – adding noise to every single plugin. Emacs uses lisp magic to avoid echo’ing the leading myspecialplugin_ stuff via a bunch of namespace lisp systems.

Vim has:

  • buffer-variable b: Local to the current buffer.
  • window-variable w: Local to the current window.
  • tabpage-variable t: Local to the current tab page.
  • global-variable g: Global.
  • local-variable l: Local to a function.
  • script-variable s: Local to a :source’ed Vim script.
  • function-argument a: Function argument (only inside a function).
  • vim-variable v: Global, predefined by Vim.

Both have had namespace issues due to these (now multiple decades old) decisions. But, I think this was a case of me using one issue to try to solve another: https://github.com/mawww/kakoune/issues/1991 by expanding its scope. Sorry about that.


#7

Absolutely, I described a similar idea here https://github.com/mawww/kakoune/issues/2527#issuecomment-433536917

Talking about vim, it offers a User autocommand which fits the same purpose: https://neovim.io/doc/user/autocmd.html#User


#8

How??


#9

Vim also has a plugin namespaces. When writing a plugin each global function and variable must be prefixed with plugin name. Kakoune most of the time follows this rule as far as I know.


#10

Following a filesystem hierarchy convention requires understanding what each part of the hierarchy is for, and correctly dividing your plugin into the appropriate categories.

It might not be much more difficult, especially after you’ve done it a couple of times already, but there’s definitely more steps beyond “just dump everything into a file”.


#11

when I’ve written my first plugin for Vim, which was a snippet manager, I’ve read some article that described a plugin structure in several simple examples and it was enough to understand the basics. After that I’ve just followed the docs of Vim itself, which were not about plugin structure but about writing in vimscript. So I guess its just about good documentation that describes how plugin is structured, by breaking it to simple pieces.

It’s natural for most of programming languages nowadays, to provide a module-like hierarchy.