Rewriting/reorganising Kakoune's documentation

Kakoune has generally good documentation, and pretty much everything you need to know to use Kakoune interactively and write scripts is in there somewhere, but sometimes it can be hard to find the thing you want to know, or even to discover that it exists in the first place. For example, the commands page documents most commands, except for things like execute-keys, evaluate-commands, hook, and map, which have their own pages, except for exec and eval which share a page. None of those pages actually talk about how command arguments work, though - thatā€™s command-parsing. command-parsing mentions expansions, but doesnā€™t go into detail since the expansions page is all about them, although it mentions parsing in return.

Just today somebody made a PR adding a bunch of details to the expansions page, and when I pointed out that it duplicated some of the information in the command-parsing page, they said:

I didnā€™t realize the command-parsing page contained that (I donā€™t think Iā€™ve ever actually read it to be honest). The information [in this PR] is what I though was missing from expansions (and ended up learning from looking at the source code).

It seems to me that Kakouneā€™s documentation has grown organically, with things being added and amended over time as Kakoune has added and changed features, but without much attention to the larger structure.

So:

  • does anybody else agree that it can be difficult to find things in Kakouneā€™s documentation?
  • would anybody else like to see the documentation reorganised?
  • is anybody willing to contribute financially to reorganising the documentation?

If nobody has a problem with the state of Kakouneā€™s documentation, then fair enough. On the other hand, if thereā€™s consensus that Somebody Should Do Something, Iā€™ll start thinking about what a reorganisation might look like, when I can find the time. After all, Iā€™ve already contributed to Kakouneā€™s documentation, and Iā€™ve written long-form documentation before. If people are willing to contribute financially, Iā€™m willing to make the time.

9 Likes

When I first started Kakoune I just browsed through all doc pages, so now I know where to find stuff. I usually do that if I find a new tool.

Maybe some improvement is needed, but I would say that interconnecting the pages a bit more, like refer to the execute-keys , evaluate-commands , hook , and map pages from the commands page, would be a simple step to make it clearer.

2 Likes

I think the overall organization of the pages is satisfactory, except when it comes to command-parsing, execeval and expansions.

The list of values in ā€œexpansionsā€ should be moved to a ā€œvaluesā€ page, similar to options and registers. This page would simply explain the available values and their type
ā€œcommand parsingā€ should be renamed to ā€œstringsā€ since itā€™s really what itā€™s all about. It would document how all the string types work and what are their usecases.
Then ā€œexpansionsā€ should just explain what the various %ā€¦{} expand to and in which context, and link to options/registers/values.

Expansion and string could be merged into one page since the two topics are essentially interlinked

edit: just saw the part about financial contribution only now, Iā€™m willing to do it

2 Likes

Same here. But I would like to have ability to search over doc in a way how Vim does it. Vim has best doc system in my opinion.

1 Like

Iā€™ve definitely found myself wondering about the organization of the docs pages. Reorganizing all the docs would be a worthy effort, but maybe an intermediate solution - create an ā€œindexā€ page and have :docs with no arguments start there.

It would serve as the top level documentation page that lists all of the current documentation pages, and short blurbs about what they cover. Once those blurbs are written, I think itā€™d become pretty clear where the overlaps are, or what pages need to be merged.

I realized Iā€™m talking about creating a documentation page for documentation pages - more things to maintain - but it might be a lower effort means of making the existing docs more accessible to newer userā€™s who donā€™t know where to begin. Additionally, while the autocomplete listing of docs are nice, itā€™s not the most accessible for folks who donā€™t even know what things like ā€œexpansionsā€ means.

3 Likes

I like this plan. If you donā€™t have time to try it out in the near future, I might give it a shot.

Yeah, thatā€™d be great. Of course, one reason Vimā€™s documentation is great is because they use a custom syntax with different markup for different cross references - so :foo is always a command, 'foo' is always a runtime option, +foo is always a compile-time option, and *foo* is a generic cross-reference. All that stuff is picked up by Vimā€™s documentation-indexer.

Kakoune uses Asciidoc syntax, which isnā€™t quite as convenient. As best I can tell, the syntax for making something a cross-reference anchor is [[like-this]], or even worse, anchor:like-this[]. That means Kakouneā€™s internal AsciiDoc renderer would need to filter out that text, and that means that jumping to a particular anchor gets more complex because you have to load the buffer and do the jump before it gets reformatted.

It should be possible, but it feels like it would be a lot of work.

That would be rather nice. Itā€™d also be a good place to document features like ā€œyou can hit Enter on a cross reference to follow the linkā€.

3 Likes

Vim uses tags for navigating in documentation. It generates ctags compatible tags per doc file, and later these tags are used in documentation for jumping. What is really cool is that Kakoune uses asciidoc, and ctags can uderstand asciidoc. And Kakoune understands ctags (ahem), so I wonder if Kakoune could utilize ctags for better navigation in docs. Basically we have everything we need for that.

POSIX only requires ctags to support C and Fortran, and even Exuberant Ctags doesnā€™t support AsciiDoc. Universal Ctags apparently does, although itā€™s not listed in the docs.

Meanwhile, although Kakoune understands ctags, ā€œThis script requires the readtags command available in ctags source but not installed by defaultā€.

While Kakoune depends on these third-party tools for good reasons, I donā€™t think itā€™s a good idea to have Kakouneā€™s built-in documentation system require them. Generally, Kakouneā€™s basic functionality requires only standard POSIX tools.

The ctags format is simple enough that we could probably write our own indexer, and the resulting tags file should be small enough that industrial-strength readtags isnā€™t needed for good performance, but weā€™d still have the problems of updating Kakouneā€™s AsciiDoc renderer, and making sure that looking up a tag jumps to the right spot before formatting is applied.

Well, ctags support of asciidoc isnā€™t really needed in this case - tag files can be generated by kakoune itself, and for parsing we basically need a tool that can do regex search over file, such as awk or grep, which fits POSIX requirements.

True. I was astonished when Iā€™ve compared readtags with grep performance on 607MiB file (4 million lines) - it is in order of magnitude faster compared to grep. But this difference isnā€™t so great for small files, and I suppose documentation tags will be relatively small.

I also think that this requirement can be wrapped by using posix tools combination in case readtags isnā€™t installed but thatā€™s another topic.

  • does anybody else agree that it can be difficult to find things in Kakouneā€™s documentation?

Exceptionally hard, and documentation is the lifeblood of a long-lived editor.

  • would anybody else like to see the documentation reorganised?

Yes, not even that picky about how ā€“ I feel most reworks would be an improvement.

  • is anybody willing to contribute financially to reorganising the documentation?

If it ended with inline, searchable, referable documentation, absolutely.

My question is simply, what would it cost: just to an order of magnitude? You could start a bountysource today if @mawww basically pre-agreed to merge the result (with obviously some information on where it was going).

This is because readtags can do a binary search if the tags file is sorted, which is why Kakoune relies on readtags in the first place instead of grep its way through the tags.

More generally, we already support cross references in the documention (just hit enter on the links), we also have a rudimentary index system through the :doc command auto completion. I guess this could be expanded to support additional syntax, say :doc :command-name for prompt commands (with auto completion, as soon as you type :doc : the completion shows all commands in the documentation pages, same if you type :doc <key> to get a key documentation, and we just need to find a syntax for optionsā€¦).

I am pretty proud that doc.kak provides the current system in only 165 lines of kak script (rendering a subset of asciidoc included), we have a bit of room to improve it before it becomes horribly complex.

As to the general organization of the pages, this grew up in an organic fashion, so I am all for a good rationalization of all that.

All that to say, I believe we are not that far from a good help system, we need to reorganize and cleanup the pages, and add a few features to the doc command completion system so that we can easily search for a command, key or option.

Yeah, it should be easy enough to write a ctags-compatible indexer, and a custom lookup tool, but it does mean that Kakouneā€™s existing ctags support isnā€™t much help.

I originally mentioned financial support because Iā€™ve done documentation writing before and felt pretty confident I could bang something out.

Since that time, however, Iā€™ve been convinced by @occivinkā€™s description of how the docs could be reorganised without a full rewrite, and the general tone here seems to be that searchability is a more pressing need. Iā€™m a lot less confident about my ability to build a documentation searching system in Kakoune-script, so I donā€™t really want to take peopleā€™s money when I might not have anything to show for it at the end.

Why it is so? The only change needed for current implementation to work with documentation tags (provided by some indexer) is to replace readtags with grep or awk, in case readtags isnā€™t installed. The rest could be used as is.

Hi all,
I know this problem not as a programmer but a report writer. Over the past five years I have constructed 47 reports which equates to over 110,000 words using mainly tex/latex to pdf and MSWord/MacTeX/TexWorks as editors. Which are all no longer in the tool chain thanks to your efforts on the Kakoune modal editing, done right.

My problems are exactly the same as described in this thread. With multiple documents poor cross-referencing with duplication of attempts to find subject matter dependant on the day/month/year. I am currently believe that my solution lies with scribble not as a racket documentation tool but as a preprocessor for a PMBOK (<- make mum proud moment).

I am still in my first steps of scribble and will be continuing my education this semester with more reports giving greater opportunity to grasp scribble. Thought it was worth mentioning as a tool in this discussion thread and with your programming experience determining if it provides a solution for Kakoune documentation.

Whatever the out come is whatā€™s one more report, sign me up. Happy to help on the Kakoune documentation. I am a part-time programmer with just three yearsā€™ experience. Make that two years, damn you, Haskell.

This would be my first experience contributing to an open source project so training wheels are tightly secure, with git man page on hotkey.

Obligatory links below:

srcibble
scribble/bnf
BNF Grammer examples
scribble/xref

The reference below is an overview of the tool chain: mouse event on the heading ā€˜6.1.2 Layer Roadmapā€™ to see section link. (<- mum that means click the mouse button.)

Flatt M, & Barzilay, E 2019, Scribble: The Racket Documentation Tool, Racket documentation series, v7.2, viewed 24 Feburary 2019,
https://docs.racket-lang.org/scribble/layers.html#(part._.Layer_.Roadmap).

The bar for me is fairly low. I want to be able to do :h def or :doc def or something similar.

You raise a good point @Screwtapello, I would like to donate to a collaborative expense that promotes the usability of Kakoune. Iā€™m investing a considerable amount of time with the editor and finding myself with a list of features that Iā€™m unable to quickly implement. Has there been any discussion on financially backing Kakoune and community-driven features?

I currently support Polar-Bookshelf on Open Collective which promotes open source projects and would subscribe monthly to Kakoune development.

If youā€™re unfamiliar with the site: ā€œOpen Collective is an online funding platform for open and transparent communities. We provide the tools to raise money and share your finances in full transparency.ā€

Yes, the site is open source with a github repo.

Asciidoctor is a good example on open-collective that everyone can relate to with the use of asciidocs for Kakoune documentation.

@robertmeta yes the sentiment of documentation as a secondary concern is widespread. This can be seen on thousands of github readme.md landing pages and with no considerable investment coming from Github (or a competitor) to inspire design. A userā€™s first experience with your hard work is the bland flat and unimaginative dull landscape of markdown in either light or dark grey and now continuously looping gifs. Sorry, bit of a rant and I too prefer reading an applications documentation inside the environment or pdf.

1 Like

@Screwtapello congrats! The PR about searching in plugins doc is now merged https://github.com/mawww/kakoune/pull/3704

Do you mind recapping what the impact on plugins authors who want to benefit from this new feature? Thanks

4 Likes

The thirty-second summary: If you include a file with the .asciidoc extension in your plugin, Kakouneā€™s :doc command can now find it and display it just like all the built-in documentation.

The standard autorestore and lint plugins now have very basic documentation which you can browse with :doc autorestore and :doc lint. :doc itself now has documentation too - you can display it with :doc doc, but maybe one day it will be updated to display that documentation with bare :doc, instead of displaying an error.

I wrote a lot more on the wiki about what kind of documentation plugins should have and where to put it, but this is definitely not a command or requirement ā€” itā€™s helpful to have some basic conventions to start with, but Iā€™m sure theyā€™ll shift over time as new plugin authors copy whatever popular plugin authors do.

5 Likes

Thanks a lot for your explanations.

The new doc.kak now searches inside the /autoload directory which is definitely the way to go.

Unfortunately, for situations where plugins are handled by plug.kak, the /plugins directory is voluntarily managed independently for the classical autoloading system, meaning that .asciidoc inside it are not found.

I donā€™t expect this exotic directory scanning to be handled by Kakoune itself, but does anyone has an elegant solution in mind that could be patched on plug.kak? Thanks!

@Delapouite I feel that with the autoload and built-in module system that plug.kak is not necessary to have a declarative kakrc now.

A doc_search_paths option could be added to allow plugins to register their path.

doc.kak

declare-option -docstring 'Search paths for the documentation' str-list doc_search_paths "%val{config}/autoload" "%val{runtime}/doc" "%val{runtime}/rc"

define-command doc -params 1..2 -shell-script-candidates %{
  eval "set -- $kak_quoted_opt_doc_search_paths"
  ...
  find -L "$@" ...
} %{
  ...
}

plug.kak

set-option -add global doc_search_paths "%val{config}/plugins"