Rewriting/reorganising Kakoune's documentation

schickm · February 2, 2019, 5:39pm

I’ve definitely found myself wondering about the organization of the docs pages. Reorganizing all the docs would be a worthy effort, but maybe an intermediate solution - create an “index” page and have :docs with no arguments start there.

It would serve as the top level documentation page that lists all of the current documentation pages, and short blurbs about what they cover. Once those blurbs are written, I think it’d become pretty clear where the overlaps are, or what pages need to be merged.

I realized I’m talking about creating a documentation page for documentation pages - more things to maintain - but it might be a lower effort means of making the existing docs more accessible to newer user’s who don’t know where to begin. Additionally, while the autocomplete listing of docs are nice, it’s not the most accessible for folks who don’t even know what things like “expansions” means.

Screwtapello · February 5, 2019, 5:24am

I like this plan. If you don’t have time to try it out in the near future, I might give it a shot.

Yeah, that’d be great. Of course, one reason Vim’s documentation is great is because they use a custom syntax with different markup for different cross references - so :foo is always a command, 'foo' is always a runtime option, +foo is always a compile-time option, and *foo* is a generic cross-reference. All that stuff is picked up by Vim’s documentation-indexer.

Kakoune uses Asciidoc syntax, which isn’t quite as convenient. As best I can tell, the syntax for making something a cross-reference anchor is [[like-this]], or even worse, anchor:like-this[]. That means Kakoune’s internal AsciiDoc renderer would need to filter out that text, and that means that jumping to a particular anchor gets more complex because you have to load the buffer and do the jump before it gets reformatted.

It should be possible, but it feels like it would be a lot of work.

That would be rather nice. It’d also be a good place to document features like “you can hit Enter on a cross reference to follow the link”.

andreyorst · February 5, 2019, 6:06am

Screwtapello:

Yeah, that’d be great. Of course, one reason Vim’s documentation is great is because they use a custom syntax with different markup for different cross references - so :foo is always a command, 'foo' is always a runtime option, +foo is always a compile-time option, and *foo* is a generic cross-reference. All that stuff is picked up by Vim’s documentation-indexer.

Kakoune uses Asciidoc syntax, which isn’t quite as convenient. As best I can tell, the syntax for making something a cross-reference anchor is [[like-this]] , or even worse, anchor:like-this[] . That means Kakoune’s internal AsciiDoc renderer would need to filter out that text, and that means that jumping to a particular anchor gets more complex because you have to load the buffer and do the jump before it gets reformatted.

Vim uses tags for navigating in documentation. It generates ctags compatible tags per doc file, and later these tags are used in documentation for jumping. What is really cool is that Kakoune uses asciidoc, and ctags can uderstand asciidoc. And Kakoune understands ctags (ahem), so I wonder if Kakoune could utilize ctags for better navigation in docs. Basically we have everything we need for that.

Screwtapello · February 5, 2019, 8:03am

POSIX only requires ctags to support C and Fortran, and even Exuberant Ctags doesn’t support AsciiDoc. Universal Ctags apparently does, although it’s not listed in the docs.

Meanwhile, although Kakoune understands ctags, “This script requires the readtags command available in ctags source but not installed by default”.

While Kakoune depends on these third-party tools for good reasons, I don’t think it’s a good idea to have Kakoune’s built-in documentation system require them. Generally, Kakoune’s basic functionality requires only standard POSIX tools.

The ctags format is simple enough that we could probably write our own indexer, and the resulting tags file should be small enough that industrial-strength readtags isn’t needed for good performance, but we’d still have the problems of updating Kakoune’s AsciiDoc renderer, and making sure that looking up a tag jumps to the right spot before formatting is applied.

andreyorst · February 5, 2019, 8:31am

Well, ctags support of asciidoc isn’t really needed in this case - tag files can be generated by kakoune itself, and for parsing we basically need a tool that can do regex search over file, such as awk or grep, which fits POSIX requirements.

True. I was astonished when I’ve compared readtags with grep performance on 607MiB file (4 million lines) - it is in order of magnitude faster compared to grep. But this difference isn’t so great for small files, and I suppose documentation tags will be relatively small.

I also think that this requirement can be wrapped by using posix tools combination in case readtags isn’t installed but that’s another topic.

robertmeta · February 6, 2019, 12:44am

does anybody else agree that it can be difficult to find things in Kakoune’s documentation?

Exceptionally hard, and documentation is the lifeblood of a long-lived editor.

would anybody else like to see the documentation reorganised?

Yes, not even that picky about how – I feel most reworks would be an improvement.

is anybody willing to contribute financially to reorganising the documentation?

If it ended with inline, searchable, referable documentation, absolutely.

My question is simply, what would it cost: just to an order of magnitude? You could start a bountysource today if @mawww basically pre-agreed to merge the result (with obviously some information on where it was going).

mawww · February 6, 2019, 3:43am

This is because readtags can do a binary search if the tags file is sorted, which is why Kakoune relies on readtags in the first place instead of grep its way through the tags.

More generally, we already support cross references in the documention (just hit enter on the links), we also have a rudimentary index system through the :doc command auto completion. I guess this could be expanded to support additional syntax, say :doc :command-name for prompt commands (with auto completion, as soon as you type :doc : the completion shows all commands in the documentation pages, same if you type :doc <key> to get a key documentation, and we just need to find a syntax for options…).

I am pretty proud that doc.kak provides the current system in only 165 lines of kak script (rendering a subset of asciidoc included), we have a bit of room to improve it before it becomes horribly complex.

As to the general organization of the pages, this grew up in an organic fashion, so I am all for a good rationalization of all that.

All that to say, I believe we are not that far from a good help system, we need to reorganize and cleanup the pages, and add a few features to the doc command completion system so that we can easily search for a command, key or option.

Screwtapello · February 7, 2019, 6:02am

Yeah, it should be easy enough to write a ctags-compatible indexer, and a custom lookup tool, but it does mean that Kakoune’s existing ctags support isn’t much help.

I originally mentioned financial support because I’ve done documentation writing before and felt pretty confident I could bang something out.

Since that time, however, I’ve been convinced by @occivink’s description of how the docs could be reorganised without a full rewrite, and the general tone here seems to be that searchability is a more pressing need. I’m a lot less confident about my ability to build a documentation searching system in Kakoune-script, so I don’t really want to take people’s money when I might not have anything to show for it at the end.

andreyorst · February 7, 2019, 9:59am

Why it is so? The only change needed for current implementation to work with documentation tags (provided by some indexer) is to replace readtags with grep or awk, in case readtags isn’t installed. The rest could be used as is.

duncan · February 23, 2019, 4:11pm

Hi all,
I know this problem not as a programmer but a report writer. Over the past five years I have constructed 47 reports which equates to over 110,000 words using mainly tex/latex to pdf and MSWord/MacTeX/TexWorks as editors. Which are all no longer in the tool chain thanks to your efforts on the Kakoune modal editing, done right.

My problems are exactly the same as described in this thread. With multiple documents poor cross-referencing with duplication of attempts to find subject matter dependant on the day/month/year. I am currently believe that my solution lies with scribble not as a racket documentation tool but as a preprocessor for a PMBOK (<- make mum proud moment).

I am still in my first steps of scribble and will be continuing my education this semester with more reports giving greater opportunity to grasp scribble. Thought it was worth mentioning as a tool in this discussion thread and with your programming experience determining if it provides a solution for Kakoune documentation.

Whatever the out come is what’s one more report, sign me up. Happy to help on the Kakoune documentation. I am a part-time programmer with just three years’ experience. Make that two years, damn you, Haskell.

This would be my first experience contributing to an open source project so training wheels are tightly secure, with git man page on hotkey.

Obligatory links below:

srcibble
scribble/bnf
BNF Grammer examples
scribble/xref

The reference below is an overview of the tool chain: mouse event on the heading ‘6.1.2 Layer Roadmap’ to see section link. (<- mum that means click the mouse button.)

Flatt M, & Barzilay, E 2019, Scribble: The Racket Documentation Tool, Racket documentation series, v7.2, viewed 24 Feburary 2019,
https://docs.racket-lang.org/scribble/layers.html#(part._.Layer_.Roadmap).

robertmeta · February 27, 2019, 7:07pm

The bar for me is fairly low. I want to be able to do :h def or :doc def or something similar.

duncan · March 2, 2019, 4:19pm

You raise a good point @Screwtapello, I would like to donate to a collaborative expense that promotes the usability of Kakoune. I’m investing a considerable amount of time with the editor and finding myself with a list of features that I’m unable to quickly implement. Has there been any discussion on financially backing Kakoune and community-driven features?

I currently support Polar-Bookshelf on Open Collective which promotes open source projects and would subscribe monthly to Kakoune development.

If you’re unfamiliar with the site: “Open Collective is an online funding platform for open and transparent communities. We provide the tools to raise money and share your finances in full transparency.”

Yes, the site is open source with a github repo.

Asciidoctor is a good example on open-collective that everyone can relate to with the use of asciidocs for Kakoune documentation.

@robertmeta yes the sentiment of documentation as a secondary concern is widespread. This can be seen on thousands of github readme.md landing pages and with no considerable investment coming from Github (or a competitor) to inspire design. A user’s first experience with your hard work is the bland flat and unimaginative dull landscape of markdown in either light or dark grey and now continuously looping gifs. Sorry, bit of a rant and I too prefer reading an applications documentation inside the environment or pdf.

Delapouite · October 1, 2020, 3:15pm

@Screwtapello congrats! The PR about searching in plugins doc is now merged https://github.com/mawww/kakoune/pull/3704

Do you mind recapping what the impact on plugins authors who want to benefit from this new feature? Thanks

Screwtapello · October 2, 2020, 4:51pm

The thirty-second summary: If you include a file with the .asciidoc extension in your plugin, Kakoune’s :doc command can now find it and display it just like all the built-in documentation.

The standard autorestore and lint plugins now have very basic documentation which you can browse with :doc autorestore and :doc lint. :doc itself now has documentation too - you can display it with :doc doc, but maybe one day it will be updated to display that documentation with bare :doc, instead of displaying an error.

I wrote a lot more on the wiki about what kind of documentation plugins should have and where to put it, but this is definitely not a command or requirement — it’s helpful to have some basic conventions to start with, but I’m sure they’ll shift over time as new plugin authors copy whatever popular plugin authors do.

Delapouite · October 4, 2020, 10:38am

Thanks a lot for your explanations.

The new doc.kak now searches inside the /autoload directory which is definitely the way to go.

Unfortunately, for situations where plugins are handled by plug.kak, the /plugins directory is voluntarily managed independently for the classical autoloading system, meaning that .asciidoc inside it are not found.

I don’t expect this exotic directory scanning to be handled by Kakoune itself, but does anyone has an elegant solution in mind that could be patched on plug.kak? Thanks!

alexherbo2 · October 4, 2020, 1:54pm

@Delapouite I feel that with the autoload and built-in module system that plug.kak is not necessary to have a declarative kakrc now.

A doc_search_paths option could be added to allow plugins to register their path.

doc.kak

declare-option -docstring 'Search paths for the documentation' str-list doc_search_paths "%val{config}/autoload" "%val{runtime}/doc" "%val{runtime}/rc"

define-command doc -params 1..2 -shell-script-candidates %{
  eval "set -- $kak_quoted_opt_doc_search_paths"
  ...
  find -L "$@" ...
} %{
  ...
}

plug.kak

set-option -add global doc_search_paths "%val{config}/plugins"

Screwtapello · October 5, 2020, 12:04am

Unlike scripts which need to all be loaded when Kakoune starts up, documentation only needs to be available before the user first types :doc. Plugin managers could just symlink any *.asciidoc files they find into the autoload directory during the startup process, maybe inside a “managed-documentation” directory to make it clear it’s managed by an automated system and shouldn’t be messed with.

duncan · October 5, 2020, 2:25am

@Jori_Bomanson is this functionality something that could or would be added to your search-doc.kak plugin. Maybe with a flag :search-doc --plugins?

Just a thought.

github.com

jbomanson/search-doc.kak/blob/master/rc/search-doc.kak#L17


      
              "%val(runtime)/doc" \
              "%val(runtime)/rc"
          
          evaluate-commands %sh(
              test -d "$kak_opt_plug_install_dir" &&
                  printf "%s" 'set-option -add global search_doc_path "%opt(plug_install_dir)"'
          )
          
          declare-option -hidden str search_doc_docstring "search-doc <topic>: search kak documentation for a topic"
          
          # A shell script that must be safe to include in %sh(...)-expansions.
          # Requires a variable "task" to be set.
          declare-option -hidden str search_doc_script %(
              get_directories_0 ()
              {
                  local some=
                  eval set -- "$kak_quoted_opt_search_doc_path"
                  for directory
                  do
                      if test -d "$directory"; then
                          printf '%s\0' "$directory"

now on my todo list, thanks Jori.

Jori_Bomanson · October 9, 2020, 11:11am

Yeah, I think it is best for search-doc.kak to search the same files as doc.kak. Thanks for prompting me to this thread!

For now, I updated search-doc.kak to look for the same *.asciidoc files as doc.kak in https://github.com/mawww/kakoune/pull/3704. If a doc_search_paths option or similar will be added to doc.kak, then it can be added to search-doc.kak as well.

duncan · September 17, 2021, 5:40pm

Hi all, I am going through the docs doc/pages/*.asciidoc.

Started at buffers.asciidoc and finished on scopes.asciidoc (alphabetical order) linking all references in each file to their file name nodes. The most welcome file was the last one scopes.asciidoc as it has a central pivot point for references and only the one node duplication to options.asciidoc.

I found it quite interesting hope you do too, made with this plugin: kakoune-plantuml.kak the pics source code is below.

Source code

@startuml

[*] → Buffers

Buffers → Commands
Buffers → Buffers

Commands → Buffers : debug buffers
Commands → Buffers : fifo buffers
Commands → Buffers : scratch buffers
Commands → Options : builtin options
Commands → Buffers : debug buffers
Commands → Buffers : debug buffers
Commands → Options : declare options
Commands → Options : set option
Commands → Options : unset option
Commands → Options : update option
Commands → Commands : declaring new commands
Commands → Commands : aliases
Commands → Scopes
Commands → Commands : aliases
Commands → Scopes
Commands → Execeval
Commands → Execeval
Commands → Mapping
Commands → Scopes
Commands → Mapping
Commands → Scopes
Commands → Hooks
Commands → Scopes
Commands → Options : builtin options
Commands → Hooks : disabling hooks
Commands → Hooks
Commands → Scopes
Commands → Hooks
Commands → Faces : markup strings
Commands → Faces
Commands → Scopes
Commands → Faces
Commands → Scopes
Commands → Highlighters
Commands → Highlighters
Commands → Commands : decaring new commands
Commands → Faces : markup strings
Commands → Registers

Command_Parsing → Expansions

Execeval → Registers
Execeval → Buffers : debug buffers
Execeval → Hooks : disabling hooks
Execeval → Mapping
Execeval → Hooks

Expansions → Command_Parsing
Expansions → Command_Parsing : typed expansions
Expansions → Commands : declaring new commands
Expansions → Options
Expansions → Registers
Expansions → Keys

Faces → Highlighters

Highlighters → Regex : groups
Highlighters → Options : types
Highlighters → Faces : markup strings

Hooks → Scopes
Hooks → Commands : hooks
Hooks → Mapping
Hooks → Mapping
Hooks → Mapping
Hooks → Execeval : execute keys specific switches
Hooks → Regex : groups
Hooks → Keys
Hooks → Execeval

Keys → Mapping
Keys → Hooks
Keys → Options : builtin options
Keys → Mapping : default mappings
Keys → Mapping : default mappings
Keys → Mapping : default mappings
Keys → Mapping : default mappings
Keys → Options
Keys → Options
Keys → Options
Keys → Options
Keys → Mapping : default mappings
Keys → Mapping : default mappings
Keys → Registers
Keys → Expansions : shell expansions
Keys → Registers
Keys → Registers
Keys → Registers
Keys → Registers
Keys → Registers
Keys → Expansions : shell expansions
Keys → Expansions
Keys → Registers
Keys → Commands
Keys → Expansions : typed expansions
Keys → Modes : user mode

Mapping → Modes
Mapping → Scopes
Mapping → Expansions
Mapping → Keys

Modes → Hooks
Modes → Keys
Modes → Mapping
Modes → Keys : movement
Modes → Keys : changes
Modes → Keys : insert mode
Modes → Keys : goto commands
Modes → Keys : view commands
Modes → Keys : prompt commands
Modes → Keys : object commands

Options → Scopes
Options → Highlighters : specs highlighters
Options → Highlighters : specs highlighters
Options → Faces : markup strings
Options → Hooks : disabling hooks
Options → Keys : movement
Options → Faces : markup strings
Options → Expansions : option expansions

Regex → Regex : compatibility
Regex → Regex : modifiers

Registers → Keys : changes
Registers → Keys : searching
Registers → Keys : macros
Registers → Keys : marks
Registers → Registers : marks
Registers → Keys : changes through external programs
Registers → Keys : marks
Registers → Expansions : value expansions

Scopes → Commands
Scopes → Faces
Scopes → Highlighters
Scopes → Hooks
Scopes → Mapping
Scopes → Options
Scopes → Expansions : value expansions
Scopes → Options
Scopes → Options

Scopes → [*]

hide empty description

@enduml

Cool guys, bye .

Update: By removing the duplication from each file both generlisation (to a file itself) and specialisation (to a files subsection) links we get a more workable default. This makes the assumption that the duplicated links are unnecessary and one link can be provided at the end of a subsection (idea only). The graphs do look more deliberate to a user’s intentions. And a lot more like Eve Online

Updated source code

@startuml

[*] → Buffers

Buffers → Commands
Buffers → Buffers

Commands → Buffers : debug buffers
Commands → Buffers : fifo buffers
Commands → Buffers : scratch buffers
Commands → Options : builtin options
Commands → Options : declare options
Commands → Options : set option
Commands → Options : unset option
Commands → Options : update option
Commands → Commands : declaring new commands
Commands → Commands : aliases
Commands → Scopes
Commands → Execeval
Commands → Mapping
Commands → Hooks : disabling hooks
Commands → Hooks
Commands → Faces : markup strings
Commands → Faces
Commands → Highlighters
Commands → Registers

Command_Parsing → Expansions

Execeval → Buffers : debug buffers
Execeval → Hooks
Execeval → Hooks : disabling hooks
Execeval → Mapping
Execeval → Registers

Expansions → Commands : declaring new commands
Expansions → Command_Parsing
Expansions → Command_Parsing : typed expansions
Expansions → Keys
Expansions → Options
Expansions → Registers

Faces → Highlighters

Highlighters → Faces : markup strings
Highlighters → Options : types
Highlighters → Regex : groups

Hooks → Commands : hooks
Hooks → Execeval
Hooks → Execeval : execute keys specific switches
Hooks → Keys
Hooks → Mapping
Hooks → Regex : groups
Hooks → Scopes

Keys → Commands
Keys → Expansions
Keys → Expansions : shell expansions
Keys → Expansions : typed expansions
Keys → Hooks
Keys → Mapping
Keys → Mapping : default mappings
Keys → Modes : user mode
Keys → Options
Keys → Options : builtin options
Keys → Registers

Mapping → Expansions
Mapping → Keys
Mapping → Modes
Mapping → Scopes

Modes → Hooks
Modes → Keys
Modes → Keys : movement
Modes → Keys : changes
Modes → Keys : insert mode
Modes → Keys : goto commands
Modes → Keys : view commands
Modes → Keys : prompt commands
Modes → Keys : object commands
Modes → Mapping

Options → Expansions : option expansions
Options → Faces : markup strings
Options → Highlighters : specs highlighters
Options → Hooks : disabling hooks
Options → Keys : movement
Options → Scopes

Regex → Regex : compatibility
Regex → Regex : modifiers

Registers → Expansions : value expansions
Registers → Keys : changes
Registers → Keys : searching
Registers → Keys : macros
Registers → Keys : marks
Registers → Keys : changes through external programs
Registers → Registers : marks

Scopes → Commands
Scopes → Expansions : value expansions
Scopes → Faces
Scopes → Highlighters
Scopes → Hooks
Scopes → Mapping
Scopes → Options

Scopes → [*]

hide empty description

@enduml