Kak-spec - Unit testing for Kakoune scripts!

Hello!

kak-spec

I have written a unit test framework called kak-spec for kakoune scripts and plugins. The framework aims to make the development of robust kakoune scripts an easier and more pleasant experience.

Story of kak-spec

One of the big strengths of kakoune is that it can be configured extensively by writing *.kak scripts and plugins. Unfortunately, developing and learning to develop such scripts can be laborious, as some in this forum may have experienced. Personally, I have developed many kakoune scripts in an often painful ad-hoc manner. This is particularly the case when it comes to testing. I have been the sole alpha and beta tester of my scripts, and most of that testing has consisted simply of manual use of my scripts.

Needless to say, such ad-hoc development with manual testing has lead to errors. For example, I have had cases where I write a script that works, but much later I realize that it fails when used with multiple selections, or with a selection containing only a single character, or in an empty buffer, or on the first or last character of the buffer, or after a kakoune update has changed some behavior. Most of these problems would have been avoided by proper testing.

Given these experiences, I looked for existing testing practices for kakoune scripts and found that the kakoune repository itself has a regression test script that can be adapted for plugins as well. That is great! I proceeded to copy it and use it for some personal plugins that I have. This was a nice improvement–I was finally unit testing some of my scripts. I should have done so from day one!

Despite having now unit tests, my experience was not quite as smooth as I had come to expect from using frameworks for other languags like rspec for ruby and crystal spec for crystal. I admit the comparison is not quite fair though, since one is a testing script for a single project whereas the others are entire unit testing frameworks aimed at large user bases of entire languages. Nevertheless, I grew to want changes to the facts that the kakoune regression test script

  • requires each single test to be specified as a directory of multiple files,
  • takes its time to run each test using a dedicated kakoune process, and
  • is not available as a conveniently installable script that you can easily run on any plugin.

Motivated by the above, I developed the unit testing framework kak-spec that

  • conveniently allows writing multiple test cases in a single file,
  • saves time by starting one kakoune process per test file rather than test case, and
  • is available as an installable executable that can be ran easily on any spec file you choose.

For more information, please see the repository kak-spec. I hope that kak-spec can be of use to many developers in the kakoune community. Please let me know if you have suggestions/thoughts/issues.

Cheers!

10 Likes

Hi Jori,
Installed and ran my first dry-run of kak-spec from the command line and it worked perfectly, thank you. It’s a big project :+1: well done.

A casual question no rush to answer needed. My goal is a spec/*.kak-spec for each 410 of the below. I have read your documentation and source/example/spec files but I’m still just a little unfamiliar.

Question) how would I provide a specific kak-spec entry point for options/functions/regex/strings in javaConstants.kak or a best guess?

Code snippet and the entry points:

# javaConstants.kak +290
declare-option str javax_swing_abstractbutton_active "nop"

define-command java-constants-add %{
# javaConstants.kak +2283
  try %{
    execute-keys -draft '%s (?:javax\.swing\.[A-Za-z])<ret>'
    try %{
      execute-keys -draft '%s (?:javax\.swing\.[A-I])<ret>'
      try %{
        execute-keys -draft '%s (?:javax\.swing\.AbstractButton)<ret>'
        evaluate-commands %opt{javax_swing_abstractbutton_active}
        set-option -add window static_words %opt{javax_swing_abstractbutton}
        set-option window javax_swing_abstractbutton_active "nay"
      }
    }
  }
}

define-command java-constants-remove %{
# javaConstants.kak +5957
  try %{
    evaluate-commands %opt{javax_swing_abstractbutton_active}
  } catch %{
    try %{
      execute-keys -draft '%s (?:javax\.swing\.AbstractButton)<ret>'
    } catch %{
      set-option -remove window static_words %opt{javax_swing_abstractbutton}
      set-option window javax_swing_abstractbutton_active "nop"
    }
  }
}

If required the repo is:
git clone https://bitbucket.org/KJ_Duncan/kakoune-java.kak.git
or
plug "KJ_Duncan/kakoune-java.kak" domain "bitbucket.org"

Planned new directory structure for batch running kak-spec would be:

.
├── rc
│   ├── javaAnnotationEnumHierarchy.kak
│   ├── javaClassHierarchy.kak
│   ├── javaConstants.kak
│   ├── javaInterfaceHierarchy.kak
│   ├── java.kak
│   └── javaPolicy.kak
├── spec
│   └── regex-javax-swing.kak-spec
├── README.md
└── UNLICENSE

2 directories, 9 files

Don’t go all out, just looking for a bit more direction to get my head around the work you have done. Thanks in advance, be back in a couple of weeks. Bye :wave:


Thanks Jori, appreciate you going the extra distance. I have read, copied, and pasted. Now of to learn it as I just introduced a bug that keeps moving the posts on me. I’ll be back to post the events and any lessons learned in a couple of weeks.

Bye :wave:


I’m in deep now Jori, loving the kak-spec.

For future readers of this thread reference was made to:

# my file has these and ran with kak-spec errors.
hook -group java-constant-field window InsertIdle .* java-constants-add
hook -group java-constant-field window ModeChange push:.*:insert java-constants-remove

## edited: now works with InsertIdle without a wrapper function
kak-spec \
    -title 'java-constants-add then java-constants-remove removes constants from static_words' \
    -input ' javax.swing.AbstractButton' \
    -eval %(
        set-option window filetype java
        java-constants-add
    ) \
    -exec '%Hdi' \
    -eval %(
        set-option window filetype java
        java-constants-remove
    ) \
    -expect-%opt(javax_swing_abstractbutton_active) 'nop' \
    '-expect-%sh(eval set -- "$kak_quoted_opt_static_words"; printf "%s\n" "$@" | sort)' \
        regex(((?!BORDER_PAINTED_CHANGED_PROPERTY|VERTICAL_TEXT_POSITION_CHANGED_PROPERTY).)*)

Good on you Jori, Thanks buddy.

Just wanted to say that it looks awesome and I expect it to aid my progress as a kakoune-extender. Will have a look as soon as possible.

1 Like

Delightful. Excellent.

Next I would love to see some tutorials for how to build simple plugins. I would be thrilled to pair on them, but I don’t have a strong mental model for what’s possible. I would especially be interested in trying to build them test-first using a Universal Architecture approach (in the style of Gary Bernhardt’s “Boundaries” talk).

Invitations to pair on plugins are welcome. :wink:

1 Like

Thanks for the encouraging comments everyone! :smiley:

@duncan It is good to hear that your dry-run worked!

With a directory structure like that, for the specs you could create a helper file spec/spec-helper.kak-no-autoload:

source "%opt(kak_spec_source_dir)/../rc/javaInterfaceHierarchy.kak"
source "%opt(kak_spec_source_dir)/../rc/javaPolicy.kak"
source "%opt(kak_spec_source_dir)/../rc/javaConstants.kak"
source "%opt(kak_spec_source_dir)/../rc/javaClassHierarchy.kak"
source "%opt(kak_spec_source_dir)/../rc/java.kak"
source "%opt(kak_spec_source_dir)/../rc/javaAnnotationEnumHierarchy.kak"

That way you can easily source your plugin in individual spec files. I looked into your repository and sketched the following spec file:

spec/regex-javax-swing.kak-spec
source "%opt(kak_spec_source_dir)/spec-helper.kak-no-autoload"

declare-option str-list saved_static_words

kak-spec \
    -title 'java filetype hook populates static_words' \
    -eval %(
        set-option window filetype java
    ) \
    '-expect-%sh(eval set -- "$kak_quoted_opt_static_words"; printf "%s\n" "$@" | sort)' \
        regex(AbsentInformationException.*ZoneRulesException)

kak-spec \
    -title 'java-constants-add adds constants to static_words' \
    -input ' javax.swing.AbstractButton' \
    -eval %(
        set-option window filetype java
        java-constants-add
    ) \
    -expect-%opt(javax_swing_abstractbutton_active) 'nay' \
    '-expect-%sh(eval set -- "$kak_quoted_opt_static_words"; printf "%s\n" "$@" | sort)' \
        regex(.*BORDER_PAINTED_CHANGED_PROPERTY.*VERTICAL_TEXT_POSITION_CHANGED_PROPERTY.*)

kak-spec \
    -title 'java-constants-add called twice adds constants to static_words only once' \
    -input ' javax.swing.AbstractButton' \
    -eval %(
        set-option window filetype java
        java-constants-add
        java-constants-add
    ) \
    -expect-%opt(javax_swing_abstractbutton_active) 'nay' \
    '-expect-%sh(eval set -- "$kak_quoted_opt_static_words"; printf "%s\n" "$@" | sort)' \
        regex(((?!BORDER_PAINTED_CHANGED_PROPERTY).)*BORDER_PAINTED_CHANGED_PROPERTY((?!BORDER_PAINTED_CHANGED_PROPERTY).)*)

kak-spec \
    -title 'java-constants-remove removes constants from static_words' \
    -input ' javax.swing.AbstractButton' \
    -eval %(
        set-option window filetype java
        java-constants-add
        java-constants-remove
    ) \
    -expect-%opt(javax_swing_abstractbutton_active) 'nop' \
    '-expect-%sh(eval set -- "$kak_quoted_opt_static_words"; printf "%s\n" "$@" | sort)' \
        regex(((?!BORDER_PAINTED_CHANGED_PROPERTY|VERTICAL_TEXT_POSITION_CHANGED_PROPERTY).)*)

Some notes about it:

  • I noticed that you use some InsertIdle hooks. Presently, those might not have time to run before kak-spec quits. Unfortunately, there is currently no way to tell kak-spec to keep running longer in order to let idle events fire. Hence, to work with kak-spec it would be best to wrap any *Idle hook callbacks in commands and to call those commands in specs.

  • In your case the static_words option is huge! Currently kak-spec does not offer any convenience features to make testing such huge arrays easier. Therefore, I had to think a bit and I considered two ways to test it:

    1. Converting the static_words list to a newline delimited string and matching that string against a regular expression. This is a compact but heuristic solution. Nevertheless, I went with this option in the above example for the compactness. The expectation arguments are a bit hard on the eyes though.
    2. Specifying a list of all the values with expectations like -expect-%opt(static_words)-( AbsentInformationException abstract AbstractMethodError ... ZoneRulesException). However, there would have been hundreds of elements to list in your case.
  • On my machine 3/4 of the tests I sketched work, but I use a kakoune version without the -remove switch to set-option that your plugin depends on. I am suspecting that the switch might be the reason for the failure and therefore I did not look into the matter further.

I hope this helps you and perhaps others to get started using kak-spec.

@jbrains That is an interesting video. Also, I agree that it would be nice for there to be more plugin development tutorials. I just found Intro to Kakoune completers by @Screwtapello which looks really nice from a quick glance. I will need to take a closer look at that.

1 Like

Personally, I have developed many kakoune scripts in an often painful ad-hoc manner. This is particularly the case when it comes to testing. I have been the sole alpha and beta tester of my scripts, and most of that testing has consisted simply of manual use of my scripts.

I felt an outpouring of sympathy upon reading your words! Thank you for your hard work - I’ll be sure to try and use this tool. And this definitely deserves a bump.

EDIT: I might add some test specs to the repository for my first plugin just to show off and be cool.