Add Steel as an optional plugin system

[mattwparas]

Notes:

• I still need to rebase up with the latest master changes, however doing so causes some headache with the lock file, so I'll do it after some initial feedback. Also, this depends on the event system in Add an event system #8021.
• The large diff size is a combination of lock file changes + the dependency on the event system PR. The diff has ended up quite large with all of the other stuff
• I'm currently pointing to the master branch of steel as a dependency. This will point to a stable release on crates once I cut a release.

---

Opening this just to track progress on the effort and gather some feedback. There is still work to be done but I would like to gather some opinions on the direction before I continue more.

You can see my currently functioning helix config here and there are instructions listed in the STEEL.md file. The main repo for steel lives here, however much documentation is in works and will be added soon.

The bulk of the implementation lies in the engine.rs and scheme.rs files.

Design

Given prior conversation about developing a custom language implementation, I attempted to make the integration with Steel as agnostic of the engine as possible to keep that door open.

The interface I ended up with (which is subject to change and would love feedback on) is the following:

pub trait PluginSystem {
    /// If any initialization needs to happen prior to the initialization script being run,
    /// this is done here. This is run before the context is available.
    fn initialize(&self) {}

    fn engine_name(&self) -> PluginSystemKind;

    /// Post initialization, once the context is available. This means you should be able to
    /// run anything here that could modify the context before the main editor is available.
    fn run_initialization_script(&self, _cx: &mut Context) {}

    /// Allow the engine to directly handle a keymap event. This is some of the tightest integration
    /// with the engine, directly intercepting any keymap events. By default, this just delegates to the
    /// editors default keybindings.
    #[inline(always)]
    fn handle_keymap_event(
        &self,
        _editor: &mut ui::EditorView,
        _mode: Mode,
        _cxt: &mut Context,
        _event: KeyEvent,
    ) -> Option<KeymapResult> {
        None
    }

    /// This attempts to call a function in the engine with the name `name` using the args `args`. The context
    /// is available here. Returns a bool indicating whether the function exists or not.
    #[inline(always)]
    fn call_function_if_global_exists(
        &self,
        _cx: &mut Context,
        _name: &str,
        _args: &[Cow<str>],
    ) -> bool {
        false
    }

    /// This is explicitly for calling a function via the typed command interface, e.g. `:vsplit`. The context here
    /// that is available is more limited than the context available in `call_function_if_global_exists`. This also
    /// gives the ability to handle in progress commands with `PromptEvent`.
    #[inline(always)]
    fn call_typed_command_if_global_exists<'a>(
        &self,
        _cx: &mut compositor::Context,
        _input: &'a str,
        _parts: &'a [&'a str],
        _event: PromptEvent,
    ) -> bool {
        false
    }

    /// Given an identifier, extract the documentation from the engine.
    #[inline(always)]
    fn get_doc_for_identifier(&self, _ident: &str) -> Option<String> {
        None
    }

    /// Fuzzy match the input against the fuzzy matcher, used for handling completions on typed commands
    #[inline(always)]
    fn available_commands<'a>(&self) -> Vec<Cow<'a, str>> {
        Vec::new()
    }

    /// Retrieve a theme for a given name
    #[inline(always)]
    fn load_theme(&self, _name: &str) -> Option<Theme> {
        None
    }

    /// Retrieve the list of themes that exist within the runtime
    #[inline(always)]
    fn themes(&self) -> Option<Vec<String>> {
        None
    }

    /// Fetch the language configuration as monitored by the plugin system.
    ///
    /// For now - this maintains backwards compatibility with the existing toml configuration,
    /// and as such the toml error is exposed here.
    #[inline(always)]
    fn load_language_configuration(&self) -> Option<Result<Configuration, toml::de::Error>> {
        None
    }
}

If you can implement this, the engine should be able to be embedded within Helix. On top of that, I believe what I have allows the coexistence of multiple scripting engines, with a built in priority for resolving commands / configurations / etc.

As a result, Steel here is entirely optional and also remains completely backwards compatible with the existing toml configuration. Steel is just another layer on the existing configuration chain, and as such will be applied last. This applies to both the config.toml and the languages.toml. Keybindings can be defined via Steel as well, and these can be buffer specific, language specific, or global. Themes can also be defined from Steel code and enabled, although this is not as rigorously tested and is a relatively recent addition. Otherwise, I have been using this as my daily driver to develop for the last few months.

I opted for a two tiered approach, centered around a handful of design ideas that I'd like feedback on:

The first, there is a init.scm and a helix.scm file - the helix.scm module is where you define any commands that you would like to use at all. Any function exposed via that module is eligible to be used as a typed command or via a keybinding. For example:

;; helix.scm

(provide shell)

;;@doc
;; Specialized shell - also be able to override the existing definition, if possible.
(define (shell cx . args)
  ;; Replace the % with the current file
  (define expanded (map (lambda (x) (if (equal? x "%") (current-path cx) x)) args))
  (helix.run-shell-command cx expanded helix.PromptEvent::Validate))

This would then make the command :shell available, and it will just replace the % with the current file. The documentation listed in the @doc doc comment will also pop up explaining what the command does:

Once the helix.scm module is require'd - then the init.scm file is run. One thing to note is that the helix.scm module does not have direct access to a running helix context. It must act entirely stateless of anything related to the helix context object. Running init.scm gives access to a helix object, currently defined as *helix.cx*. This is something I'm not sure I particularly love, as it makes async function calls a bit odd - I think it might make more sense to make the helix context just a global inside of a module. This would also save the hassle that every function exposed has to accept a cx parameter - this ends up with a great deal of boilerplate that I don't love. Consider the following:

;;@doc
;; Create a file under wherever we are
(define (create-file cx)
  (when (currently-in-labelled-buffer? cx FILE-TREE)
    (define currently-selected (list-ref *file-tree* (helix.static.get-current-line-number cx)))
    (define prompt
      (if (is-dir? currently-selected)
          (string-append "New file: " currently-selected "/")
          (string-append "New file: "
                         (trim-end-matches currently-selected (file-name currently-selected)))))

    (helix-prompt!
     cx
     prompt
     (lambda (cx result)
       (define file-name (string-append (trim-start-matches prompt "New file: ") result))
       (temporarily-switch-focus cx
                                 (lambda (cx)
                                   (helix.vsplit-new cx '() helix.PromptEvent::Validate)
                                   (helix.open cx (list file-name) helix.PromptEvent::Validate)
                                   (helix.write cx (list file-name) helix.PromptEvent::Validate)
                                   (helix.quit cx '() helix.PromptEvent::Validate)))

       (enqueue-thread-local-callback cx refresh-file-tree)))))

Every function call to helix built ins requires passing in the cx object - I think just having them be able to reference the global behind the scenes would make this a bit ergonomic. The integration with the helix runtime would make sure whether that variable actually points to a legal context, since we pass this in via reference, so it is only alive for the duration of the call to the engine.

Async functions

Steel has support for async functions, and has successfully been integrated with the tokio runtime used within helix, however it requires constructing manually the callback function yourself, rather than elegantly being able to use something like await. More to come on this, since the eventual design will depend on the decision to use a local context variable vs a global one.

Built in functions

The basic built in functions are first all of the function that are typed and static - i.e. everything here:

• Typed
• Static

However, these functions don't return values so aren't particularly useful for anything but their side effects to the editor state. As a result, I've taken the liberty of defining functions as I've needed/wanted them. Some care will need to be decided what those functions actually exposed are.

Examples

Here are some examples of plugins that I have developed using Steel:

File tree

Source can be found here

filetree.webm

Recent file picker

Source can be found here

recent-files.webm

This persists your recent files between sessions.

Scheme indent

Since steel is a scheme, there is a relatively okay scheme indent mode that only applied on .scm files, which can be found here. The implementation requires a little love, but worked enough for me to use helix to write scheme code 😄

Terminal emulator

I did manage to whip up a terminal emulator, however paused the development of it while focusing on other things. When I get it back into working shape, I will post a video of it here. I am not sure what the status is with respect to a built in terminal emulator, but the one I got working did not attempt to do complete emulation, but rather just maintained a shell to interact with non-interactively (e.g. don't try to launch helix in it, you'll have a bad time 😄 )

Steel as a choice for a language

I understand that there is skepticism around something like Steel, however I have been working diligently on improving it. My current projects include shoring up the documentation, and working on an LSP for it to make development easier - but I will do that in parallel with maintaining this PR. If Steel is not chosen and a different language is picked, in theory the API I've exposed should do the trick at least with matching the implementation behavior that I've outlined here.

Pure rust plugins

As part of this, I spent some time trying to expose a C ABI from helix to do rust to rust plugins directly in helix without a scripting engine, with little success. Steel supports loading dylibs over a stable abi (will link to documentation once I've written it). I used this to develop the proof of concept terminal emulator. So, you might not be a huge fan of scheme code, but in theory you can write mostly Rust and use Steel as glue if you'd like - you would just be limited to the abi compatible types.

System compatibility

I develop off of Linux and Mac - but have not tested on windows. I have access to a windows system, and will get around to testing on that when the time comes.

[atahrijouti]

Hi @mattwparas and great work btw. I've tested this PR on Windows and it worked as expected. (Windows, no WSL, and a sprinkle of MSYS2 for some binaries. Not sure if MSYS2 was relevant)

I tested it prior to your recent documentation updates, so I had to compare your dotfiles to the examples and get close tk what you had and it worked 💪

[nnym]

:config-reload resets the editor's configuration without reloading the Steel configuration files and I haven't found any command or function that can do that. There is :update-configuration! but it apparently does nothing. I set my keymap in init.scm so having :config-reload reload it too would be great.

[mattwparas]

:config-reload resets the editor's configuration without reloading the Steel configuration files and I haven't found any command or function that can do that. There is :update-configuration! but it apparently does nothing. I set my keymap in init.scm so having :config-reload reload it too would be great.

This is something that will come with some more changes to the configuration system (and as a result, I don't have a perfect fix), but in the mean time, this should help force a reload of the relevant parts of the init when you call config-open:

(require (prefix-in helix.static. "helix/static.scm"))

;; Define a function for your keybindings
(define (setup-keybindings)
  (define scm-keybindings (hash "insert" (hash "ret" ':scheme-indent "C-l" ':insert-lambda)))

  ;; Grab whatever the existing keybinding map is
  (define standard-keybindings (deep-copy-global-keybindings))

  ;; File tree stuff
  (define file-tree-base (deep-copy-global-keybindings))
  (add-global-keybinding (hash "normal" (hash "C-r" (hash "f" ":recentf-open-files"))))

  (merge-keybindings standard-keybindings scm-keybindings)
  (merge-keybindings file-tree-base FILE-TREE-KEYBINDINGS)

  (set-global-buffer-or-extension-keymap (hash "scm" standard-keybindings FILE-TREE file-tree-base)))

;;;;;;;;;;;;;;;;;;;;;;;;;; Options ;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Define a function for your options
(define (configure-options)
  (file-picker (fp-hidden #f))
  (cursorline #t)
  (soft-wrap (sw-enable #t)))

;; New config reload, shadows the old one so it should have the things set up
(define (config-reload)
  (helix.config-reload)
  (configure-options)
  (setup-keybindings))

;; Otherwise, before the init file is done, setup things like usual
(setup-keybindings)
(configure-options)

[nnym]

Thanks but it seems that I can't override :config-reload whether I define it in init.scm or helix.scm and whether I export it or don't; so I tried changing its name.

(define (main)
	(helix.theme "everblush")
	; keymap
)

(define (crl)
	(helix.config-reload)
	(main))

Now I can use :crl which resets the configuration and very briefly (for 1 frame or so) flashes the theme set in main before returning the purple default theme; this happens every time that I run :crl except the first (because the theme has not been reset to the default before that). My keymap is likewise reset to the default. I don't know what happens under the hood but it looks like config-reload runs after main somehow. Adding (time/sleep-ms 1000) between the calls to config-reload and main only adds a delay and does not fix anything so perhaps config-reload runs after :crl exits. Running main through enqueue-thread-local-callback-with-delay with a 0 (!) ms delay fixes it; enqueue-thread-local-callback does not help. Running :config-reload and then :main works as expected too.

[mattwparas]

Thanks but it seems that I can't override :config-reload whether I define it in init.scm or helix.scm and whether I export it or don't; so I tried changing its name.

(define (main)
	(helix.theme "everblush")
	; keymap
)

(define (crl)
	(helix.config-reload)
	(main))

Now I can use :crl which resets the configuration and very briefly (for 1 frame or so) flashes the theme set in main before returning the purple default theme; this happens every time that I run :crl except the first (because the theme has not been reset to the default before that). My keymap is likewise reset to the default. I don't know what happens under the hood but it looks like config-reload runs after main somehow. Adding (time/sleep-ms 1000) between the calls to config-reload and main only adds a delay and does not fix anything so perhaps config-reload runs after :crl exits. Running main through enqueue-thread-local-callback-with-delay with a 0 (!) ms delay fixes it; enqueue-thread-local-callback does not help. Running :config-reload and then :main works as expected too.

Yeah so not sure how I didn't run into this, maybe I'm running a build that is slightly different, but it has an event queue for configuration updates that won't get processed until after the steel script exits, so I should have seen this coming 😅

Your solution with the 0 ms delay callback, as ugly as it is, should get the job done, since it'll first process the config-reload event, followed by applying the modifications to the resulting config object in a callback. I believe the resulting configuration changes that I mentioned before should fix this whole process

[CreggEgg]

What is left to be done here? Also, are you looking for contributions and if so, what can I do to be helpful?

[jkilopu]

Hi, great project! I noticed an issue with the implementation of shell in helix.scm. We should use apply to pass a list of arguments to helix.run-shell-command.

(define (shell . args)
  (define expanded (map (lambda (x) (if (equal? x "%") (current-path) x)) args))
  (helix.run-shell-command expanded) ;; should be changed to (apply helix.run-shell-command expanded)
)

[mattwparas]

What is left to be done here? Also, are you looking for contributions and if so, what can I do to be helpful?

I have to address some lingering comments and otherwise clean up the code, then work on integrating the upcoming configuration system changes.

If you're particularly interested in contributing, the best thing you can do at this point is to use this branch and report issues, or alternatively, fix those issues and open a PR against this branch. I wouldn't necessarily recommend committing to adding new features to this unless its plainly obvious those features should be available, i.e. the lack of them now appears to be an oversight.

I think the first iteration of the plugin system might be a trimmed down version of this branch since I've done quite a bit of hacking on exposing APIs that might not get added in the first go, or need further thought in order to hone the exact API that does get exposed. For example, the work that I've done with the component API hasn't even really been reviewed yet; this will probably need some love before it gets merged, but the configuration part will most likely be more straightforward.

[mattwparas]

Hi, great project! I noticed an issue with the implementation of shell in helix.scm. We should use apply to pass a list of arguments to helix.run-shell-command.

(define (shell . args)
  (define expanded (map (lambda (x) (if (equal? x "%") (current-path) x)) args))
  (helix.run-shell-command expanded) ;; should be changed to (apply helix.run-shell-command expanded)
)

Good catch, I'll push a fix for this

[StringNick]

@mattwparas hi, i have problem with ur config open-term/xplr

on mac zsh is located /bin/zsh

[StringNick]

i changed pass to /bin/zsh on mac, after call exit in opened terminal, something crashing with this

same for :xplr

[merisbahti]

I don't know if this is the place for this discussion, but I managed to install this version of helix and I wanna build a plugin.

I want to either make a "git blame" or "copilot suggestion" plugin - but both of them make use of some kind of ghost text and I've tried searching for it but cannot find a way to make ghost text.

I'm not sure if this is possible in this plugin-system today, or would it require some more capabilities provided from helix?

[adriangalilea]

I don't know if this is the place for this discussion, but I managed to install this version of helix and I wanna build a plugin.

I want to either make a "git blame" or "copilot suggestion" plugin - but both of them make use of some kind of ghost text and I've tried searching for it but cannot find a way to make ghost text.

I'm not sure if this is possible in this plugin-system today, or would it require some more capabilities provided from helix?

I'm a broken record:
What about text suggestion support? #10259

[kirawi]

AFAICT It does not expose such an API. I think the plan is to get a minimal version that can replace the config out and then add on to it w/ subsequent PRs to reduce the collective effort.