-
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: universal REPL #2806
Comments
I think this would be a cool feature that could fit into core. At first I thought it might be better as a plugin since it isn't based on some hypothetical REPL standard (something like LSP for REPL) but it looks like the plugin is just interfacing with stdin/stdout of the repl program which seems straightforward and consistent. There are some rough edges to that plugin that I think could be designed around: the I don't have any ideas to get the ball rolling but it could be smart to canvas popular repl plugins from vim/emacs/etc and source ideas there. |
@the-mikedavis, thanks for your response. To be clear, I'm not suggesting to re-implement that plugin exactly (I agree with you that it is rough and about the I mostly do work in R, and a bit of julia. For R, there is the very nice [Sidenote: there is an official julia plugin for R, but it doesn't have any REPL features. It does however allow for nice entry of some Unicode characters, which can also be used in non-julia files, and is a useful feature in its own right. But that is a topic for a different discussion.] Also, I am very aware of the "kitchen sink" and "bloat" concern. Indeed, this functionality goes beyond simple editing, but for interactive languages (and there are many) it can be considered "essential" (I would never go back to copy-pasting code into a different terminal/window). Then there are external tools like tmux that could be leveraged, but here we have the problem of catering to all kinds of users with their vastly heterogeneous environments, and so it really is better not to rely on the presence of external tools (approaches like that also always feel kind of kludgy). Obviously, there may be users that do not consider this feature important, but I hope there are enough who do. Plugins create a plethora of fragmentation problems (as exemplified by [Other sidenote: SpaceVim has built-in REPL functionality and defines some consistent bindings across different languages (layers in SpaceVim). Unfortunately they are pretty cumbersome because they require like 4 key presses, which is really too much]. Thanks, and sorry for the long post. |
Just wanted to add to the reference material. I'm most familiar with
As far as use cases go, I'm also coming from an R-for-analytics background. In the domain of scientific programming, interactive execution in a REPL is a very central part of development. |
the Jupyter protocol fits this description, with plenty of implementations by this point |
I always thought of Jupyter as something web based, but maybe I don't understand it very well. Could it work in a console editor like helix? |
Yeah. While the web-based notebook is by now Jupyter's calling card, the project started life as a python console REPL. It still supports that console REPL, along with a few other frontends (a Qt REPL, alternative web UIs, etc). They all talk to backends over a ZeroMQ-based protocol. Granted: these first-party UIs represent the overwhelming majority of Jupyter use, so their "first among equals" status would be even stronger than VSCode's is for LSP and DAP. That hasn't stopped people from trying, though:
(VS Code) Python Interactive window:
I haven't really used in-editor Jupyter support since a decade ago—but from what I can tell, a few features have emerged as conventional:
helix already has many discussions in flight for 4 and 5, but for these other features:
|
@riwsky Thanks a lot for your detailed response! In that case, it does sounds like a good approach. I agree with your comment about keybindings. Ideally, you want to be able to "spam" the |
I wrote a temporary workaround using tmux, but it would be nice if we could query helix internal state to get the file type as an argument, i.e. #!/bin/bash
# copy text from system clipboard
tmux set-buffer "$(xclip -o -sel clipboard)"
# select bottom pane
tmux select-pane -t "{down-of}" 2> /dev/null
# if no bottom pane, then start a REPL based on file type
if [ $? -ne 0 ]; then
# TODO(danj): replace this with ft from helix
ft=$(ps -u | rg -m 1 -o '\.(py|jl|R)')
declare -A repl
repl['.py']='ipython'
repl['.jl']='julia'
repl['.R']='R'
if [ ${repl[$ft]+_} ]; then
tmux split-window -v -p 35 ${repl[$ft]} # automaticaly focuses new pane
sleep 1 # sleep so tmux list-panes updates with this new pane
fi
fi
# if there is a bottom pane, check that it is a valid REPL before pasting
cmd=$(tmux list-panes -f '#{pane_at_bottom}' -F '#{pane_current_command}')
if [[ $cmd =~ python|R|julia ]]; then
tmux paste-buffer -p
tmux send-keys C-m C-m
tmux select-pane -t "{up-of}"
fi An alternative might be to allow keymaps that are specific to languages in languages.toml, so you could do something like |
I definitely agree with that! I just want to share my thoughts on the "launching a REPL if one doesn't exist": What I like best about my current neovim/slime setup is that I'm free to easily setup the REPL I want, how I want it, in a terminal window or a tmux pane. The plugin simply sends the code to the target REPL, with the convenience of setting magic comments like I don't think that the Helix REPL feature should setup a REPL for the user if one isn't active. With different languages requiring different setups, this level of abstraction can get difficult to implement very quickly. What if there are different active REPLs? How to choose the environment and working directory of the REPL? The user can easily launch their REPL in the language they want, in the environment they want. Sourcing a Python environment or setting a Julia project is easy and I think that a user who needs REPL integration knows how to do it. I think the REPL integration in Helix should focus on implementing a convenient way to target an existing REPL and reliably send the selected text or the current cell to that REPL. Some people use different magic comments for their cells, so I think this could be a configuration option, that defaults to "# %%". |
Another benefit of the REPL being outside of the editor, on top of being easier to setup the desired environment, is that the REPL can be on a ssh connection. I myself run the editor locally and pass the code to a remote REPL on a server. |
Just adding some cool reference material. -https://git.sr.ht/~detegr/nvim-bqn -https://github.com/mrossinek/deuterium -https://github.com/dccsillag/magma-nvim Hopefully someone talented can implement something like that. |
I'm very impressed with Helix and I would also find the ability to have basic repl integration extremely useful. However, I would suggest this will take considerable design effort to get right. Often I am a great believer in just building something and then refining it based on usage and feedback. However, in other systems I've used which adopted this approach with respect to interfacing with repls and sub-processes generally, the result has tended to be overly complex with way too many options and points of confusion with respect to correct behaviour in various corner cases. The biggest source of problems seems to centre around definitions of output, return values, error codes and handling of stdout and stderr. Things become more complicated because different languages have different semantics here - some languages always return a value (Clojure, FP langs) , some only when the code explicitly does so (Python, many OOP langs) and some rely on stdout/sterr and process return codes (shells, CLI tools). Sometimes you may want the value returned by the repl or sub-process and sometimes you might want whatever was sent to stdout and/or stderr and sometimes you only care about the exit code. Ability to specify this in a clean and concise m,anner will be critical to a solid implementation. The other things to possibly think about would be session management. Sometimes, you want all the code evaluation to happen within the same session context and other times, you mgiht want the code executed in a clean context or you may just want an easy way to clear the context. The challenge is in how to provide this functionality in a flexible manner which is not complicated and which doesn't detract from its big strength - clarity and simplicity without sacrificing functionality. When I compare my neovim and helix configuration where I have largely equivalent functionality, the benefits of Helix are obvious. The ability to interface with a repl needs to maintain that clarity and simplicity if possible. |
I've been using a variation on the solution proposed by @danjenson above. I think this is a nice solution for a few reasons, and am increasingly in favor of a solution that brings this feature through composition with other tools, rather than being built in to helix itself. To simplify things further, the whole interface is a simple keymapping. I just pipe the selection into tmux (assuming you have an active session called [keys.normal."\\"]
space = [ ":pipe-to tmux load-buffer - \\; paste-buffer -t helix-target:0" ] This is a super simple solution, but seems to be quite feature complete for my needs. Of course it's a one-way street and you don't get any output back in |
This is really awesome and looks very promising. As far as I am concerned (and I am likely not alone, many people with some data science / ml / scientific programming background around me are in the same case), having a way to use functionality 'a la jupyter(lab)' is what will make me take the step to move to Helix. Wondering: even if a native / official solution is still in the pipelines, would it be possible to have a repo or folder on this repo somewhere, either Helix or wider community maintained, that:
? The idea would be that, in the meantime waiting for 'the ultimate Helix native solution', there may be 'perfectly good enough' temporary solutions, and that making these easy to use, and nurturing an ecosystem of users, and helping users migrate to better solutions as these are developed, would be very helpful and help the user community grow. |
@jerabaul29 I don't know if this fits your needs, but I use euporie to edit and run jupyter python notebooks in my console, it also has a vim mode (i wish it had a helix mode). But I agree it would be really awesome to have something like this in helix itself, with more familiar keybindings and faster iteration while writing code. |
I would like to share a setup that is perfectly working for me. TLDR
Here's the result, with ipython and Julia REPLs: ExplanationThe coding setup is based on tmux, with Helix containing the code and a REPL on the pane directly to the right. The target REPL can be anywhere though, even in another tmux session, you just need to modify the So, inspired by @dgkf, I added
to my Helix config. Once we select the text we want to send to the REPL with Helix's select mode, That would be enough for many, but I find it counter-productive to have the need to select the text to be sent every time, specially because in my REPL workflow I work with cell headers in my scripts, like
The ideal scenario for me is to be anywhere in the file and, with a keystroke, execute the whole cell containing the cursor and go to the next cell. I would like to have this bound to a key, but could only achieve this with a macro. So, for now, I need to re-type the macro every time I start a new Helix instance. Inspired by this reddit comment, I arrived at
It seems long, but the logic is straight forward:
Now just type I was 90% happy with this because I still needed to setup the macro every time and it doesn't work for the last cell. To mitigate the last cell problem, a simple solution is to have
at the end of the file and Finally, to avoid needing to re-type the long macro every time, a solution is to add another bind to Helix, that will call the
Double So now I can |
@sourproton Thanks for sharing! Is there some way to adapt that script to pipe to Jupyter lab? I use Helix on a remote machine and would like to be able to display interactive plots (so can't use tmux). |
I wanted to barge in real quick to say that the "cell division standard" in buffers could be set as the "Form Feed" character, inserted through Ctrl+L in e.g. Emacs, which is a standard of the ASCII code for page break. As such, it perfectly delimits pieces of code larger than "blocks" but shorter than "buffers", would not collide with any other character nor interfere with other editors that do not use it. |
Yes, even a light weight cell input-output cell that is not as feature rich as euporie would be a pretty neat addition to helix |
Couldnt we just integrate euphorie? I know, the core team wants everything to be done in Rust, and still, I see that this could do the job: https://rustpython.github.io/ |
the issue isn't just that that is not written in rust which is generally a blocker but also that its specific to python. We will not include any language-specific tools or integration in core. That kind ow integration has to wait for plugins |
Its not specific to Python. There are dozens of languages supported. Also Rust |
@pascalkuthe Even Matlab, Wolfram has notebook options |
I think including something like https://gitlab.com/Screwtapello/kakoune-repl-buffer can make sense because that interfaces with an arbitrary shell command instead of integrating with one specific REPL. Maype something fancier beyond that may make sense but I don't think we should integrate language specific notebooks/repls into core (and I still count euporie as that, yes it suuports some other languages but its definitly primarily oriented towards python and not universal at all). At most we could have an arbitrary REPL mode that can have an API that plugins could provide backends to (with the default backend just being stdout/stdin) |
Just wanted to provide a heads up, A protocol similar to that of LSP or DAP for repls already exists, has existed for almost a decade now and that is nrepl: https://nrepl.org/nrepl/1.1/index.html. Albeit majorly used in clojure(script) community it is language and editor agnostic as mentioned in the following page: https://nrepl.org/nrepl/beyond_clojure.html, stating and I quote:
with numerous examples being provided for different languages e.g Python, Lua, R and others. The only odd thing I found in the protocol personally is the usage of bencode for the data transport, but that is merely the default provided option, and the reasoning was provided in the author here: https://stackoverflow.com/questions/59594424/why-bencode-has-been-used-for-transporting-clojure-code-to-nrepl-in-cider. What I and others particularly love about nrepl is its simplicity, it doesn't attempt to overcomplicate things while being easy to implement, even bencode the default format as mentioned in the reasoning is easy to implement manually. Other reasons include its stability, as it has had a decade to be rock-solid. I understand the argument that it lacks in language support compared to others like euporie leveraging Juptyer and various backends implemented in it, but those solutions are limited such in a way highlighted prior, they aren't a real "fit" because they don't work over a protocol which is a general requirement to an editor that doesn't want to build language-specific features to core, and doesn't have a plugin system yet. Something like nrepl should've had similar spotlight and adoption to LSP or DAP years ago, but unfortunately that wasn't the case. However it's never too late. And doing it over shell isn't going to give nobody a decent repl experience. |
From reading some of the comments in this thread, I fear there is a risk of conflating two different user cases/ideas here. One the one side, we have the 'notebook' concept with approaches like Jupyter, euporie and literate style programming environments, similar to Emacs' org-mode etc. On the other side, you have the repl driven development approach, which is similar, but not the same. The differences are somewhat subtle and not obvious from the outside. You really need to have used both approaches to apprciate these differences. For this reason, I'm not convinced that incorporating something like euporie is appropriate. THe nrepl approach is interesting, but I'm not sure it is correct either. There are other alternatives, such as socket based repls, which are also popular and which apparently address some of the percieved weaknesses with nrepl. I"m also not sure how well nrepl scales to other languages (I've only used it with Clojure/Clojurescript). My gut feeling here is that what Helix should really do is take the time to examine and assess how other editors have addressed both the two use cases (i.e. notebookand repl driven development), work out what has been done well and what could be done better and then see if an initial 'common ground' could be found which could be built into Helix as a core feature and which would then be leveraged on to develop more specific functionality for each of the use cases (something which would likely be part of the evenual plugin ecosystem?). BTW with respect to repl driven development, I think the neovim conjure plugin may be a good source of ideas. It provides a framework for implementing repls in a number of languages and is able to take advantage of existing interfaces e.g. for Clojure, it communicates over a nrepl connection. For repl driven development, how it itnerfaces with the repl is quite nice and a model which would liekly work well with Helix (due to also being based around a modal interface). |
People here seem to be conflating All Helix needs to do is start a kernel process for a specified language and then pass messages to it via a ZeroMQ socket. A user can then take whatever frontend they want, a notebook, a jupyterlab session, a terminal session and talk to the kernel to run code or inspect the state. Multiple frontends can be connected to a single kernel process at the same time so you can be inspecting and modifying kernel state in a kernel process from a terminal session and a JupyterLab session simultaneously. Helix would just be another frontend to the kernel process, able to talk to it via the specified jupyter protocol. |
I can see, that the optional/additional integration by Microsoft - dotnet interactive - can be an advantage here. Yes, this would pull dotnet in - but you do not depend on it for normal jupyter support. It is only to add a couple of really cool features, and comes with very little code to pay for: The main benefit is probably the sharing of variables between languages. So Python can bind the result of a calculation to a name, and F# can consume it. Please note, that this is ultimately simply another implementation of the Jupyter protocol. That means, a dotnet core runs a python and a R core, who then can communicate. Still compatible with regular old standards. There is also a neat alternative to .ipynb called .dib, but I forgot what its purpose was, and I am currently struggling to find proper doc to it. But I remind it was a worthwhile consideration as well. dotnet/interactive#3378 Much fun altogether |
I think the easiest and most direct way to support this feature would be to implement a Jupyter client. By easiest, I mean all of the possible errors cases and rough edges of running arbitrary code have been smoothed out already. By direct, I mean there is no need to invent some bespoke way of running code and managing session state. Moreover, the ecosystem is massive as indicated by the over 100 kernels already implemented and the APIs provided by Jupyter are well-specified, stable and battle-tested. In addition to supporting a basic REPL, this would would also naturally support the core pieces for full blow literate programming plugins. This isn't an "experimental" concept either. There are example implementations floating around, such as the Neovim plugin molten which runs can run code in any document and display the output to the user. No browser, no JSON, just fenced code blocks in Markdown and Neovim. And actually, this is document format agnostic as well. The markdown is not necessary, but just a common default. Moreover, Jupyter has already specified a file format (i.e., notebooks) for which we can save code outputs, so the code outputs can be saved, exported, and then re-imported later just as is done inside the browser-based frontend, except this is all achieved outside of any browser. Not that I think this should be supported in the core, but rather just speaking to the feature-completeness of Jupyter. |
Also notice, that Microsoft has implemented an improved notebook file format. I guess it might be a bit out of scope, but it may prove useful. |
@sourproton many thanks, your solution works very well and is my daily driver :) . Only one thing that is problematic: sometimes, I have a "multiple line selection", and I press I tried to modify the config entries you provide:
To a number of variations adding either |
I found the above solutions work for the most part but had trouble with some valid python code causing issues (mainly classes and extra lines within functions). So I wrote a little rust tool to slot into the keyboard shortcut call that fixes most of the indentation issues. If anyone is interested you can find it here: pypaste. |
Sharing my recipe for Helix + Kitty + virtual environment REPL workflow. Tried to make it simple, robust and straightforward. Inspired by danjenson and sourproton solutions. Prerequisites:
[] `~/.local/bin/ in $PATH
#!/bin/sh
if [ -f ./pyproject.toml ] && [ -f ./poetry.lock ]; then
kitty @ launch --type=window --title REPL --keep-focus --cwd current --hold --copy-env sh -c 'pyenv exec poetry run ipython' >/dev/null
elif [ -f ./Project.toml ] && [ -f ./Manifest.toml ]; then
kitty @ launch --type=window --title REPL --keep-focus --cwd current --hold --copy-env sh -c 'julia -i -e "using Pkg; Pkg.activate(\".\")"' >/dev/null
else
echo "Necessary workspace dependencies not found!"
exit 1
fi Make it executable:
[keys.normal]
"C-ret" = ":sh launch_workspace_repl.sh"
"S-ret" = [":pipe-to kitty @ send-text --match 'title:REPL' --bracketed-paste enable --stdin", ":pipe-to kitty @ send-text --match 'title:REPL' '\r'"]
Commentary:
|
I'm pretty impressed with this:
https://gitlab.com/Screwtapello/kakoune-repl-buffer
(also, watch the video here: https://asciinema.org/a/379916)
If something like this could be added to helix, it could potentially change my life. I know some other users are looking for this kind of feature as well (#1710).
Any thoughts?
The text was updated successfully, but these errors were encountered: