Zellij crashing when making web request in plugin

[lannonbr]

Issue description

When I run a plugin that tries to make a HTTP request using the web_request command, it crashes zellij and the following error is outputted in the logs:

WARN   |zellij_utils::ipc        | 2023-12-28 11:29:29.029 [router    ] [zellij-utils/src/ipc.rs:223]: Error in IpcReceiver.recv(): InvalidMarkerRead(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }) 
ERROR  |zellij_client            | 2023-12-28 11:29:29.030 [router    ] [zellij-client/src/lib.rs:356]: Received empty message from server 

Minimal reproduction

https://github.com/lannonbr/zellij-http-plugin-example

I'm running it by compiling the plugin with cargo build and then running it in an active zellij session with the zellij plugin command.

zellij plugin -- file:./target/wasm32-wasi/debug/zellij-http-plugin-example.wasm

Other relevant information

Zellij verison: 0.39.2

Rustc --version --verbose output:

rustc 1.74.0 (79e9716c9 2023-11-13)
binary: rustc
commit-hash: 79e9716c980570bfd1f666e3b16ac583f0168962
commit-date: 2023-11-13
host: aarch64-apple-darwin
release: 1.74.0
LLVM version: 17.0.4

[imsnif]

Hey, thanks for the heads up! For me this works

What I did:

1. Started a new Zellij session
2. Cloned your repository: https://github.com/lannonbr/zellij-http-plugin-example
3. Did cargo build
4. Did zellij plugin -- file:target/wasm32-wasi/debug/zellij-http-plugin-example.wasm
5. Focused on the plugin and pressed the Down arrow
6. Got the above in the logs

The error logs are unfortunately not very helpful, they're just the client disconnecting. The only thing I can think of that would cause this is running this plugin on an older version of Zellij...

[lannonbr]

So I tried the following:

• recompiling the plugin with the lastest version of rust (1.75)
• cleared my zellij cache folder
• made sure that both the zellij instance and zellij-tile were both on 0.39.2
• Tried running your weather-pal plugin as I was using such as a template on how to write a plugin with web requests.

And all of the above still ended with the client disconnecting with the same IPC error

Both plugins start and I see the initial default state / UI, but as soon as it tries making the HTTP request it breaks.

[imsnif]

Hum... when the plugin is already loaded, could you try zellij start-or-reload-plugin <plugin_url> and then after it recompiles and reloads try again?

Also, how did you install Zellij?

[lannonbr]

Okay, so tried the start-or-reload-plugin, the plugin does properly recompile, but when I press the down arrow to trigger the http request it exits again.

For how I installed Zellij: on the mac I was testing this, on I got it through homebrew. Went to another apple silicon mac I have and tried installing it via cargo install zellij and getting the same issue there as well.

[imsnif]

cargo install zellij --locked? I doubt that's it though...

Might be it's a mac thing (I'm on arch), this API is fairly new. Maybe @jaeheonji or @tlinford can see if they can reproduce (not urgent, only if/when you have the time!) ? (specifically the steps here: #3036 (comment)

[imsnif]

@lannonbr - does it happen with every url? Does it happen with running a command through a plugin?

[lannonbr]

Yes it seems to be any URL. And I tried using the run_command and subsequent Event::RunCommandResult callback, and it seems that there is no error when running run_command, but the result never seems to return.

[imsnif]

Alright. Super odd. I'm going to wait and see if the people I ping have a chance to try and reproduce. Or anyone else encountering this...

[imsnif]

@lannonbr - another question: can you load plugins from the web? Like if you did a github release in your repo, placed the .wasm file there and loaded it with an http url with zellij plugin? Does it work? I'm wondering if the issue is with the http client or something different.

[lannonbr]

Oh interesting, I published the wasm up as a release here: https://github.com/lannonbr/zellij-http-plugin-example/releases/download/0.1.0/zellij-http-plugin-example.wasm

So I try running the following

zellij plugin -- https://github.com/lannonbr/zellij-http-plugin-example/releases/download/0.1.0/zellij-http-plugin-example.wasm

And it is crashing the same way as if I tried making a web request in the plugin

[imsnif]

Alright, getting warmer. Would you be comfortable running a standalone project that does this? https://github.com/zellij-org/zellij/blob/main/zellij-server/src/background_jobs.rs#L256-L316

I'm curious to see if it works or you get a more specific error...

If it's too much, I can try to find some time to make a debug repository and point you at it

[lannonbr]

Yeah I think I could try extracting this portion into a standalone rust binary and see if things are continuing to break or not.

[lannonbr]

So I tried extracting solely the web request portion and made this reproduction: https://github.com/lannonbr/zellij-3036-repro

That said, it works as intended on my machine, so I am now assuming something is breaking in the channels code you have to send said data up and down between the server and plugin.

[imsnif]

Well, I get next steps would be some good ol' println debugging. Would you be up to compiling the project from source and doing some back and forth?

[lannonbr]

Sure yeah let me grab the full source and I can start doing more deeper debugging.

[imsnif]

Cool. To start us off, could you place some logs inside the linked method in background_jobs? Mostly just to see if we get there when you make the WebRequest and if so, where it gets stuck.

You can do this eg. like this log::info!("Making request to url: {:?}", url) and then it'll get to the Zellij log. To be 100% sure it's working, you can also place one here, which is the entry point for the web request on the Zellij side: https://github.com/zellij-org/zellij/blob/main/zellij-server/src/plugins/zellij_exports.rs#L661

My guess right now btw is that we do get to the background jobs but fail there, probably due to the openssl packaging. But let's follow the facts first.

[imsnif]

A few things that might trip you up when running from source:

1. Be sure to run the project with cargo x run --singlepass so that it goes through our build-system (and doesn't take ages to compile plugins in debug mode)
2. When loading the plugin from within the dev environment, be sure to use the created executable and not the zellij one. eg target/dev-opt/zellij plugin -- <path>

I hope that's everything :)

[lannonbr]

Okay, so I got a working dev version of zellij up and running and have been logging out things. The current spot I am at is the following line in the background_jobs code: https://github.com/zellij-org/zellij/blob/main/zellij-server/src/background_jobs.rs#L282

Where it runs an await on the request to actually trigger the web request. I put a log of the req on the line right before it, and that prints out, but if I put a log after I don't see it.

I also have also expanded the await? into a match req.await and tried logging if it hits the Err arm rather than letting it propagate up the chain if it fails, and sadly none of those logs are appearing. as well

Doing some further exploration of the initial error I saw in the logs at the start of all of this:

WARN   |zellij_utils::ipc        | 2023-12-29 16:07:18.033 [router    ] [zellij-utils/src/ipc.rs:224]: Error in IpcReceiver.recv(): InvalidMarkerRead(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }) 
ERROR  |zellij_client            | 2023-12-29 16:07:18.033 [router    ] [zellij-client/src/lib.rs:356]: Received empty message from server 

That is the ipc going from the server -> client, so my next step is trying to figure out if there is another thread somewhere else that is triggering a IpcSenderWithContext.send call outside of the background_jobs

[truehumandesign]

I am getting the same error if I just use zellij --layout strider or zellij plugin --zellij:strider

Zellij verison: 0.39.2

Rustc --version --verbose output:

rustc 1.75.0 (82e1608df 2023-12-21)
binary: rustc
commit-hash: 82e1608dfa6e0b5569232559e3d385fea5a93112
commit-date: 2023-12-21
host: aarch64-apple-darwin
release: 1.75.0
LLVM version: 17.0.6

it just exits with Received empty message from server

In the logs I get the following:

WARN   |zellij_utils::ipc        | 2023-12-30 11:41:57.863 [router    ] [zellij-utils/src/ipc.rs:223]: Error in IpcReceiver.recv(): InvalidMarkerRead(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
ERROR  |zellij_client            | 2023-12-30 11:41:57.863 [router    ] [zellij-client/src/lib.rs:356]: Received empty message from server

Before these logs I also have about 50 entries of:

ERROR  |zellij::sessions         | 2023-12-30 13:13:23.022 [main      ] [src/sessions.rs:56]: Failed to read resurrection layout file: Os { code: 2, kind: NotFound, message: "No such file or directory" }

My output of zellij setup --check is:

❯ zellij setup --check
[Version]: "0.39.2"
[CONFIG DIR]: "/Users/tobias/.config/zellij"
[LOOKING FOR CONFIG FILE FROM]: "/Users/tobias/.config/zellij/config.kdl"
[CONFIG FILE]: Well defined.
[DATA DIR]: "/Users/tobias/Library/Application Support/org.Zellij-Contributors.Zellij"
[PLUGIN DIR]: "/Users/tobias/Library/Application Support/org.Zellij-Contributors.Zellij/plugins"
 Builtin, default plugins will not be loaded from disk.
 Create a custom layout if you require this behavior.
[LAYOUT DIR]: "/Users/tobias/.config/zellij/layouts"
[SYSTEM DATA DIR]: "/usr/share/zellij"
[ARROW SEPARATOR]: 
 Is the [ARROW_SEPARATOR] displayed correctly?
 If not you may want to either start zellij with a compatible mode: 'zellij options --simplified-ui true'
 Or check the font that is in use:
 https://zellij.dev/documentation/compatibility.html#the-status-bar-fonts-dont-render-correctly
[MOUSE INTERACTION]:
 Can be temporarily disabled through pressing the [SHIFT] key.
 If that doesn't fix any issues consider to disable the mouse handling of zellij: 'zellij options --disable-mouse-mode'
[DEFAULT EDITOR]: hx
[FEATURES]: []
[DOCUMENTATION]: zellij.dev/documentation

Installation was done via Homebrew - DATA DIR, PLUGIN DIR AND SYSTEM DATA DIR don't exist

[imsnif]

Friends, I appreciate the enthusiasm and willingness to help - but let's not conflate things. The IPC error and the "received empty message from server" errors are red herrings. They're not the issue - these happen every time the server crashes. Let's work with what we have from @lannonbr 's debugging because I think we're getting somewhere.

@lannonbr - I would like to try to recreate this error out of Zellij now that we've narrowed this down so that we have a minimal reproduction of what happens. In your test project, could you try including these dependencies verbatim? https://github.com/zellij-org/zellij/blob/main/zellij-utils/Cargo.toml#L58-L61

[lannonbr]

Okay yep so I did have that exact version of surf with the specified feature of curl-client set. I did not have openssl-sys added, but I just added it and in my test project I still am getting the http response back normally without any error. Is there somewhere in the code I should be using openssl-sys or is it being used by surf somewhere behind the scenes when you add said crate?

[imsnif]

Hmm. Quite a head scratcher. You don't need to use openssl-sys anywhere, no. Let's try it the other way: could you try building Zellij without openssl-sys and seeing if this maybe changes things for you?

[lannonbr]

Removed openssl-sys and still crashing as before

[imsnif]

Alright, thanks for checking. For the moment I'm out of ideas. I'll check back here if I can think of anything - I'm sorry I don't have any solution or workaround for you. I guess I need more data, specifically which platforms this fails on and what they have in common (or, maybe... a log somewhere from surf/curl telling us what happened :) ).

[imsnif]

Another thing we can try @lannonbr (if you don't mind me using you to collect more info) is with different http clients for surf: https://github.com/http-rs/surf#features (replacing curl-client in Cargo.toml). I'd be curious if this makes a difference.

[truehumandesign]

Friends, I appreciate the enthusiasm and willingness to help - but let's not conflate things. The IPC error and the "received empty message from server" errors are red herrings. They're not the issue - these happen every time the server crashes. Let's work with what we have from @lannonbr 's debugging because I think we're getting somewhere.

@lannonbr - I would like to try to recreate this error out of Zellij now that we've narrowed this down so that we have a minimal reproduction of what happens. In your test project, could you try including these dependencies verbatim? https://github.com/zellij-org/zellij/blob/main/zellij-utils/Cargo.toml#L58-L61

@imsnif should i open up a new issue for the strider plugin then?

[lannonbr]

Fascinating, tried all 3 other clients, h1-client, h1-client-rustls, and hyper-client, and all of them works as expected on ARM macOS.

So as you can see below, my plugin is now working as expected:

So it is the curl-client that is breaking on mac

[lannonbr]

Ohh interesting, So I didn't grab curl from homebrew, so it was using a x86 version that likely came with the xcode command line tools.

Original Curl output:

curl 8.4.0 (x86_64-apple-darwin23.0) libcurl/8.4.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1
Release-Date: 2023-10-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

Then when I installed curl with homebrew:

curl 8.5.0 (aarch64-apple-darwin23.0.0) libcurl/8.5.0 (SecureTransport) OpenSSL/3.2.0 zlib/1.2.12 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libssh2/1.11.0 nghttp2/1.58.0 librtmp/2.3 OpenLDAP/2.6.6
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

So I swapped back to curl-client in my local dev build of zellij and there was an update for the curl-sys crate to update to that same version of curl 8.5.0. and it is working again.

I tried running the plugin again with the new curl installed with the binary release of zellij (0.39.2) instead of my dev build and it is still crashing so I gotta assume something was updated in curl 8.5.0 that fixed the crash on ARM macOS

[rcalimlim]

I also had this issue, but this thread helped me get it fixed, thanks!

For anyone else running into this issue with an Apple Silicon Mac, I did the following:

1. Run curl -V to check if you're using the x86 binary (curl 8.X.X (x86_64-apple-darwin23.0))
2. If you are, you can use brew to install an ARM version: brew install curl
3. You might be prompted to do so by brew, but if not, be sure to set your environment variables correctly (I use fish shell, so translate to your shell config):

curl is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have curl first in your PATH, run:  
  fish_add_path /opt/homebrew/opt/curl/bin

For compilers to find curl you may need to set:
  set -gx LDFLAGS "-L/opt/homebrew/opt/curl/lib"
  set -gx CPPFLAGS "-I/opt/homebrew/opt/curl/include"

4. Confirm your environment is calling the right curl after sourcing your config or restarting your shell
5. Backup your zellij config files
6. Uninstall zellij (ex. brew uninstall zellij or cargo uninstall zellij)
7. Reinstall zellij with whichever supported pkg man
8. Confirm it's fixed by opening a basic zellij session and running zellij action start-or-reload-plugin https://github.com/lannonbr/zellij-http-plugin-example/releases/download/0.1.0/zellij-http-plugin-example.wasm from the command line. If zellij doesn't crash, then you should be good 👍
9. You likely won't need to restore your config from backup, but do so if it got wiped

@imsnif This seems like it could be a pretty impactful bug, as I'm sure a lot of people are using Apple Silicon with the provided x86 curl binary and clearing the cache doesn't work for this fix. Would it be worth it to list this as a gotcha or, better, pin the curl version for the compilation process?

[imsnif]

First @rcalimlim - I'm sorry for just now replying to this directly, I have been referring to your great comment above for a while. So first - thanks for looking into this! This has already helped quite a few people out of some trouble.

Second - @jaeheonji - do you think we can somehow pin the curl version as @rcalimlim suggested in the comment above? This has the effect of sometimes failing the http(s) request either when loading a plugin from the web or when performing an http(s) request from a plugin. I think I saw curl-sys has some sort of macos configuration parameter that might be related? If you have the time to look into this that would be great!!

[kinsomicrote]

Hello, for some reason, this does not work for me after following the steps outlined above. I currently get this when I run curl -V:

curl 8.8.0 (aarch64-apple-darwin23.4.0) libcurl/8.8.0 (SecureTransport) OpenSSL/3.3.0 zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libssh2/1.11.0 nghttp2/1.61.0 librtmp/2.3 OpenLDAP/2.6.7
Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

I found this issue through hiasr/vim-zellij-navigator#1. It worked for the OP of that issue but does not work for me. I'm not sure what I might be doing wrong. I encounter this when working with the vim-zellij-navigator.
I downloaded the plugin to my machine as done in #3319, and it worked fine.

The logs I see from Zellij look like this:

INFO   |zellij_client            | 2024-05-28 12:31:18.152 [main      ] [zellij-client/src/lib.rs:178]: Starting Zellij client! 
INFO   |zellij_server            | 2024-05-28 12:31:18.162 [main      ] [zellij-server/src/lib.rs:284]: Starting Zellij server! 
INFO   |zellij_server            | 2024-05-28 12:31:18.217 [main      ] [zellij-server/src/lib.rs:1068]: Compiling plugins using Cranelift 
INFO   |zellij_server::plugins   | 2024-05-28 12:31:18.217 [wasm      ] [zellij-server/src/plugins/mod.rs:203]: Wasm main thread starts 
WARN   |zellij_server::route     | 2024-05-28 12:31:18.217 [server_router] [zellij-server/src/route.rs:1037]: Server not ready, trying to place instruction in retry queue... 
WARN   |zellij_server::route     | 2024-05-28 12:31:18.218 [server_router] [zellij-server/src/route.rs:1138]: Server ready, retrying sending instruction. 
WARN   |zellij_utils::input::layo| 2024-05-28 12:31:18.218 [wasm      ] [zellij-utils/src/input/layout.rs:105]: Overriding plugin alias 
WARN   |zellij_utils::input::layo| 2024-05-28 12:31:18.218 [wasm      ] [zellij-utils/src/input/layout.rs:105]: Overriding plugin alias 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.218 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.220 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.223 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
INFO   |zellij_server::plugins::p| 2024-05-28 12:31:18.223 [async-std/runti] [zellij-server/src/plugins/plugin_loader.rs:533]: Loaded plugin 'tab-bar' from cache folder at '/Users/jioke/Library/Caches/org.Zellij-Contributors.Zellij' in 2.293041ms 
INFO   |zellij_server            | 2024-05-28 12:31:18.224 [async-std/runti] [zellij-server/src/lib.rs:1068]: Compiling plugins using Cranelift 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.224 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
INFO   |zellij_server::plugins::p| 2024-05-28 12:31:18.226 [async-std/runti] [zellij-server/src/plugins/plugin_loader.rs:533]: Loaded plugin 'status-bar' from cache folder at '/Users/jioke/Library/Caches/org.Zellij-Contributors.Zellij' in 4.591375ms 
INFO   |zellij_server            | 2024-05-28 12:31:18.226 [async-std/runti] [zellij-server/src/lib.rs:1068]: Compiling plugins using Cranelift 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.226 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.227 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.229 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:18.230 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:21.315 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:23.995 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
ERROR  |zellij_utils::input::layo| 2024-05-28 12:31:26.316 [screen    ] [zellij-utils/src/input/layout.rs:1088]: Failed to read layout dir: Os { code: 2, kind: NotFound, message: "No such file or directory" } 
WARN   |zellij_utils::ipc        | 2024-05-28 12:31:26.553 [router    ] [zellij-utils/src/ipc.rs:226]: Error in IpcReceiver.recv(): InvalidMarkerRead(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }) 
ERROR  |zellij_client            | 2024-05-28 12:31:26.553 [router    ] [zellij-client/src/lib.rs:375]: Received empty message from server 

Any hint on how to resolve this? Thanks.

[meisocafe]

@kinsomicrote What happens if you try to download the url for the plugin file using curl from the command line? Does it correctly download the file? If that's not the case, then the problem is related to curl and not zellij.

[kinsomicrote]

@meisocafe I just tried now and I was able to download the plugin using curl.

[rcalimlim]

@kinsomicrote Can you confirm that you rebuilt zellij after installing the Apple Silicon version of curl? Truthfully, I'm a bit ignorant of the compilation/linking process for deps, but I'm pretty sure the issue is that your zellij is still using the bad curl because it was originally built/installed before you downloaded the correct curl binary.

I would try this:

1. backup your zellij config (for me this is ~/$HOME/.config/zellij
2. uninstall zellij using the package manager that you installed it with
3. confirm once more that curl -V produced the correct Apple Silicon version
4. download and build zellij from cargo with

cargo install zellij --locked

5. try to reproduce the issue with this newly built version of zellij (hopefully this should fix it)

[kinsomicrote]

@rcalimlim Thank you, that fixed it!

[towry]

This pr fixed the issue for me #3349

tested on my mac x86_64-darwin.