Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

miniflare segfaults during startup on Ubuntu 22.04 (jammy) #651

Closed
aroman opened this issue Aug 10, 2023 · 10 comments
Closed

miniflare segfaults during startup on Ubuntu 22.04 (jammy) #651

aroman opened this issue Aug 10, 2023 · 10 comments

Comments

@aroman
Copy link

aroman commented Aug 10, 2023

> npx wrangler dev --port $MC_SERVER_PORT --var MC_CONFIG_LOCAL_SUFFIX:$MC_CONFIG_LOCAL_SUFFIX LOG_LEVEL:$LOG_LEVEL GIT_SHA:$GIT_SHA

[WebServer]  ⛅️ wrangler 3.5.0
------------------
[WebServer] wrangler dev now uses local mode by default, powered by 🔥 Miniflare and 👷 workerd.
To run an edge preview session for your Worker, use wrangler dev --remote
[WebServer] Your worker has access to the following bindings:
- Durable Objects:
  - ROOM_DURABLE_OBJECT: RoomDurableObject
- Vars:
  - ENVIRONMENT: "Local"
  - WORKER_NAME: "magiccircle-gg"
  - MC_CONFIG_LOCAL_SUFFIX: "(hidden)"
  - LOG_LEVEL: "(hidden)"
  - GIT_SHA: "(hidden)"
[WebServer] ⎔ Starting local server...
[WebServer] workerd/util/symbolizer.c++:99: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
[WebServer] *** Received signal #11: Segmentation fault
stack: /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1693d32 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1693c85 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168513d /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1685843 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168be6c /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168e956 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168cfdd /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167fc76 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167f848 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167f624 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167f60c /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326483e /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326443d /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3262728 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32624ea /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1670[29](https://github.com/magic-circle-studio/magiccircle.gg/actions/runs/5817705610/job/15772781605#step:8:30)3 /lib/x86_64-linux-gnu/libc.so.6@29d8f /lib/x86_64-linux-gnu/libc.so.6@29e3f /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected][30](https://github.com/magic-circle-studio/magiccircle.gg/actions/runs/5817705610/job/15772781605#step:8:31)807.0/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167002d
[WebServer] ✘ [ERROR] MiniflareCoreError [ERR_RUNTIME_FAILURE]: The Workers runtime failed to start. There is likely additional logging output above.
@aroman
Copy link
Author

aroman commented Aug 10, 2023

I was able to fix this by adding apt-get update && apt-get install -y libc++1 to my GitHub Action. Still, this might be good to document somewhere...

@mrbbot
Copy link
Contributor

mrbbot commented Aug 10, 2023

Hey! 👋 This is very concerning, cloudflare/workerd#793 and cloudflare/workerd#800 should mean you don't need to install libc++1 yourself. Any chance you can share your worker code or a minimal reproduction? I've just to run a very basic worker using [email protected] dev on Ubuntu 22.04 and it seems to be working. 😕

@aroman
Copy link
Author

aroman commented Aug 10, 2023

hey @mrbbot, yes — I had exactly the same impression from reading those bugs. Our worker code is quite complex, so I can't share it, but I can provide some more context:

  1. I am running ubuntu via Microsoft's playwright-1.36.2-jammy docker container image.

  2. Here is a gist containing two files: the full YAML file of my github action, and slightly more full crash logs. (Note: the YAML file includes the installation of libc; prior to including that step I got the crash you saw).

  3. I can share a bit more about our worker:

  • compatibility date 2023-03-21
  • it's a DurableObject (1 binding)
  • we use WebSockets
  • the error occurs during startup, seemingly prior to any HTTP/WS requests flowing

@mrbbot
Copy link
Contributor

mrbbot commented Aug 11, 2023

Hey! Thanks for the additional information. Unfortunately, I'm still unable to reproduce this issue with playwright-1.36.2-jammy, compatibility_date = "2023-03-21", and a basic Durable Object using WebSockets. 🙁


Something that may help identity the issue is running a debug build of workerd. This may give us a more useful stack trace. You'll want to have a local Docker container running the amd64 playwright-1.36.2-jammy image, then unzip and copy this file to node_modules/workerd/bin/workerd.

$ docker run -it --rm --platform=linux/amd64 mcr.microsoft.com/playwright:v1.36.2-jammy /bin/bash
$ docker cp path/to/unzipped/workerd <container_id>:/home/pwuser/workerd

To actually symbolize the stack traces, you'll need llvm-symbolizer installed, but installing it will probably bring in libc++1. 😅 Instead, we can use a script that dumps what would've been passed to llvm-symbolizer to a file, then symbolize it later.

$ more llvm-symbolizer.sh 
#!/bin/bash
cp /dev/stdin "stack-$RANDOM.txt"
$ chmod +x llvm-symbolizer.sh
$ export LLVM_SYMBOLIZER=$PWD/llvm-symbolizer.sh
# Reproduce issue
$ npx wrangler dev ...

Then apt install clang and run $ cat stack-<number>.txt | llvm-symbolizer-14 --pretty-print --relativenames to see the stack trace. Paste that here, and we'll see what we get from it. 👍

@aroman
Copy link
Author

aroman commented Aug 12, 2023

the plot thickens: i've just experienced this segfault with libc++1 being installed, and not in a docker container. I can repro this issue on a fresh install of Ubuntu 22.04, with even the most trivial worker script. It looks like it's a race condition; see below

here is the symbolicated segfault:

[WebServer] ⎔ Starting local server...
[WebServer] *** Received signal #11: Segmentation fault
stack: /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1693d32 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1693c85 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168513d /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1685843 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168be6c /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168e956 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168cfdd /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167fc76 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167f848 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167f624 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167f60c /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326483e /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326443d /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3262728 /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32624ea /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1670293 /lib/x86_64-linux-gnu/libc.so.6@29d8f /lib/x86_64-linux-gnu/libc.so.6@29e3f /__w/magiccircle.gg/magiccircle.gg/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167002d
kj::Table<kj::HashMap<kj::String, kj::Own<workerd::Worker::Isolate::WeakIsolateRef const, std::nullptr_t> >::Entry, kj::HashIndex<kj::HashMap<kj::String, kj::Own<workerd::Worker::Isolate::WeakIsolateRef const, std::nullptr_t> >::Callbacks> >::insert(kj::HashMap<kj::String, kj::Own<workerd::Worker::Isolate::WeakIsolateRef const, std::nullptr_t> >::Entry&&) at ??:0:0

kj::HashMap<kj::String, kj::Own<workerd::Worker::Isolate::WeakIsolateRef const, std::nullptr_t> >::insert(kj::String, kj::Own<workerd::Worker::Isolate::WeakIsolateRef const, std::nullptr_t>) at ??:0:0

workerd::server::Server::InspectorService::registerIsolate(kj::StringPtr, workerd::Worker::Isolate*) at ??:0:0

workerd::server::Server::makeWorker(kj::StringPtr, workerd::server::config::Worker::Reader, capnp::List<workerd::server::config::Extension, (capnp::Kind)3>::Reader) at ??:0:0

workerd::server::Server::makeService(workerd::server::config::Service::Reader, kj::HttpHeaderTable::Builder&, capnp::List<workerd::server::config::Extension, (capnp::Kind)3>::Reader) at ??:0:0

workerd::server::Server::startServices(workerd::jsg::V8System&, workerd::server::config::Config::Reader, kj::HttpHeaderTable::Builder&, kj::ForkedPromise<void>&) at ??:0:0

workerd::server::Server::run(workerd::jsg::V8System&, workerd::server::config::Config::Reader, kj::Promise<void>) at ??:0:0

workerd::server::CliMain::serve()::'lambda'(workerd::jsg::V8System&, workerd::server::config::Config::Reader)::operator()(workerd::jsg::V8System&, workerd::server::config::Config::Reader) const at ??:0:0

void workerd::server::CliMain::serveImpl<workerd::server::CliMain::serve()::'lambda'(workerd::jsg::V8System&, workerd::server::config::Config::Reader)>(workerd::server::CliMain::serve()::'lambda'(workerd::jsg::V8System&, workerd::server::config::Config::Reader)&&) at ??:0:0

workerd::server::CliMain::serve() at ??:0:0

kj::Function<kj::MainBuilder::Validity ()>::Impl<auto workerd::server::cliMethod<kj::_::BoundMethod<workerd::server::CliMain&, workerd::server::CliMain::addServeOptions(kj::MainBuilder&)::'lambda5'(auto&, auto&&...), workerd::server::CliMain::addServeOptions(kj::MainBuilder&)::'lambda6'(auto&, auto&&...)> >(auto&&)::'lambda'(auto&&...)>::operator()() at ??:0:0

kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) at ??:0:0

kj::MainBuilder::MainImpl::operator()(kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>) at ??:0:0

kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0>(kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**)::$_0&&) at ??:0:0

kj::runMainAndExit(kj::ProcessContext&, kj::Function<void (kj::StringPtr, kj::ArrayPtr<kj::StringPtr const>)>&&, int, char**) at ??:0:0

main at ??:0:0

?? at ??:0:0

__libc_start_main at ??:0:0

_start at ??:0:0


[WebServer] ✘ [ERROR] MiniflareCoreError [ERR_RUNTIME_FAILURE]: The Workers runtime failed to start. There is likely additional logging output above.

@aroman
Copy link
Author

aroman commented Aug 12, 2023

ok, even more digging, with repro steps

  1. I can reliably repro this bug on even the most trivial worker script; see below.

  2. this crash is definitely non-deterministic, but I can consistently reproduce it under certain circumstances; I've found the most reliable way to reproduce it is by spawning multiple servers quickly, like this:

repro-segfault.sh

#!/bin/bash

for i in {1..10}
do
  LLVM_SYMBOLIZER=$(which llvm-symbolizer) wrangler dev --port=$((9100+i)) &
done
sleep 10s
num_alive=$(pgrep "workerd" | wc -l)
echo "num_alive: $num_alive"
pkill -f "node_modules"

wrangler.toml

name = ""
main = "./index.js"
compatibility_date = "2022-05-03"

index.js

export default {
  fetch() {
    return new Response('Hello worker!', {
        headers: {
          'content-type': 'text/plain',
        },
    });
  },
};

If you run that on macOS, all 10 dev servers will successfully boot. If you run that on ubuntu 22.04, 1-9 of them will fail to boot. As a workaround, I'm running wrangler with --remote, which does not exhibit the segfault.

  1. Given the above, my guess is there is some sort of race condition which may be agitated by various factors, such as CPU load.

@mrbbot
Copy link
Contributor

mrbbot commented Aug 26, 2023

Hey! 👋 Apologies for the delayed follow-up, I've forwarded this to the runtime team. We also hit this error recently when trying to upgrade miniflare in wrangler: https://github.com/cloudflare/workers-sdk/actions/runs/5977688436/job/16219115403#step:7:196. Another user on Discord has also reported a similar issue. 👍

@ericmatthys
Copy link

I'm observing signal 11: segmentation fault errors intermittently running wrangler pages dev on GitHub Actions ubuntu runners. It happens frequently, but not every time.

workerd/util/symbolizer.c++:99: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
  *** Received signal #11: Segmentation fault
  stack: /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@16966b2 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1696605 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1686f8d /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1687693 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168dcbc /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1691266 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168f8ed /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@16807e6 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@16803b8 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1680194 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@168017c /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326b86e /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326b46d /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3269758 /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@326951a /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1671293 /lib/x86_64-linux-gnu/libc.so.6@29d8f /lib/x86_64-linux-gnu/libc.so.6@29e3f /home/runner/work/example/example/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@167102d

ohodson added a commit to cloudflare/workerd that referenced this issue Aug 29, 2023
Fixes a race between initializing the InspectorService::isolates field
on the inspector service thread and
Server::InspectorServiceIsolateRegistrar::registerIsolate inserting an
isolate into the InspectorSerivce::isolates field on a worker thread.

This change ensures that the InspectorService instance is initialized before
worker threads can start registering isolates.

Test: manual check repro steps in
      cloudflare/miniflare#651
      no longer cause crashes.

Bug: #1075
Bug: cloudflare/miniflare#651
Bug: EW-7716
ohodson added a commit to cloudflare/workerd that referenced this issue Aug 29, 2023
Fixes a race between initializing the InspectorService::isolates field
on the inspector service thread and
Server::InspectorServiceIsolateRegistrar::registerIsolate inserting an
isolate into the InspectorSerivce::isolates field on a worker thread.

This change ensures that the InspectorService instance is initialized before
worker threads can start registering isolates.

Test: manual check repro steps in cloudflare/miniflare#651 no longer repro.

Bug: #1075
Bug: cloudflare/miniflare#651
Bug: EW-7716
ohodson added a commit to cloudflare/workerd that referenced this issue Aug 30, 2023
Fixes a race between initializing the InspectorService::isolates field
on the inspector service thread and
Server::InspectorServiceIsolateRegistrar::registerIsolate inserting an
isolate into the InspectorSerivce::isolates field on a worker thread.

This change ensures that the InspectorService instance is initialized before
worker threads can start registering isolates.

Test: manual check repro steps in cloudflare/miniflare#651 no longer repro.

Bug: #1075
Bug: cloudflare/miniflare#651
Bug: EW-7716
ohodson added a commit to cloudflare/workerd that referenced this issue Aug 30, 2023
Fixes a race between initializing the InspectorService::isolates field
on the inspector service thread and
Server::InspectorServiceIsolateRegistrar::registerIsolate inserting an
isolate into the InspectorSerivce::isolates field on a worker thread.

This change ensures that the InspectorService instance is initialized before
worker threads can start registering isolates.

Test: manual check repro steps in cloudflare/miniflare#651 no longer repro.

Bug: #1075
Bug: cloudflare/miniflare#651
Bug: EW-7716
@mrbbot
Copy link
Contributor

mrbbot commented Sep 8, 2023

Hey! 👋 The fix for this has now been released in [email protected], [email protected], and [email protected]. 👍

@mrbbot mrbbot closed this as completed Sep 8, 2023
@kudlav
Copy link

kudlav commented Sep 13, 2023

Still getting similar error with [email protected], [email protected] and [email protected] when running the following command offline: npx wrangler dev -e local

workerd/util/symbolizer.c++:98: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
workerd/server/server.c++:2518: error: Uncaught exception: kj/async-io-unix.c++:778: disconnected: remote.worker_do_not_log; Request failed due to internal error
stack: /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32d45ff /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32d27ce /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@223fdbb /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32a91e0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32a9402 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3296fb0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32bc800 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1985280 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1e38f81 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1d5809d /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1d55930 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1af8bc0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1af98c9 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@199c8a0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@199ced0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@199dff0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@199e620 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@19e2dad /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1985280 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@222a31b /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@22297b0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@16b6ae0 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1af8e80 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1af9228 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32911ff /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3291593 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@199f96d /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@32911ff /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3292788 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3292fd1 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3299b54 /project/node_modules/@cloudflare/workerd-linux-64/bin/workerd@328f497

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants