mixnode hard restart after tokio-runtime-worker panicked #1123

altogo · 2022-02-19T11:43:27Z

after some times (around 15000 mixed packet) the mixnode panicked and hard restart.

Feb 19 06:39:56 vultr.guest nym-mixnode[13470]: thread 'tokio-runtime-worker' panicked at 'attempt to multiply with overflow', /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/num/mod.rs:804:5
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]: stack backtrace:
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:    0: rust_begin_unwind
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:              at ./rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:    1: core::panicking::panic_fmt
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:              at ./rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:    2: core::panicking::panic
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:              at ./rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:50:5
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:    3: mixnet_client::client::Client::make_connection
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]:    4: mixnet_client::client::Client::send_without_response
Feb 19 06:39:56 vultr.guest nym-mixnode[13470]: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

An example:

mixnode is running perfectly
being in active selection
after some times 'between 6 to 10 hours and 13000 and 17000 packet mixed
mixnode shutdow and restart
mixnode is running with systemd startup.

Which area of Nym were you using?
VPS linux
CPU:2 vCore
RAM:4096 MB
Storage:80 GB SSD

Application: mixnode
OS: debian 11 x64
Version: Nym Mixnode (0.12.1)

Additional context
will run it with backtrace=full to see if i have more info

The text was updated successfully, but these errors were encountered:

altogo · 2022-02-19T19:26:37Z

the log with backtrace=full

Feb 19 19:13:08 vultr.guest nym-mixnode[21793]: thread 'tokio-runtime-worker' panicked at 'attempt to multiply with overflow', /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/num/mod.rs:804:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]: stack backtrace:
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    0:     0x557ba37c598c - std::backtrace_rs::backtrace::libunwind::trace::hf6a6dfd7da937cb0
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    1:     0x557ba37c598c - std::backtrace_rs::backtrace::trace_unsynchronized::hc596a19e4891f7f3
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    2:     0x557ba37c598c - std::sys_common::backtrace::_print_fmt::hb16700db31584325
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:67:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    3:     0x557ba37c598c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h231c4190cfa75162
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:46:22
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    4:     0x557ba37e807c - core::fmt::write::h2a1462b5f8eea807
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/fmt/mod.rs:1163:17
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    5:     0x557ba37be6f5 - std::io::Write::write_fmt::h71ddfebc68685972
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/io/mod.rs:1696:15
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    6:     0x557ba37c73b0 - std::sys_common::backtrace::_print::hcc197d4bebf2b369
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:49:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    7:     0x557ba37c73b0 - std::sys_common::backtrace::print::h335a66af06738c7c
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:36:9
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    8:     0x557ba37c73b0 - std::panicking::default_hook::{{closure}}::h6fac9ac9c8b79e52
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:210:50
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:    9:     0x557ba37c6f65 - std::panicking::default_hook::h341c1030c6a1161b
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:227:9
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   10:     0x557ba37c7a64 - std::panicking::rust_panic_with_hook::h50680ff4b44510c6
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:624:17
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   11:     0x557ba37c7512 - std::panicking::begin_panic_handler::{{closure}}::h9371c0fbb1e8465a
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:519:13
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   12:     0x557ba37c5e34 - std::sys_common::backtrace::__rust_end_short_backtrace::h9b3efa22a5768c0f
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:139:18
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   13:     0x557ba37c74a9 - rust_begin_unwind
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   14:     0x557ba34751e1 - core::panicking::panic_fmt::h23b9203e89cc61cf
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   15:     0x557ba347512d - core::panicking::panic::h0ba7146865b2f9d6
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:                                at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:50:5
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   16:     0x557ba35088c6 - mixnet_client::client::Client::make_connection::h2a307b185800afa7
Feb 19 19:13:08 vultr.guest nym-mixnode[21793]:   17:     0x557ba3508c65 - mixnet_client::client::Client::send_without_response::h795d09f55367bcec

securixhk · 2022-03-20T14:34:38Z

Same here and couldn't figure out the exact root cause.

As mentioned by @falcon-light : it's a real concern due to the potential impact on the popularity.

The same investigation results than mentioned by @altogo : every ~6 hours, every ~10-15K mixed packets, etc.

I've crossed over the hardware/network monitoring data with the exact failures timestamp : nothing relevant can be highlighted (hardware resources are fine, etc.).

Thanks

mfahampshire · 2022-03-21T13:41:49Z

@altogo @falcon-light @securixhk just wanted to drop in and say (1) that this issue is in the backlog and will be gotten to, but also (2) clarify something in here:

The number of packets mixed as seen in the network explorer is not matched to the amount of reputation that the node has in any formal way. This information is self-reported, so isn't used as a measure of a node's reputation or popularity when the active / inactive sets are being chosen.

As such, that your mixnode resets to 0 packets mixed won't have a formal influence on whether or not the node is selected to be included in the active set in each epoch.

tommyv1987 · 2022-03-21T14:44:41Z

Hi all, @mfahampshire - got in here before me.

Thank you for raising the issue here, as Max mentioned it's in the backlog

@securixhk - via automation using systemd as specified Here

@securixhk - what is your hardware specification - are you using Ubuntu (our binaries are built against Ubuntu 20.04)?

To add RUST_BACKTRACE=1 you will need to set in your nym-mixnode service file:

[Unit]
Description=Nym Mixnode (0.12.1)
StartLimitInterval=350
StartLimitBurst=10

[Service]
User=nym
LimitNOFILE=65536
Environment="RUST_BACKTRACE=1"
ExecStart=/home/nym/nym-mixnode run --id mix0100
KillSignal=SIGINT
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

securixhk · 2022-03-21T16:53:28Z

Thanks a lot @mfahampshire for these relieving details and confirming that it's already in the backlog, on your end :).

Thanks @tommyv1987 for the input. I've adjusted the service file : will drop here the backtrace results, later.

Hardware specifications :

Ubuntu 20.04 (LTS) x64 (updated)
2 vCPUs
4GB Memory
SSD : space quite unused (only OS and Nym :) );
Using the prebuilt binaries provided by Nym.

Thanks.

securixhk · 2022-03-23T17:20:01Z

Hey @tommyv1987,

For your information, here are the backtrace outputs :

- Backtrace 1 :

Mar 22 04:14:07 nym nym-mixnode[69412]: thread 'tokio-runtime-worker' panicked at 'attempt to multiply with overflow', /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/num/mod.rs:789:5
Mar 22 04:14:07 nym nym-mixnode[69412]: stack backtrace:
Mar 22 04:14:07 nym nym-mixnode[69412]: 0: rust_begin_unwind
Mar 22 04:14:07 nym nym-mixnode[69412]: at ./rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
Mar 22 04:14:07 nym nym-mixnode[69412]: 1: core::panicking::panic_fmt
Mar 22 04:14:07 nym nym-mixnode[69412]: at ./rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
Mar 22 04:14:07 nym nym-mixnode[69412]: 2: core::panicking::panic
Mar 22 04:14:07 nym nym-mixnode[69412]: at ./rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:50:5
Mar 22 04:14:07 nym nym-mixnode[69412]: 3: mixnet_client::client::Client::make_connection
Mar 22 04:14:07 nym nym-mixnode[69412]: 4: mixnet_client::client::Client::send_without_response
Mar 22 04:14:07 nym nym-mixnode[69412]: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

- Backtrace full :

Mar 23 17:41:58 nym nym-mixnode[79264]: thread 'tokio-runtime-worker' panicked at 'attempt to multiply with overflow', /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/num/mod.rs:789:5
Mar 23 17:41:58 nym nym-mixnode[79264]: stack backtrace:
Mar 23 17:41:58 nym nym-mixnode[79264]: 0: 0x55845c0c3e5c - std::backtrace_rs::backtrace::libunwind::trace::h3fea1eb2e0ba2ac9
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
Mar 23 17:41:58 nym nym-mixnode[79264]: 1: 0x55845c0c3e5c - std::backtrace_rs::backtrace::trace_unsynchronized::h849d83492cbc0d59
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Mar 23 17:41:58 nym nym-mixnode[79264]: 2: 0x55845c0c3e5c - std::sys_common::backtrace::_print_fmt::he3179d37290f23d3
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:67:5
Mar 23 17:41:58 nym nym-mixnode[79264]: 3: 0x55845c0c3e5c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h140f6925cad14324
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:46:22
Mar 23 17:41:58 nym nym-mixnode[79264]: 4: 0x55845c0e6c8c - core::fmt::write::h31b9cd1bedd7ea38
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/fmt/mod.rs:1150:17
Mar 23 17:41:58 nym nym-mixnode[79264]: 5: 0x55845c0bd395 - std::io::Write::write_fmt::h1fdf66f83f70913e
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1667:15
Mar 23 17:41:58 nym nym-mixnode[79264]: 6: 0x55845c0c5870 - std::sys_common::backtrace::_print::he7ac492cd19c3189
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:49:5
Mar 23 17:41:58 nym nym-mixnode[79264]: 7: 0x55845c0c5870 - std::sys_common::backtrace::print::hba20f8920229d8e8
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:36:9
Mar 23 17:41:58 nym nym-mixnode[79264]: 8: 0x55845c0c5870 - std::panicking::default_hook::{{closure}}::h714d63979ae18678
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:210:50
Mar 23 17:41:58 nym nym-mixnode[79264]: 9: 0x55845c0c5427 - std::panicking::default_hook::hf1afb64e69563ca8
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:227:9
Mar 23 17:41:58 nym nym-mixnode[79264]: 10: 0x55845c0c5f24 - std::panicking::rust_panic_with_hook::h02231a501e274a13
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:624:17
Mar 23 17:41:58 nym nym-mixnode[79264]: 11: 0x55845c0c59d2 - std::panicking::begin_panic_handler::{{closure}}::h5b5e738edf361af7
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:519:13
Mar 23 17:41:58 nym nym-mixnode[79264]: 12: 0x55845c0c4324 - std::sys_common::backtrace::__rust_end_short_backtrace::h601a115def7987b1
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:141:18
Mar 23 17:41:58 nym nym-mixnode[79264]: 13: 0x55845c0c5969 - rust_begin_unwind
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
Mar 23 17:41:58 nym nym-mixnode[79264]: 14: 0x55845bd72421 - core::panicking::panic_fmt::h7a58c8fffc5559a4
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
Mar 23 17:41:58 nym nym-mixnode[79264]: 15: 0x55845bd7236d - core::panicking::panic::hcb81dec512b2768f
Mar 23 17:41:58 nym nym-mixnode[79264]: at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:50:5
Mar 23 17:41:58 nym nym-mixnode[79264]: 16: 0x55845be07c81 - mixnet_client::client::Client::make_connection::h4a02efc6746721bd
Mar 23 17:41:58 nym nym-mixnode[79264]: 17: 0x55845be08020 - mixnet_client::client::Client::send_without_response::h10c9a969883b3ca1
Mar 23 17:41:58 nym systemd[1]: nym-mixnode.service: Main process exited, code=killed, status=6/ABRT
Mar 23 17:41:58 nym systemd[1]: nym-mixnode.service: Failed with result 'signal'.
Mar 23 17:42:28 nym systemd[1]: nym-mixnode.service: Scheduled restart job, restart counter is at 1.
Mar 23 17:42:28 nym systemd[1]: Stopped Nym Mixnode (0.12.1).

Thanks.

miscab · 2022-05-06T09:13:32Z

Got the same issue for nym-mixnode of version 1.0.1 on ubuntu 20.04.

May 06 16:24:49 reer198 nym-mixnode[566543]: thread 'tokio-runtime-worker' panicked at 'attempt to multiply with overflow', /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/num/mod.rs:827:5
May 06 16:24:49 reer198 nym-mixnode[566543]: note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

miscab · 2022-05-07T06:34:03Z

This problem repeated always, as a result, hurting the rewards of my machine cuz the uptime is interrupted. This can be observed from more than one nodes. Please check and fix it soon otherwise, it's pretty depressing.

tommyv1987 · 2022-05-08T07:33:05Z

Duplicate ticket raised here—#1259.

tommyv1987 · 2022-05-08T07:33:26Z

Changed status to in progress on dev board.

vlasinus · 2022-05-08T14:58:46Z

restart every ~25k mixed packets

os: Ubuntu 22.04 LTS
cpu: Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz
mem: 98mb
Identity Key: 3ERjVpCiwKdYfk9fyumfQhy291dxnuXncu4Qn7VkVH7M
mix-node: v 1.0.1

thread 'tokio-runtime-worker' panicked at 'attempt to multiply with overflow', /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/num/mod.rs:827:5
stack backtrace:
0: 0x55f6ada9d7ad - std::backtrace_rs::backtrace::libunwind::trace::hee598835bc88d35b
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x55f6ada9d7ad - std::backtrace_rs::backtrace::trace_unsynchronized::h9cdc730ba5cf5d72
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x55f6ada9d7ad - std::sys_common::backtrace::_print_fmt::h75aeaf7ed30e43fa
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:66:5
3: 0x55f6ada9d7ad - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h606862f787600875
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:45:22
4: 0x55f6adac28bc - core::fmt::write::he803f0f418caf762
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/fmt/mod.rs:1190:17
5: 0x55f6ada963f8 - std::io::Write::write_fmt::h70bc45872f37e7bb
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/io/mod.rs:1657:15
6: 0x55f6ada9fa27 - std::sys_common::backtrace::_print::h64d038cf8ac3e13e
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:48:5
7: 0x55f6ada9fa27 - std::sys_common::backtrace::print::h359300b4a7fccf65
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:35:9
8: 0x55f6ada9fa27 - std::panicking::default_hook::{{closure}}::hf51be35e2f510149
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:295:22
9: 0x55f6ada9f6f0 - std::panicking::default_hook::h03ca0f22e1d2d25e
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:314:9
10: 0x55f6adaa0179 - std::panicking::rust_panic_with_hook::h3b7380e99b825b63
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:698:17
11: 0x55f6ada9fe29 - std::panicking::begin_panic_handler::{{closure}}::h8e849d0710154ce0
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:586:13
12: 0x55f6ada9dc74 - std::sys_common::backtrace::__rust_end_short_backtrace::hedcdaddbd4c46cc5
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:138:18
13: 0x55f6ada9fb79 - rust_begin_unwind
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
14: 0x55f6ad7539c3 - core::panicking::panic_fmt::he1bbc7336d49a357
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
15: 0x55f6ad75388d - core::panicking::panic::h4241c5ccea17faca
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:48:5
16: 0x55f6ad7ea162 - mixnet_client::client::Client::make_connection::h4226b710190d38f7
17: 0x55f6ad7ea4f9 - <mixnet_client::client::Client as mixnet_client::client::SendWithoutResponse>::send_without_response::hce47c089868a0e7c
nym-mixnode.service: Main process exited, code=killed, status=6/ABRT
nym-mixnode.service: Failed with result 'signal'.
nym-mixnode.service: Consumed 1min 3.400s CPU time.
nym-mixnode.service: Scheduled restart job, restart counter is at 1.
Stopped Nym Mixnode.
nym-mixnode.service: Consumed 1min 3.400s CPU time.

tommyv1987 · 2022-05-09T11:19:34Z

Hi all,

We now have a fix in place. I will update this thread once the binary has been published.

Thanks.

securixhk · 2022-05-09T11:37:15Z

Thanks a lot for this great update :) !

miscab · 2022-05-09T11:59:33Z

Hi all,

We now have a fix in place. I will update this thread once the binary has been published.

Thanks.

Hi, tommyv, may I know what time you set for this update publish? thanks.

tommyv1987 · 2022-05-09T14:27:35Z

Hi @miscab

Not to sure on the binary published date, but it will happen shortly.

However, if you're in urgent need to change the binary now.

Here is what I recommend.

Follow the instructions on this page:
https://nymtech.net/docs/next/run-nym-nodes/build-nym

However, once you've successfully cloned the repository - checkout this commit - 0f6f47c

git checkout 0f6f47c5ac48cfff614893fadf00b7f35bdbfb4e

Now in the root of the directory you can run ->
NETWORK=mainnet cargo build --release

This will produce your binary in the /target/release directory. If you've built the binary on your local machine I would suggest SCP the binary onto your machine:-

Check the binary has compiled correctly by running ./nym-mixnode --help which should render the following:

~/Documents/github/nym/target/release$ ./nym-mixnode --help


      _ __  _   _ _ __ ___
     | '_ \| | | | '_ \ _ \
     | | | | |_| | | | | | |
     |_| |_|\__, |_| |_| |_|
            |___/

             (mixnode - version 1.0.1)

    
nym-mixnode 1.0.1
Nymtech

Build Timestamp:    2022-05-09T14:23:25.796656424+00:00
Build Version:      1.0.1
Commit SHA:         0f6f47c5ac48cfff614893fadf00b7f35bdbfb4e
Commit Date:        2022-05-09T12:23:14+00:00
Commit Branch:      develop
rustc Version:      1.60.0
rustc Channel:      stable
cargo Profile:      release
Network:            Mainnet

scp nym-mixnode youruser@youraddress:/path/to_your_executable if you have a service file, make sure you change the permissions accordingly so it can be executed.

If you've built the binary on your server, the same process applies as above, minus the the copying of the binary. This should get you going with the latest binary in the interim.

If you have any other questions don't hesitate to ask.

Thanks.

vlasinus · 2022-05-09T18:11:08Z

Hi @miscab

Not to sure on the binary published date, but it will happen shortly.

However, if you're in urgent need to change the binary now.

Here is what I recommend.

Follow the instructions on this page: https://nymtech.net/docs/next/run-nym-nodes/build-nym

However, once you've successfully cloned the repository - checkout this commit - 0f6f47c

git checkout 0f6f47c5ac48cfff614893fadf00b7f35bdbfb4e

Now in the root of the directory you can run -> NETWORK=mainnet cargo build --release

This will produce your binary in the /target/release directory. If you've built the binary on your local machine I would suggest SCP the binary onto your machine:-

Check the binary has compiled correctly by running ./nym-mixnode --help which should render the following:
~/Documents/github/nym/target/release$ ./nym-mixnode --help


      _ __  _   _ _ __ ___
     | '_ \| | | | '_ \ _ \
     | | | | |_| | | | | | |
     |_| |_|\__, |_| |_| |_|
            |___/

             (mixnode - version 1.0.1)

    
nym-mixnode 1.0.1
Nymtech

Build Timestamp:    2022-05-09T14:23:25.796656424+00:00
Build Version:      1.0.1
Commit SHA:         0f6f47c5ac48cfff614893fadf00b7f35bdbfb4e
Commit Date:        2022-05-09T12:23:14+00:00
Commit Branch:      develop
rustc Version:      1.60.0
rustc Channel:      stable
cargo Profile:      release
Network:            Mainnet
scp nym-mixnode youruser@youraddress:/path/to_your_executable if you have a service file, make sure you change the permissions accordingly so it can be executed.

If you've built the binary on your server, the same process applies as above, minus the the copying of the binary. This should get you going with the latest binary in the interim.

If you have any other questions don't hesitate to ask.

Thanks.

Installed this build. Problem is solved. Thx.

miscab · 2022-05-10T10:51:50Z

Hi @miscab

Not to sure on the binary published date, but it will happen shortly.

However, if you're in urgent need to change the binary now.

Here is what I recommend.

Follow the instructions on this page: https://nymtech.net/docs/next/run-nym-nodes/build-nym

However, once you've successfully cloned the repository - checkout this commit - 0f6f47c

git checkout 0f6f47c5ac48cfff614893fadf00b7f35bdbfb4e

Now in the root of the directory you can run -> NETWORK=mainnet cargo build --release

This will produce your binary in the /target/release directory. If you've built the binary on your local machine I would suggest SCP the binary onto your machine:-

Check the binary has compiled correctly by running ./nym-mixnode --help which should render the following:
~/Documents/github/nym/target/release$ ./nym-mixnode --help


      _ __  _   _ _ __ ___
     | '_ \| | | | '_ \ _ \
     | | | | |_| | | | | | |
     |_| |_|\__, |_| |_| |_|
            |___/

             (mixnode - version 1.0.1)

    
nym-mixnode 1.0.1
Nymtech

Build Timestamp:    2022-05-09T14:23:25.796656424+00:00
Build Version:      1.0.1
Commit SHA:         0f6f47c5ac48cfff614893fadf00b7f35bdbfb4e
Commit Date:        2022-05-09T12:23:14+00:00
Commit Branch:      develop
rustc Version:      1.60.0
rustc Channel:      stable
cargo Profile:      release
Network:            Mainnet
scp nym-mixnode youruser@youraddress:/path/to_your_executable if you have a service file, make sure you change the permissions accordingly so it can be executed.

If you've built the binary on your server, the same process applies as above, minus the the copying of the binary. This should get you going with the latest binary in the interim.

If you have any other questions don't hesitate to ask.

Thanks.

That's an amazing solution, thanks for your wonderful work!!!

altogo added bug Something isn't working bug-needs-triage A bug that needs discussing and triage qa Quality Assurance labels Feb 19, 2022

altogo assigned tommyv1987 Feb 19, 2022

tommyv1987 mentioned this issue May 8, 2022

[Issue] mixnode crashed with an error #1259

Closed

jstuczyn self-assigned this May 9, 2022

jstuczyn mentioned this issue May 9, 2022

Fixed overflow in determining reconnection backoff #1260

Merged

1 task

jstuczyn closed this as completed in #1260 May 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mixnode hard restart after tokio-runtime-worker panicked #1123

mixnode hard restart after tokio-runtime-worker panicked #1123

altogo commented Feb 19, 2022 •

edited

Loading

altogo commented Feb 19, 2022 •

edited

Loading

securixhk commented Mar 20, 2022

mfahampshire commented Mar 21, 2022

tommyv1987 commented Mar 21, 2022 •

edited

Loading

securixhk commented Mar 21, 2022

securixhk commented Mar 23, 2022

miscab commented May 6, 2022

miscab commented May 7, 2022

tommyv1987 commented May 8, 2022

tommyv1987 commented May 8, 2022

vlasinus commented May 8, 2022

tommyv1987 commented May 9, 2022

securixhk commented May 9, 2022

miscab commented May 9, 2022 •

edited

Loading

tommyv1987 commented May 9, 2022 •

edited

Loading

vlasinus commented May 9, 2022

miscab commented May 10, 2022

mixnode hard restart after tokio-runtime-worker panicked #1123

mixnode hard restart after tokio-runtime-worker panicked #1123

Comments

altogo commented Feb 19, 2022 • edited Loading

altogo commented Feb 19, 2022 • edited Loading

securixhk commented Mar 20, 2022

mfahampshire commented Mar 21, 2022

tommyv1987 commented Mar 21, 2022 • edited Loading

securixhk commented Mar 21, 2022

securixhk commented Mar 23, 2022

miscab commented May 6, 2022

miscab commented May 7, 2022

tommyv1987 commented May 8, 2022

tommyv1987 commented May 8, 2022

vlasinus commented May 8, 2022

os: Ubuntu 22.04 LTS cpu: Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz mem: 98mb Identity Key: 3ERjVpCiwKdYfk9fyumfQhy291dxnuXncu4Qn7VkVH7M mix-node: v 1.0.1

tommyv1987 commented May 9, 2022

securixhk commented May 9, 2022

miscab commented May 9, 2022 • edited Loading

tommyv1987 commented May 9, 2022 • edited Loading

vlasinus commented May 9, 2022

miscab commented May 10, 2022

altogo commented Feb 19, 2022 •

edited

Loading

altogo commented Feb 19, 2022 •

edited

Loading

tommyv1987 commented Mar 21, 2022 •

edited

Loading

os: Ubuntu 22.04 LTS
cpu: Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz
mem: 98mb
Identity Key: 3ERjVpCiwKdYfk9fyumfQhy291dxnuXncu4Qn7VkVH7M
mix-node: v 1.0.1

miscab commented May 9, 2022 •

edited

Loading

tommyv1987 commented May 9, 2022 •

edited

Loading