Wekan Crash if more than 2GB of RAM is used

[r4nc0r]

Issue

Other Wekan issues can be added here.

Server Setup Information:

• Did you test in newest Wekan?: yes
• For new Wekan install, did you configure root-url correctly so Wekan cards open correctly https://github.com/wekan/wekan/wiki/Settings ?
• Wekan version: 4.96
• Deployment Method: docker swarm
• Http frontend if any (Caddy, Nginx, Apache, see config examples from Wekan GitHub wiki first):
• Node Version:
• MongoDB Version: 4.0.11

Problem description:
if NodeJs uses more than 2GB of Memory, Wekan crashes with a Exception:

I researched a little and as it turns out NodeJs v12 has a standard limit of 2GB Memory Consumption, but it can be changed with the following Nodejs CLI Command: --max-old-space-size xxxx where xxxx is the memory in MB (source) Can this be implemented to raise the Wekan max Memory Limit?

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory,
,
<--- Last few GCs --->,
,
[1:0x38a2e80] 324979 ms: Mark-sweep 2029.2 (2080.6) -> 2018.8 (2083.1) MB, 1608.6 / 0.1 ms (average mu = 0.120, current mu = 0.022) allocation failure scavenge might not succeed,
[1:0x38a2e80] 327768 ms: Mark-sweep 2031.4 (2083.1) -> 2022.9 (2063.4) MB, 2738.9 / 0.1 ms (+ 0.1 ms in 30 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2751 ms) (average mu = 0.061, current mu = 0.018) allocation f,
,
<--- JS stacktrace --->,
,
==== JS stack trace =========================================,
,
0: ExitFrame [pc: 0x140a859],
Security context: 0x22ae44d408d1 ,
1: changeField [0x369877f3a6b9] [/build/programs/server/packages/ddp-server.js:~287] [pc=0x273d069851f8](this=0x36d784d60269 ,0x06e82165b7a9 <String[18]: Ngb63ADH8RE3BT3C8Y>,0x24027c246289 <String[#6]: status>,0x1a40249114f9 <String[6]: online>,0x36d784d603e1 ,0x24027c240639 ),
2: ...,
,
1: 0xa18150 node::Abort() [node],
2: 0xa1855c node::OnFatalError(char const*, char const*) [node],
3: 0xb970be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node],
4: 0xb97439 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node],
5: 0xd546b5 [node],
6: 0xd54d46 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node],
7: 0xd61605 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node],
8: 0xd624b5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node],
9: 0xd64f6c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node],
10: 0xd2b98b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node],
11: 0x106e0bc v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node],
12: 0x140a859 [node],

[xet7]

Related #3580

[r4nc0r]

Do you mean the env var NODE_OPTIONS: --max_old_space_size=4096? If this is enough to fix the problem it would be great :)

[xet7]

@r4nc0r

Please try does it work.

[r4nc0r]

Works :)

[r4nc0r]

Could you add this to the offical wekan documentation?

[jankapunkt]

The real question is why it's using >2GB of RAM.

Are there any unbusted chaches?
Memory leaks?
Too many publictations/observers?
Assets Like Images loaded into RAM?

[xet7]

@jankapunkt

Yes sure. When Wekan is running, CPU usage and RAM usage is increasing all the time, as mentioned in some of these issues.

[xet7]

Also too much data is published to browserside, making browsers crash #3893

[Ben0it-T]

Definitely a stupid question ...
Should we add the env var NODE_OPTIONS: --max_old_space_size=4096 with snap (for prevent Wekan crashes if NodeJs uses more than 2GB of Memory) ?

If yes, where/how ?
On Debian 10, the unit file snap.wekan.wekan.service, we can see
EnvironmentFile=-/etc/environment

Should we add the env var NODE_OPTIONS: --max_old_space_size=4096 in /etc/environment ?

Thanks

[xet7]

@Ben0it-T

There are no stupid questions. There are only my stupid answers.

WeKan Snap does not have that setting option yet.

To try that option, you could use start-wekan.sh from https://github.com/wekan/wekan or some service file, and bundle version from https://github.com/wekan/wekan/wiki/Raspberry-Pi using bundle version for your server CPU architecture.

If adding that option helps, that setting could be added to some Snap settings at https://github.com/wekan/wekan/tree/master/snap-src/bin .

If that option does not help, then you should wait when I have finished partial stupid rewrite of WeKan.

[Ben0it-T]

@xet7
Thank you for your answer

For the moment I have not yet deployed Wekan within our team. I use it for my personal use.
I plan to deploy it in a few weeks for our team. So I have not yet encountered this type of problem...

At first, it could probably be interesting to know the details about the heap and size, with getHeapStatistics().
Maybe in Admin Panel > Version ?

• total_heap_size
• total_heap_size_executable
• total_physical_size
• total_available_size
• used_heap_size
• heap_size_limit
• ...

What do you think about that ?

[xet7]

@Ben0it-T

You can send PR to add them to:

• https://github.com/wekan/wekan/blob/master/server/statistics.js
• https://github.com/wekan/wekan/blob/master/client/components/settings/informationBody.jade
• https://github.com/wekan/wekan/blob/master/client/components/settings/informationBody.js
• https://github.com/wekan/wekan/blob/master/i18n/en.i18n.json

Thanks!

[Ben0it-T]

@xet7
A little feedback on my stupid question about NODE_OPTIONS with snap
On the latest Wekan version (5.75) with snap, if I put NODE_OPTIONS in /etc/environment, it works !

Ex1 :

root@xxxxxxxx:~# cat /etc/environment
# Nodejs
NODE_OPTIONS="--max_old_space_size=2048"
root@xxxxxxxx:~# snap restart wekan

Ex2 :

root@xxxxxxxx:~# cat /etc/environment
# Nodejs
NODE_OPTIONS="--max_old_space_size=4096"
root@xxxxxxxx:~# snap restart wekan

[xet7]

Somebody got this:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[1:0x2d9ee80]  1270397 ms: Mark-sweep 2036.4 (2065.4) -> 2030.9 (2066.9) MB, 1982.8 / 0.2 ms  (average mu = 0.185, current mu = 0.131) allocation failure scavenge might not succeed
[1:0x2d9ee80]  1272372 ms: Mark-sweep 2036.5 (2066.9) -> 2032.0 (2067.9) MB, 1951.8 / 0.2 ms  (average mu = 0.098, current mu = 0.012) allocation failure GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x140dff9]
Security context: 0x1f42ab6808d1 <JSObject>
    1: changeField [0x3899d570e079] [/build/programs/server/packages/ddp-server.js:~287] [pc=0x158bfe4d16b8](this=0x09e2b6008079 <SessionDocumentView map = 0xff51671a929>,0x00191a67ef61 <String[18]: NdHSHs6pNEj8woP53M>,0x2babc7c8c171 <String[#4]: sort>,0x3fc3dcd2d069 <HeapNumber -2>,0x09e2b60081f1 <Object map = 0x292df8f7e0a9>,0x1a2b41b80639 <true>)
    2: added(...

 1: 0xa1a640 node::Abort() [node]
 2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
 3: 0xb9a9fe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9ad79 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd57ff5  [node]
 6: 0xd58686 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd64f45 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd65df5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd688ac v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd2f2cb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x10719fc v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x140dff9  [node]

So does any of these help?

https://stackoverflow.com/questions/53230823/fatal-error-ineffective-mark-compacts-near-heap-limit-allocation-failed-javas

@mfilser

Is there already some setting in WeKan for this, or should it be added?

[xet7]

Some progress:

I did this testing on Debian VM that is running on bare metal server donated to WeKan development. That Debian VM has 60 GB RAM, just in case, it could work with less.

1. So that right click Inspect console would not show Templates related errors, I clicked right top username / Templates, to open Templates board and have those created.
2. One customer gave me example board export, size 4 MB of json.
3. I checked node path:

which node

4. In start-wekan.sh I have added these, with node path, and additional stack size mentioned here https://stackoverflow.com/questions/20936486/node-js-maximum-call-stack-size-exceeded

UPDATE: This works best with size 4096. Bigger values do increase CPU usage too much.

cd .build/bundle
export NODE_OPTIONS="--max_old_space_size=4096"
/bin/bash -c "ulimit -s 65500; exec /home/wekan/.volta/bin/node --stack-size=65500 main.js"

5. At Debian VM, I installed xclip, that copies textfile to clipboard:

sudo apt -y install xclip

6. I copied board JSON to clipboard:

cat board.json | xclip -se c

7. In Chrome, I did: All Boards page / Add Board / Import / From previous export / right click to input field / paste
8. (I did not map any users in my case, because my test WeKan install has only one user. This is optional.)
9. There could be another button about mapping users, without list of users to map. In this step, I checked is there any errors at browser right click Inspect Console. There were no errors. Anyway, I did not click again (sometime it could cause duplicate imported boards), I just clicked All Boards, and opened that imported board.
10. Memory usage in that Debian VM:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            61Gi       3,4Gi        47Gi       135Mi        10Gi        57Gi
Swap:          975Mi          0B       975Mi

11. I don't see any errors in browser right click Inspect console, other than one notice about autofocus that does not matter.