Bit lint throws FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #4628

dstoyanoff · 2021-07-26T05:26:08Z

Describe the bug

When running the lint command, I am always getting the following error. It's worth mentioning that I have seen this with prior version of bit as well, so it's now something new:

bit lint

teambit.defender/eslint, linting components (32/121). spodio-bits.bits/atoms/icons/[email protected]
<--- Last few GCs --->

[27927:0x118008000]   200285 ms: Scavenge 4085.0 (4113.2) -> 4083.9 (4114.2) MB, 11.0 / 0.0 ms  (average mu = 0.377, current mu = 0.433) allocation failure 
[27927:0x118008000]   207197 ms: Mark-sweep (reduce) 4090.8 (4114.2) -> 4085.4 (4117.5) MB, 6881.7 / 0.0 ms  (+ 0.1 ms in 109 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 6912 ms) (average mu = 0.180, current mu = 0.0

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10130d6e5 node::Abort() (.cold.1) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 2: 0x1000b1c49 node::Abort() [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 3: 0x1000b1daf node::OnFatalError(char const*, char const*) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 4: 0x1001f60f7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 5: 0x1001f6093 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 6: 0x1003a54f5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 7: 0x1003a6fba v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 8: 0x1003a2689 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
 9: 0x10039ff21 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
10: 0x1003ae7da v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
11: 0x1003ae861 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
12: 0x10037bf12 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
13: 0x1006fc158 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
14: 0x100a893d9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/danielstoyanoff/.nvm/versions/node/v14.17.0/bin/node]
15: 0x3a212bbef06b 
[1]    27927 abort      bit lint

Specifications

Bit version: 0.0.467
Workspace type: harmony
Node version: 14/16
yarn version: 1.22.10
Platform: darwin (M1)

for harmony workspace

relevant env: react

Additional context

The text was updated successfully, but these errors were encountered:

GiladShoham · 2021-07-26T05:28:04Z

Thanks. I'll check it. Any steps to reproduce?

dstoyanoff · 2021-07-26T05:34:04Z

I am unfortunately not sure how I got here, since I tried to use the lint rather late. Consider the fact that I have 121 components currently, not sure if that could be related.

I am also using an M1 MBP, which is might also be related. Not sure what is the difference in linting through bit as I don't have this issue with eslint directly.

I am going to do some more experimenting and share the info here.
Here is a log from Node 16, as it seems to have some difference:

bit lint

``` teambit.defender/eslint, linting components (32/121). spodio-bits.bits/atoms/icons/[email protected] <--- Last few GCs --->

[30907:0x150008000] 181004 ms: Scavenge 4026.1 (4123.3) -> 4016.2 (4125.3) MB, 9.5 / 0.0 ms (average mu = 0.288, current mu = 0.302) allocation failure
[30907:0x150008000] 181040 ms: Scavenge 4028.0 (4125.3) -> 4017.4 (4129.0) MB, 11.4 / 0.0 ms (average mu = 0.288, current mu = 0.302) allocation failure
[30907:0x150008000] 184116 ms: Mark-sweep 4031.5 (4129.0) -> 4016.0 (4129.2) MB, 3017.4 / 0.0 ms (average mu = 0.217, current mu = 0.137) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x100854adc node::Abort() [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
2: 0x100854c68 node::errors::TryCatchScope::~TryCatchScope() [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
3: 0x10099f82c v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
4: 0x10099f7c0 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
5: 0x100b640f0 v8::internal::Heap::NotifyContextDisposed(bool) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
6: 0x100b669e8 v8::internal::Heap::HasLowYoungGenerationAllocationRate() [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
7: 0x100b63c30 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
8: 0x100b61688 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
9: 0x100b6d2ac v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
10: 0x100b6d33c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
11: 0x100b41860 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
12: 0x100e6f91c v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
13: 0x10119714c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
14: 0x420397780c4
15: 0x4203954bc4c
16: 0x4203954d674
17: 0x420398182d8
18: 0x42039757b00
19: 0x4203978ebac
20: 0x42039790724
21: 0x42039792fd8
22: 0x4203984bb40
23: 0x420396b9188
24: 0x420396f7618
25: 0x42038f5218c
26: 0x42038ccdb58
27: 0x42038f52240
28: 0x42038ccdb58
29: 0x42038f52240
30: 0x42038ccdb58
31: 0x42038f52240
32: 0x42038ccdb58
33: 0x42038f52240
34: 0x42038ccdb58
35: 0x42038f52240
36: 0x42038ccdb58
37: 0x42038f52240
38: 0x42038ccdb58
39: 0x42038f52240
40: 0x42038ccdb58
41: 0x42038f52240
42: 0x42038ccdb58
43: 0x42038f52240
44: 0x42038ccdb58
45: 0x42038f52240
46: 0x42038ccdb58
47: 0x42038f52240
48: 0x42038ccdb58
49: 0x42038f52240
50: 0x42038ccdb58
51: 0x42038f52240
52: 0x42038ea534c
53: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
54: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
55: 0x42038cec610
56: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
57: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
58: 0x42038d7ba44
59: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
60: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
61: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
62: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
63: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
64: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
65: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
66: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
67: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
68: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
69: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
70: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
71: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
72: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
73: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
74: 0x1011b22dc Builtins_ArrayMap [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
75: 0x1011306f8 Builtins_InterpreterEntryTrampoline [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
76: 0x42038d23fc0
77: 0x10115d3b4 Builtins_AsyncFunctionAwaitResolveClosure [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
78: 0x1011de4dc Builtins_PromiseFulfillReactionJob [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
79: 0x101150008 Builtins_RunMicrotasks [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
80: 0x10112e3e4 Builtins_JSRunMicrotasksEntry [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
81: 0x100af4468 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
82: 0x100af4978 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
83: 0x100af4a40 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandlev8::internal::Object) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
84: 0x100b16eac v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
85: 0x100b16ce4 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
86: 0x1007a2bac node::InternalCallbackScope::Close() [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
87: 0x1007a318c node::InternalMakeCallback(node::Environment*, v8::Localv8::Object, v8::Localv8::Object, v8::Localv8::Function, int, v8::Localv8::Value, node::async_context) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
88: 0x1007b887c node::AsyncWrap::MakeCallback(v8::Localv8::Function, int, v8::Localv8::Value) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
89: 0x1008589b8 node::fs::FSReqCallback::Reject(v8::Localv8::Value) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
90: 0x100859184 node::fs::FSReqAfterScope::Reject(uv_fs_s*) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
91: 0x100859240 node::fs::AfterNoArgs(uv_fs_s*) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
92: 0x101110a0c uv__work_done [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
93: 0x101114124 uv__async_io [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
94: 0x101125dfc uv__io_poll [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
95: 0x1011145b4 uv_run [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
96: 0x1007a3c20 node::SpinEventLoop(node::Environment*) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
97: 0x10088dd80 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
98: 0x100829a8c node::Start(int, char**) [/Users/danielstoyanoff/.nvm/versions/node/v16.3.0/bin/node]
99: 0x1051110fc
[1] 30907 abort bit lint

</details>

dstoyanoff · 2021-07-26T05:45:05Z

Increasing the memory allocated to NodeJS didn't fix the issue. I was able to lint more components (64, compared to 32), but it still threw.

NODE_OPTIONS="--max-old-space-size=8192" bit lint
⢀⠀ teambit.defender/eslint, linting components (64/121). spodio-bits.bits/atoms/icons/[email protected]
<--- Last few GCs --->

[31330:0x128008000]   457561 ms: Scavenge 8038.1 (8210.1) -> 8033.0 (8216.9) MB, 29.9 / 0.0 ms  (average mu = 0.180, current mu = 0.035) allocation failure 
[31330:0x128008000]   457626 ms: Scavenge 8045.5 (8217.6) -> 8040.6 (8222.4) MB, 21.2 / 0.0 ms  (average mu = 0.180, current mu = 0.035) allocation failure 
[31330:0x128008000]   457687 ms: Scavenge 8050.6 (8222.9) -> 8045.9 (8228.9) MB, 28.9 / 0.0 ms  (average mu = 0.180, current mu = 0.035) allocation failure

dstoyanoff · 2021-07-26T06:04:40Z

Rulling out an issue with the M1 mac as it reproduces in GitHub actions too.
@GiladShoham, let me know if there is anything I can try that comes in your mind. I don't have any other ideas for now. I can optionally give you access to this repo to test it yourself, as I am not sure if I will be easily able to create a repro example.

EtheveDamien · 2021-07-27T12:27:59Z

export NODE_OPTIONS="--max-old-space-size=8192" works for me

dstoyanoff · 2021-07-27T12:33:46Z

@EtheveDamien, how many components do you have?

From what I saw, setting different memory limits results in a different number of successfully linted components. For me the results were:

1GB - 15 components
4GB - 32 components
8GB - 64 components

I am guessing that in your case 8GB were enough to cover all your components and that's why this helps in your case. If you continue adding more components, it should at some point start breaking as well

EtheveDamien · 2021-08-11T12:12:42Z

@dstoyanoff less than 15 components

MikaelColliander · 2021-09-08T10:19:20Z

Linting breaks for me too, I have 90+ components. Also, linting process for one individual component takes a fair bit of time. Maybe linting of all components is not a part of the intended workflow. But how would one go about it to automatically lint only the components that have been touched when stepping through the pipeline?

GiladShoham · 2021-09-09T09:32:52Z

That's a real issue I'm going to tackle soon. not sure exactly when, but I'm aware of this and will take care of this.

itaymendel · 2023-03-06T07:28:33Z

resolved with new implementation of bit lint

dstoyanoff added the type/bug label Jul 26, 2021

GiladShoham self-assigned this Jul 26, 2021

itaymendel closed this as completed Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bit lint throws FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #4628

Bit lint throws FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #4628

dstoyanoff commented Jul 26, 2021 •

edited

Loading

GiladShoham commented Jul 26, 2021

dstoyanoff commented Jul 26, 2021

dstoyanoff commented Jul 26, 2021

dstoyanoff commented Jul 26, 2021

EtheveDamien commented Jul 27, 2021

dstoyanoff commented Jul 27, 2021

EtheveDamien commented Aug 11, 2021

MikaelColliander commented Sep 8, 2021

GiladShoham commented Sep 9, 2021

itaymendel commented Mar 6, 2023

Bit lint throws FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #4628

Bit lint throws FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #4628

Comments

dstoyanoff commented Jul 26, 2021 • edited Loading

Describe the bug

Specifications

Additional context

GiladShoham commented Jul 26, 2021

dstoyanoff commented Jul 26, 2021

dstoyanoff commented Jul 26, 2021

dstoyanoff commented Jul 26, 2021

EtheveDamien commented Jul 27, 2021

dstoyanoff commented Jul 27, 2021

EtheveDamien commented Aug 11, 2021

MikaelColliander commented Sep 8, 2021

GiladShoham commented Sep 9, 2021

itaymendel commented Mar 6, 2023

dstoyanoff commented Jul 26, 2021 •

edited

Loading