Out of memory

[ian13]

Fresh install using npm, when trying to launch dtd2mysql with any parameters throws out of memory error.

~$ dtd2mysql

<--- Last few GCs --->

 105 ms: Mark-sweep 1.4 (5.5) -> 1.4 (5.5) MB, 3.1 / 0.0 ms [allocation failure] [GC in old space requested].
 108 ms: Mark-sweep 1.4 (5.5) -> 1.4 (5.5) MB, 2.6 / 0.0 ms [allocation failure] [GC in old space requested].
 110 ms: Mark-sweep 1.4 (5.5) -> 1.3 (5.5) MB, 2.1 / 0.0 ms [allocation failure] [GC in old space requested].
 112 ms: Mark-sweep 1.3 (5.5) -> 1.3 (4.5) MB, 2.1 / 0.0 ms [last resort gc].
 114 ms: Mark-sweep 1.3 (4.5) -> 1.3 (4.5) MB, 2.1 / 0.0 ms [last resort gc].

[linusnorton]

Hi, would you be able to tell me the version of node you have and the filename you are trying to import.

[ian13]

Using node v6.11.0 & npm 3.10.10

The same error is encountered with any parameters, e.g.
dtd2mysql
dtd2mysql -v
DATABASE_USERNAME=XXX DATABASE_PASSWORD=XXX DATABASE_NAME=XXX dtd2mysql --fares ./RJFAF607.ZIP

[linusnorton]

Unfortunately it requires node 8 or above to run. Could you upgrade and try again?

[ian13]

Thank you - I've since upgraded node to v8.9.1

This seems to have solved the overt error, however process still dies with:

Extracting ./RJFAF607.ZIP to /tmp/dtd/fares/
Killed

Running with environment variable NODE_DEBUG=dtd2mysql gives the following error:

RangeError: Array buffer allocation failed
at new ArrayBuffer ()
at typedArrayConstructByLength ()
at new Uint8Array (native)
at new FastBuffer (buffer.js:38:5)
at Function.Buffer.alloc (buffer.js:243:10)
at new Buffer (buffer.js:156:19)
at decompress (/usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/zipEntry.js:56:20)
at Object.getData (/usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/zipEntry.js:241:20)
at /usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/adm-zip.js:377:37
at Array.forEach ()
at Object.extractAllTo (/usr/lib/node_modules/dtd2mysql/node_modules/adm-zip/adm-zip.js:372:26)
at ImportFeedCommand.doImport (/usr/lib/node_modules/dtd2mysql/dist/src/cli/ImportFeedCommand.js:53:30)
at ImportFeedCommand.run (/usr/lib/node_modules/dtd2mysql/dist/src/cli/ImportFeedCommand.js:35:24)
at container.getCommand.then.c (/usr/lib/node_modules/dtd2mysql/dist/src/index.js:7:18)
at
at process._tickCallback (internal/process/next_tick.js:188:7)

[linusnorton]

How strange, I will bump the memory and release a new version.

[linusnorton]

I've just released a new version (3.4.1) that will use more memory but that did make me think - how much memory does your system have available?

[ian13]

Upgraded to 3.4.1

System memory is

$ free
total used free shared buffers cached
Mem: 507616 301488 206128 24628 72 42684
-/+ buffers/cache: 258732 248884
Swap: 0 0 0

[linusnorton]

If I'm reading that correctly you have roughly 200~MB of memory free which I don't think will be enough. Is there any chance that you can free up some memory or try on a different machine?

[ian13]

You're quite right - 200MB available isn't sufficient.

Creating a 1 GB swapfile dealt with the issue :)