max_old_space_size too small and not configurable?

[eoghanmurray]

@i've been getting the FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory memory error and traced it to the node max_old_space_size argument.

This appears to be set to 3000 in ./bin/dtd2mysql.sh

I tried increasing to 4096 and then finally to 16384 and that seems to have solved it.

This was from calling dtd2mysql as follows:

$ DATABASE_USERNAME=dtd2mysql DATABASE_NAME=timetable NODE_OPTIONS='--max_old_space_size=16384' /usr/bin/ts-node ./src/index.ts --gtfs-zip ~/gtfs.zip

However, when I run it on the system-installed version, as follows, it seems to still get the error:
$ DATABASE_USERNAME=dtd2mysql DATABASE_NAME=timetable NODE_OPTIONS='--max_old_space_size=16384' dtd2mysql --gtfs-zip ~/gtfs.zip

Which I presume is due to the presence of --max_old_space_size=3000 in /usr/bin/dtd2mysql (just checking that now...)

[linusnorton]

Wow, that's crazy. I only end up using about 1.5GB. Can you give me the operating system, 64bit/32bit, node version and filename?

[eoghanmurray]

update: ya increasing the value that is hardcoded in /usr/lib/node_modules/dtd2mysql/bin/dtd2mysql.sh is able to fix it, so I imagine that indicates that you can't pass extra node options to dtd2mysql?

[eoghanmurray]

It's 64-bit on node v8.4 on a machine with 128GB RAM:

$ lscpu |grep Architecture
Architecture:          x86_64
$ cat /proc/meminfo |grep MemTotal
MemTotal:       131926224 kB
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS```

$ node -v
v8.4.0
$ which node
/usr/bin/node

Here's a series of snapshots from top where it shows the memory maxing out at 7GB:

top - 10:33:16 up 308 days, 21:07,  6 users,  load average: 1.73, 2.41, 4.44
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 31.8 us,  3.9 sy,  0.0 ni, 61.0 id,  3.2 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 13192622+total,  3437664 free,  6650012 used, 12183854+buff/cache
KiB Swap:  4192188 total,  3092496 free,  1099692 used. 10695166+avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                           
 6906 eoghan    20   0 1174472 101748  23000 S   0.0  0.1   0:01.17 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 2768124 1.630g  23000 R  99.3  1.3   0:44.70 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 4035812 2.841g  23000 R 104.7  2.3   1:27.07 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 5070980 3.828g  23000 R 100.7  3.0   2:07.62 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 5671464 4.405g  23000 R  97.0  3.5   2:26.66 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 6920768 5.597g  23000 R  98.3  4.4   2:57.24 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 7837004 6.471g  23000 R 155.3  5.1   3:38.92 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip
 6906 eoghan    20   0 8508128 7.113g  23000 R  97.3  5.7   4:24.66 node --max_old_space_size=8192 /usr/bin/dtd2mysql --gtfs-zip ~/gtfs.zip

(That run succeeded)

[linusnorton]

Which file are DTD/TTIS file are you using to generate the GTFS file?

[eoghanmurray]

It appears to be RJTTF921.ZIP

[linusnorton]

I've run it through locally and not had any issues. Is there any chance you can update your node version?