Cannot allocate memory

[EPiC-APOC]

/usr/local/bin/gravityOptimise.sh: fork: Cannot allocate memory

I guess it's propably because I use a big ass blocklist, about 4 milion entries lol (Number of unique domains trapped in the Event Horizon: 1859646). Did not have this problem before with a list of 3 milion entries (Number of unique domains trapped in the Event Horizon: approx 1250000).

Running latest Pihole on a RPi 2B+ (with up2date Jessy)

[mmotti]

Do you know at exactly which part of the script this happens? I might end up having to rewrite these in Python or try find a better way to handle such vast amounts of processing

[EPiC-APOC]

Not anymore 🙈
I'll reproduce the problem to re-check

[mmotti]

I've just re-wrote the scripts in Python. If you get a chance, can you give it a try to see if it resolves the issue?

[mmotti]

This should be resolved by the latest version so will close for now. Let me know if this is not the case

[EPiC-APOC]

This should be resolved by the latest version so will close for now. Let me know if this is not the case

I get this error with the new python file:

pi@RPiHole:~ $ sudo curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | sudo python3
  File "<stdin>", line 52
    print(f'{path_pihole} was not found')
                                       ^
SyntaxError: invalid syntax

curl with or without sudo doesn't make a difference (should work without, pihole installl also works without sudo for instance. (running up2date Raspbian Stretch v9.11)

[mmotti]

@EPiC-APOC please see mmotti/pihole-regex#16

This is because the script requires Python 3.6+ :-)

[EPiC-APOC]

@EPiC-APOC please see mmotti/pihole-regex#16

This is because the script requires Python 3.6+ :-)

pi@RPiHole:~ $ python --version
Python 2.7.13
pi@RPiHole:~ $ sudo apt-get install python3
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3 is already the newest version (3.5.3-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[mmotti]

Yes, 3.5.3 was probably the last version added for Raspbian Stretch.

If you follow the previous link there is a way to manually upgrade it which a couple of people seem to have had some success with

[EPiC-APOC]

Yes, 3.5.3 was probably the last version added for Raspbian Stretch.

If you follow the previous link there is a way to manually upgrade it which a couple of people seem to have had some success with

Sorry, overlooked that tip, trying this now:

https://gist.github.com/dschep/24aa61672a2092246eaca2824400d37f

[EPiC-APOC]

Success 👍

pi@RPiHole:~ $ curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | sudo python3.6
[i] Root user detected
[i] Pi-hole path exists
[i] Updating gravity (this may take a little time)
[i] Legacy lists detected
[i] Fetching domains
[i] --> 1,308,390 domains found
[i] Scanning /etc/dnsmasq.d for wildcards
[i] --> 1,260 wildcards found
[i] Identifying wildcard conflicts with gravity
[i] --> 175,982 conflicts found
[i] Fetching regexps
[i] --> 57 regexps found
[i] Checking for gravity matches
[i] --> 16,808 matches found in gravity
[i] Outputting updated gravity.list
[i] --> 1,114,341 domains remain in gravity.list
[i] Restarting Pi-hole

Will test with even bigger list and see what happens next 😂👌

[EPiC-APOC]

I Can't reproduce 'cannot allocate memory' (so far), so it seems this is working better with the new methode. Just so you know. Seems to work great, but will try to piss it off ;)

Question, I tried this in cron.d ; Raspbian Stretch (9.11) with Py3.6.5 @ Raspberry Pi 2B+:
(because crontab does not seem to work with Raspbian on Raspberry, at least never got that to work)

30 5 * * * root /usr/bin/curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | /usr/local/bin/python3.6

But than I get this error:

[i] Root user detected
[i] Pi-hole path exists
[i] Updating gravity (this may take a little time)
Traceback (most recent call last):
  File "<stdin>", line 58, in <module>
  File "/usr/local/lib/python3.6/subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'pihole': 'pihole'

When I use this at shell it works though:

/usr/bin/curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | sudo /usr/local/bin/python3.6

Did not have any problems with the previous cron.d:

30 5 * * * root PATH="$PATH:/usr/local/bin/" gravityOptimise.sh

with kind regards

[mmotti]

Crontab seems to work fine on my end for Raspbian... are you making sure to include the shell and path variables in cron? Assuming it works in the same way for cron.d Edit: It did take me a little while initially to get this working with cron before I updated the readme, but it turned out to be an issue with paths. If I recall, it was always around the subprocess call where things failed. Adding the following variables to the top of the cron job seemed to fix it for me. ``` SHELL=/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin ``` I'm fairly new to all things cron but I can only assume that things must operate a little differently when Python calls a script / executable, as opposed to calling a subshell or something from bash. The added complication is that you could append `sudo` to specific commands in bash whereas Python doesn't seem to have this ability. Not sure whether this may contribute to the path issues.

[EPiC-APOC]

tried this to test:

env -i /bin/bash --noprofile --norc

/usr/bin/curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | sudo /usr/local/bin/python3.6

(without sudo I get: Please run this script as root)

[i] Root user detected
[i] Pi-hole path exists
[i] Updating gravity (this may take a little time)
[i] Legacy lists detected
[i] Fetching domains
[i] --> 1465817 domains found
[i] Scanning /etc/dnsmasq.d for wildcards
[i] --> 1266 wildcards found
[i] Identifying wildcard conflicts with gravity
Traceback (most recent call last):
  File "<stdin>", line 144, in <module>
  File "/usr/local/lib/python3.6/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\u0455' in position 3: ordinal not in range(128)

if I just put this in crontab it just does not seem tor start:

SHELL=/bin/bash 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
06 21 * * * /usr/bin/curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | sudo /usr/local/bin/python3.6

[mmotti]

Interesting. Does this error not occur when just running normally without the env command? Might be worth creating a new issue so we can get to the bottom of this in it's entirety We don't want to use sudo within the crontab as I believe this stores credentials somewhere (if memory serves)

…

[EPiC-APOC]

It seems I got it to work now, like this:
sudo crontab -u root -e

SHELL=/bin/bash 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
30 5 * * * /usr/bin/curl -sSl https://raw.githubusercontent.com/mmotti/pihole-gravity-optimise/master/gravityOptimise.py | python3.6 >> /home/pi/EPiC/gravityOptimise.log 2>&1

=-> skipped the (absolute/direct) path @ | python3.6 ( which is in: /usr/local/bin/ )

Ran Cron Re-start:
sudo /etc/init.d/cron restart and Reboot Pi-hole (for 'just in case purposes)...

With kind regards

[mmotti]

Excellent! Cron can be some damn fiddly

…