[Dev] Funceble is too slow

[funilrys]

To quote @mitchellkrogza from mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites@4b9dbd8:

funceble from dev branch seems much slower, seems to be ignoring timeout of 2 seconds. Used to finish these builds in 20 minutes now already at 33 minutes and not even halfway ???

[funilrys]

@mitchellkrogza can you test the fix ?

[mitchellkrogza]

@funilrys testing now on https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/266230204

[mitchellkrogza]

@funilrys 😢 it finished but in double the time it usually takes. It seems specifically .se .it and .com.br are slowing it down a lot.

[funilrys]

@mitchellkrogza saw it 👍

It's related to #94

I'm into it right now 😉 The only problem is for some obscure reason for those domains those line interpret the regex as single line https://github.com/funilrys/funceble/blob/dev/funceble#L1113-L1126 🤔

[mitchellkrogza]

@funilrys let me know when you get your head around it. I'm online so will test immediately when you push a new fix.

[funilrys]

@mitchellkrogza let's test 👍 😸

[mitchellkrogza]

Underway - https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/266243864 ..... lots of duplications occuring ???

[funilrys]

Sorry but I need to laugh a second 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹 😹

Let's fix it 👍

[mitchellkrogza]

That was one hell of a good laugh 🤣 Cancelling build.

[funilrys]

Hell yeah 😹

[funilrys]

Note: That issue occurs only if we call -f, --file or --file=myawesomelist.domain

[mitchellkrogza]

.md and .sk domains also giving that weird output

[funilrys]

With the last commit @mitchellkrogza ?

[mitchellkrogza]

Nope sorry, with the previous one .... my bad .... still running previous dev version on my ubuntu box and just saw those. Will wait for update.

[mitchellkrogza]

Note: That issue occurs only if we call -f, --file or --file=myawesomelist.domain

Why is that?

[funilrys]

A note for myself 😹 it's not happening when you use ./funceble -d github.com but it appears if you use ./funceble -f myawesome.list

[funilrys]

@mitchellkrogza Let's fix test 👍

[funilrys]

17514 that's the number of line I had to read to find where was the problem 😹 😹 😹 😹 😹 😹 😹

[mitchellkrogza]

And off she goes 👍 https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/266254028

[funilrys]

Okay, let me see the difference between https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/265988407 && https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites 😭

[mitchellkrogza]

I feel your pain 😢

[funilrys]

Note: [That] https://www.diffchecker.com/arpDbj4l don't explain everything 😭

[mitchellkrogza]

Can't understand how the execution time has grown so much ??

Execution Time:
00:45:33

 Status      Percentage   Numbers     
----------- ------------ -------------
ACTIVE      99%          2509        
INACTIVE    0%           2           
INVALID     0%           0           

[funilrys]

The diffchecker tell us the difference between before (left) and now (right) so it's the refactoring that produces that issue 😭

[mitchellkrogza]

Any way I can run this in full debug mode on by Ubuntu box? Maybe produce a debug output file to see what's causing the execution time to double?

[funilrys]

I already did it over a 100 domain list but the only way is to rewrite the way we read the whois output 😅

But if you want a huge and full debug ..
bash -x ./funceble -ex -h -p -f myawesome.list 2>&1 | tee executiondebug

[mitchellkrogza]

Pulling latest dev branch now to my Ubuntu box and will run this for an hour and see. If I stop it will we still get the debug output needed?

[funilrys]

yes it's a real time input into executiondebug

[mitchellkrogza]

Running now but already I see the script pausing continuously with message expect -c 'set echo "-noecho"; set timeout 30; spawn -noecho /bin/bash -c "whois -h whois.nic.site 24x7-server-support.site > /var/tmp/whois.9524"; expect timeout { exit 1 } eof { exit 0 }'

[mitchellkrogza]

And also like here:

+ notSupported=true
+ case ${domainExtension} in
+ [[ true == true ]]
++ whois -h whois.iana.org 66cpwgln.space
++ awk '/refer/{print $NF}'
+ referer=whois.nic.space
+ regexWhois='(.*)whois(.*)'
+ [[ whois.nic.space =~ (.*)whois(.*) ]]
+ notSupported=false
+ [[ ! 66cpwgln.space =~ (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?) ]]
+ [[ false == false ]]
+ [[ 66cpwgln.space != '' ]]
+ [[ whois.nic.space != '' ]]
+ timeout 30 'whois -h whois.nic.space 66cpwgln.space > /var/tmp/whois.9524'
+ time=30
+ touch /var/tmp/whois.9524
+ command='/bin/bash -c "whois -h whois.nic.space 66cpwgln.space > /var/tmp/whois.9524"'
+ expect -c 'set echo "-noecho"; set  timeout 30; spawn -noecho /bin/bash -c "whois -h whois.nic.space 66cpwgln.space > /var/tmp/whois.9524"; expect timeout { exit 1 } eof { exit 0 }'

[funilrys]

What about bash -x ./funceble -t 1 -ex -h -p -f myawesome.list 2>&1 | tee executiondebug ?

[funilrys]

Okay let's change the default timeout to 2 or 3 seconds

[mitchellkrogza]

Shall i wait for fix? And then run this again? Stopped it after just 5 minutes already 8mb debug file.

[funilrys]

stop ip :)

My last commit only change the timeout from 30 to 3s

[mitchellkrogza]

Running now, no pausing yet. Let me run this on Hacked Malware Sites and see how it does on Travis

[mitchellkrogza]

In progress now > https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/266273751

[funilrys]

Let's test this one 😭 This one there's no loop one line after another I only search the line that get the expiration date 😅

[mitchellkrogza]

In progress > https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/266285174

[mitchellkrogza]

[funilrys]

Okay, @mitchellkrogza let's resume my long last night and day :)

• This is the last diffchecker: https://www.diffchecker.com/KxgqBi4m

  • on the left: we have the version of funceble used by https://travis-ci.org/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/builds/265988407
  • on the right: the current dev version of funceble

• This is another diffchecker: https://www.diffchecker.com/p5bSt8Cy

  • on the left: the last funceble file I used for the last Travis build ==> https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/blob/master/.dev-tools/_funceble/funceble
  • on the right: the current dev version of funceble

As you can see, we have a real problem which looks like it's out of the code ...