Updates Delay (List culling and removal of dead domains)

[mitchellkrogza]

A big build is in progress to get a reliable output of all dead domains on this hosts list. Updates to this repo will only resume in about 7 days time.

[mitchellkrogza]

Something just interesting, can you believe that burger-imperia.com and pizza-tycoon.com are still the top ranking most active bad sites out there. Every day in all my logs across 28 sites.

[xxcriticxx]

should i visit them?

[mitchellkrogza]

Burger Imperia (but of course all the nasty stuff /click redirecting and hijacking is hidden when you visit them)

[mitchellkrogza]

and pizza tycoon

[xxcriticxx]

looks like same author

[mitchellkrogza]

Absolutely and been a pain in people's back sides for years yet nobody seems to be able to take them down.

[mitchellkrogza]

So we uncovered an undocumented Travis feature. If a repo builds continually, even though all builds are passing, for more than 48 hours. Travis CI stops building after 48 hours.

We did however accumulate a lot of data which can be now used to start cleaning this list of dead domains beginning with all those that returned 404. The remainder of the testing I will probably do from one of my Ubuntu Servers and not tie up Travis like this.

So possibly later today we will have our first list of dead domains to start stripping out of the final RAW files produced.

[mitchellkrogza]

Okay, so after further investigation we did not reach any sort of timeout. One of the input sources - https://raw.githubusercontent.com/mitchellkrogza/Ultimate.Hosts.Blacklist/master/.input_sources/_Airelles_Anti_Sex_Hosts/domains.txt had an entry cz as a domain name which caused the testing script to exit and go into final commit mode.

That's the whole idea of this repo is to clean up junk information like that from all these input sources / black lists out there.

[mitchellkrogza]

@xxcriticxx new raw links pushed out today. So far lists are now stripped of 19473 dead domains. So now we resume the funceble testing again.

[xxcriticxx]

@mitchellkrogza last count was at 1.5 mil now -19k more?

[mitchellkrogza]

Remember each time I do a commit it pulls in fresh data so list sizes change all the time

[mitchellkrogza]

Ultimately we want this smaller lighter and more accurate no?

[xxcriticxx]

@mitchellkrogza correct

[mitchellkrogza]

👍

[xxcriticxx]

and no .cz domains

[mitchellkrogza]

LOL .cz yes just not cz all by itself, that issue has been resolved from all future lists where I check the combined list now for any line not containing a . and delete it before creating the hosts files.