privoxy list

[xxcriticxx]

maybe next project?

[mitchellkrogza]

?? You mean this? http:https://andrwe.org/scripting/bash/privoxy-blocklist

[xxcriticxx]

there is few scripts out there i could not get to work to my needs and went in pihole direction

i have privoxy on my router

[mitchellkrogza]

I'm actually working on another script to take all the bad ip's from Ultimate hosts and add them to iptables automatically once a day. But will look into this some time.

[mitchellkrogza]

@xxcriticxx must tell you @funilrys and I are like so close to final changes to funceble with a new travis autosave and continue mode. Once he has completed some refactoring and made final changes I will release it on this repo to find all the dead and inactive domains. you can see it action so far at: https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/commits/master

Gonne be awesome when funceble 2.0.0 merges into master.

[mitchellkrogza]

I also have a #slack channel for Ultimate hosts so shout if you want an invite to it

[xxcriticxx]

#slack on irc?

[mitchellkrogza]

No Slack on https://cybersecurity-crew.slack.com

[xxcriticxx]

Thats pass my paygrade I will hang out over here with you guys

[mitchellkrogza]

LOL ... no worries, much less noise here anyway as #Slack channels can get rather noisy

[elico]

@mitchellkrogza Was there any progress with the IPTALBES integration?
I have couple scripts here that might help.
Also it is possible to write couple nice update scripts for routers such as:

• Juniper(SRX\MX\Others)
• Mikrotik
• VyOS
• OpnSENSE?
• BSD PF\IPFW

[funilrys]

Hey @elico,
@mitchellkrogza will be back next week (normally)!

As we decided to rewrite the whole structure in order to be more efficient and less passive with tests running for weeks, we are now working with a new layout.
Which means that we are merging input source and generating all format with Python instead of an unstructured bash like we did before.

I still have to learn those format but it's not impossible (and maybe now easier) to port them to our system.
So, if you already have some useful script let me know, I'll be happy to give some time to integrate it to the current system/monster as long as it does not implies generating a 1GB file.

From internal discussions, @mitchellkrogza had the idea of using https://github.com/funilrys/hosts2iptables for one of his other projects but we conclude (at the time) that it can be a starting point for future projects or improvement.

[elico]

@funilrys https://github.com/funilrys/hosts2iptables has some really nice points but most Linux distributions has ipset support built into them so it's better to use them to be more efficient and to allow a more static iptables rules set while dynamically change or update the ipset lists.

May I ask how the 1GB file size got into your mind? even IPv4 BGP feeds are not that big and these should contain every single CIDR of the Internet.

Specifically MikroTik has a "feature" named Access Lists which they didn't revealed to me and many others yet what is the storage backend inside their system but it's Linux kernel based.. so I believe it's IPSET.
They have a nice option that can demonstrate the concept I am aiming towards and can be seen here at this video: https://www.youtube.com/watch?v=3tXVmzAAJoQ

In Linux when you create an IPSET hash:ip or hash:net which can store either a single ip address or a cidr.
The size of the hash is set at creation like this:

ipset create blockiptest hash:ip hashsize 65535 maxelem 131070
ipset create blocknettest hash:net hashsize 65535 maxelem 131070

It's easy to create something like what MikroTik offers but it will probably use lots of DNS queries.
I wrote couple scripts for the Squid-Cache project that uses ipset "dynamically":

• https://gist.github.com/elico/a54c2c8f8e1a2407b42210896b960f4b
• https://gist.github.com/elico/e0faadf0cc63942c5aaade808a87deef

When you use a command like at: https://github.com/funilrys/hosts2iptables/blob/master/hosts2iptables#L137
What iptables does is resolve the domain name using the local resolv.conf file and only add one single ip address from the set of results.
In any case plain iptables or\and ipset rules will intensively use DNS and I believe that the current domains file is sufficient to be used as a feed to these.

I will try to dig(no promises) into my MikroTik scripts to find the right way to generate the creation of an address list on the fly.
An simple example snippet for what should run on a MikroTik device to create an address list is:

/ip firewall address-list
add address=204.52.208.0/24 comment="ISRAEL" list=CountryIPBlocks
add address=2.52.0.0/14 comment="ISRAEL" list=CountryIPBlocks
add address=5.22.128.0/21 comment="ISRAEL" list=CountryIPBlocks
add address=5.28.128.0/18 comment="ISRAEL" list=CountryIPBlocks
add address=5.29.0.0/16 comment="ISRAEL" list=CountryIPBlocks
add address=5.34.168.0/21 comment="ISRAEL" list=CountryIPBlocks
add address=5.100.248.0/21 comment="ISRAEL" list=CountryIPBlocks
...

I believe that for all these IP level routing devices the best way to block domains is using a combination of tools such as a proxy and\or dns interception and\or some layer 7 kernel module.
I wrote a blacklist API server which could be used publicly but for some reason nobody used it for over a year, It offered a very simple DNS + HTTP + ICAP services.

[xxcriticxx]

@mitchellkrogza on vacation again?

[funilrys]

@xxcriticxx from his word: he is "leaving a week for some away time" 😄

@elico Interesting I did not know about ipset 👍 Thanks for letting me know 💯 I'll have to learn more about it.

Taking it like you present it, ipset seems to be a real alternative to what I did with my script.
About that example you gave us does list point to a file? Also, are comments obliged? Is there any restriction on the comment?

We have 2M of domains and IP and to comment all of those may not be useful... But from your example, it's not that hard to generate that format 👍
I may have to wait for @mitchellkrogza to take the decision if we decide to deliver that format but the code for generating that format will be in a private branch.

To answer your question, you can not see it in the history anymore as I was obliged to clean it, but before I took the time to analyze and write the whole system behind the current repository structure, we had a input-sources directory which was 4GB long ...

When I proposed my vision to @mitchellkrogza we both agreed that I'm crazy but now it's a bit joyful now to maintain this repository as we only add input sources and the system work by itself in backend which was not the case before.
I'm kinda proud of what I do with PyFunceble and all 50+ repository running it every day to test or test and clean lists. 😸

So in the future, we want to avoid generating a directory or a file which is bigger than 1GB which is a limit for an easy to maintain and distribute Git repository...
But when @mitchellkrogza will be back it's going to be crazy to respect as he promises me that we are going to have more input sources 😹 (we both are crazy sometimes 😹 )
So not sure if that restriction will stay longer than a month ...

[xxcriticxx]

if you want to see crazy come to nyc

[elico]

@funilrys There are restrictions on comments and it's not a requirement,
I think that the best solution with mikrotik devices is to write a script that will convert current ip cidr lists and or domains into some mikrotik commands.

Also about ipset, there are two or more ways to export\pack or import\unpack an ipset "memory set".
ipset tools have the option to export and pack a "set" into a binary formatted file and it can be created very easily(I wrote and have seen some code to do it couple weeks ago).
there is a tool\package which is named "ipset-persistent" which allowes a linux system administrator to save a set into a file at any reboot.
mainly ipset is used with tools such as fail2ban and for big systems that uses ipset the admin would prefer to have his big black or white ip lists saved at every reboot.

[elico]

@mitchellkrogza "shouting" can you invite me to the #slack chat?

[mitchellkrogza]

@elico please can you drop me an email on [email protected] and then I can add you

[elico]

@mitchellkrogza You'v got mail

This message was created automatically by mail delivery software.
 ----- The following addresses had successful delivery notifications ----- <[email protected]> (relayed to non-DSN-aware mailer)

[elico]

@funilrys I found a very good tech talk about ipset:

• https://youtu.be/dMz9A4Jh0_c

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.