domains-dotted-format.list is actually Squid-Cache dstdomain ACL format .

[elico]

@funilrys @mitchellkrogza I noticed that the domains-dotted-format.list is compatible with Squid-Cache dstdomain ACL.
I will try to test it in the next few days.
If it is compatible you just need to add it as compatible in the files table.

[mitchellkrogza]

Thanks @elico I made the dotted format list for DNSMASQ users and would be great if it applies to Squid too. We also have Blackweb using our lists too.

[elico]

@mitchellkrogza from first look it seems that it'w not 100% compatible but it only requires a simple cleanup:

# sort domains-dotted-format.list|grep '.ynet.co.il'
.stats.ynet.co.il
.ynet.co.il

and squid will accept only the first in the list and since it's a dotted format and by that defining that this is a "prefix" line in the ACL. ( the trie tree only accepts one end to a tree and will not accept collision\overlapping rules)

However since the list is a full match only and aimed towards a specific domains compared to a full domain and all subdomains then: domains.list
is compatible with Squid-Cache dstdomain ACL type.
Ie can accept:

# sort domains.list|grep 'ynet.co.il'
stats.ynet.co.il
ynet.co.il

and will match only these two domains as blacklisted

[mitchellkrogza]

So ... is the correct solution not then the dotted format list but with no duplicates and only root domains ?
ie. no sub domains?

[elico]

@mitchellkrogza Since this blacklist is composed and designed to target specific domains domain overlapping each other from upper level to lower level is expected to exist.
From what you told me about the list in the past the best format is domains.list and by nature it is compatible with Squid-Cache dstdomain ACL format.

Squid has a very specific trie data structure for a domain list which uses a dotted format in a very standard way.
I am almost sure that dnsmasq does the same but I need to verify it first.
Squid dstdomain ACL format has two options to add a domain:

• with a dot prefix(.ynet.co.il or .1.ynet.co.il)
• without a dot prefix(ynet.co.il or 1.ynet.co.il)

A dot prefix means that all this specific domain and all it's subdomains are a match for the ACL and are blacklisted.
So if we have a specific domain like .ynet.co.il already we cannot add the .stats.ynet.net.il or 1.ynet.net.il or any other similar subdomain.

Also if we have a subdomain in the ACL ie t.ynet.co.il squid trie structure won't allow the admin to add .ynet.co.il but will allow the admin to add ynet.co.il since they do not overlap in their domain inclusion.

Hope it gives more sense now.

[mitchellkrogza]

Please do let us know the outcome as we are always striving to perfect what we do and support all systems 100% what you say makes total sense to me and with clarification on that we will perfect our raw lists even more

[elico]

@mitchellkrogza @funilrys I tested here and there to make sure what and how things are at dnsmasq.
DNSmasq is using the same logic as Squid-Cache.
A dot prefix domain includes the domain and all it's subdomains so you need to define how and what you want the list to perform like.
An example to dnsmasq.conf blackhole: address=/.ynet.co.il/0.0.0.0
There is no tutorial or README on how to set dnsmasq to work but it's pretty simple:
/etc/dnsmasq.conf

domain-needed
bogus-priv
strict-order
expand-hosts
domain=masq.ngtech.local
conf-dir=/etc/dnsmasq.d

/etc/dnsmasq.d/blackhole.conf

address=/.ynet.co.il/0.0.0.0

I created 3 scripts that takes the domains-dotted-format.list and generates a dnsmsq blackhole config file.
Since this format is for DNSMasq only then upon creation there should be a separation between:

• catch all subdomains rule
• specific domain

Most of the domains should be blocked with all their subdomains but some exist on a very specific hosting service and should be present without a dot prefix.

What do you both think?

• http:https://gogs.ngtech.co.il/elicro/ultimate-hosts-bl-utils/src/master/update-scripts

[elico]

@mitchellkrogza @funilrys I don't know if dnsmasq have issues on specific versions such as in alpine linux but on alpine linux that I tested the dnsmasq service consumed almost 300 MB and didn't replied.
I will try to test on Ubuntu 16.04, CentOS 7, Arch Linux to see how it perform on these.

Also I just finished the basic skeleton for a DNS domains BL service that just answers if the domain is in the list or not.

• http:https://gogs.ngtech.co.il/elicro/ultimate-hosts-bl-utils/src/master/domain-check-only-dns

I will need to have an option for the list to be inclusive or exclusive:
all subdomains of a listed domains would be marked as blacklisted compared to that only a full match for a specific domain is included.

Technically speaking if a sub tld exists in the list it mostly should block also all subdomains.
IE if I have ynet.co.il then it must include all subdomains since it's saying that this "entity" is blacklisted.
For other sublevel domains such as if subdomain.ynet.co.il it might not be an inclusive for all subdomains.

[elico]

@mitchellkrogza I have tested DNSMasq with couple versions and it seems that the only system that has an issue with the block list is dnsmasq ontop of Alpine linux x86(32bit).
I have not tested it on Apline linux x86_64 since I believe it's either a libc\muslc or nettle issue.