Skip to content
/ black-mirror Public
forked from T145/black-mirror

Blacklists and whitelists that aim to promote security, safety, and sanity across the internet!

License

AGPL-3.0 license
1 star 12 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

Te-k/black-mirror

BranchesTags
 
 

Repository files navigation

logo

🌓 Reflection | 💿 Redundancy | ✅ Reliability

Blacklists and whitelists that aim to promote security, safety, and sanity across the internet!

humane-tech visits release release release docker
Made with 💖 by T145!

🛡️ Privacy Protectors

Good causes that help secure your digital life.

💕 Companion Projects

Great projects that leverage Black Mirror's content!


NXDOMAIN scanner against Black Mirror's lists

🖋️ Manifesto

Please report any hosts that are wrongly blocked or sources that do not wholly align in an issue.

Defines the logic behind why a host is permitted or blocked. It has been written to reflect common ideologies across the blacklisting community and those specific to this project.

Complete author bias reflection is not intended, which is why it speaks from the Black Mirror person. Free thought exchange is encouraged, so feel free to open discussions about any points!

📋 Attributes

  1. Produced in domain-only, IPv4-only, IPv4-CIDR-only, and IPv6-only builds.
  2. Updates at 1:27 AM & PM UTC.
  3. No excess or trailing whitespace.
  4. No lingering webscraper garbage.
  5. Lines are terminated with lf.
  6. No blank lines.
  7. No comments.

🚚 Deliverables

⚓ Hyperlinks

List Name File Content Unique Entries File Size Mirror 1 Mirror 2
black_domain Domain entries 7,475,525 176M [✔️,🔗] [✔️,🔗]
black_ipv4 IPv4 addresses 680,111 9.2M [✔️,🔗] [✔️,🔗]
black_ipv4_cidr IPv4 CIDR blocks 39,261 659K [✔️,🔗] [✔️,🔗]
black_ipv6 IPv6 addresses 9,789 305K [✔️,🔗] [✔️,🔗]
white_domain Domain entries 416,479 8.7M [✔️,🔗] [✔️,🔗]
white_ipv4 IPv4 addresses 13,976 192K [✔️,🔗] [✔️,🔗]
white_ipv4_cidr IPv4 CIDR blocks 1,462 23K [✔️,🔗] [✔️,🔗]
white_ipv6 IPv6 addresses 2,793 107K [✔️,🔗] [✔️,🔗]

Source Code

🧮 Checksum Evaluation

cat black_domain.txt | sha256sum -c black_domain.checksums --status && echo $?

A return code of 0 means the check was successful. The specific checksum command can be any of the following:

  • md5sum
  • b2sum
  • sha1sum
  • sha224sum
  • sha256sum
  • sha384sum
  • sha512sum

🐙 Fetching GitHub Releases

Provided below are some examples to fetch release artifacts leveraging the GitHub API.

Get all build artifacts

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[].browser_download_url'

Get a build artifact & its checksum

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[] | select(.name | startswith("black_domain")).browser_download_url'

Get a single build artifact

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[] | select(.name | startswith("black_domain")) | select(.name | endswith(".txt")).browser_download_url'

🛠️ Usage

Desktop OS Hosts File

mawk '{print "0.0.0.0 " $0}' black_domain.txt >>hosts
# mawk '{print ":: " $0}' black_domain.txt >>hosts
mawk '{print "0.0.0.0 " $0}' black_ipv4.txt >>hosts
mawk '{print ":: " $0}' black_ipv6.txt >>hosts

dnsmasq

Many popular platforms such as OpenWRT, DDWRT, and Pihole use DNSmasq as their choice TCP powerhouse. After inspecting many domain blocklists you'll inevitably run across a list in the dnsmasq.conf format. This list doesn't support it because you can use the addn-hosts parameter to add hosts in the list. Target a file that has the hosts in a format similar to the Desktop OS Hosts File format.

If you're using the RADVD daemon, prepend any listed hosts with ::. Otherwise, even if you have IPv6 support set up, prepend hosts with 0.0.0.0.

This has been tested across all the mentioned platforms using dig{6} on a small sample size and had each host null-routed successfully. DNSmasq's man page discusses configuration further, and DDWRT's ad blocking wiki page provides some examples.

Amazon EC2 DNS Resolver

Follow this guide to create a DNS server on a Amazon EC2 instance.

pihole

If you'd like to update when some sources do or not extract a production build, just use the single-line list sources.pihole. Note that this list only contains Pihole-compatible sources, and not every handled source. Some manual configuration may also be required.

unbound

Similar to dnsmasq, but requires more manual configuration. Name any products as a *.conf file. Then follow Steffinstanly's instructions on how to apply blocklists.

personalDNSfilter

Use the domain list.

adguard

Leverage the sources.adguard list or the domain list.

📚 Sources

Please report any redundant sources in an issue!

⬛ Blacklists

TODO

⬜ Whitelists

Applied to generated blacklists.

TODO

🥢 Duplicates

Sources that contain duplicate and potentially deprecated data.

🧟 Zombies

Sources that are dead or deprecated and not included but may be worth mentioning.

🕵🏻 Lamers Unwelcome

🃏 Interesting Oddities

📦 Big Data Lists

Typically used by other blacklist projects as whitelists.

List Name URL
Alexa https://s3.amazonaws.com/alexa-static/top-1m.csv.zip
Umbrella https://downloads.majestic.com/majestic_million.csv
Statvoo https://statvoo.com/dl/top-1million-sites.csv.zip
OpenPageRank https://www.domcop.com/files/top/top10milliondomains.csv.zip
Quantcast https://toplists.net.in.tum.de/archive/quantcast/quantcast-top-sites-2021-06-25_0900_UTC.txt.xz
Tranco list https://tranco-list.eu/

🌐 IP Block Providers

Simply provide IP blocks for entire geographic regions.

🎉 Special Thanks

🌟 Stargazers

Stargazers repo roster for @T145/black-mirror

🍝 Forkers

Forkers repo roster for @T145/black-mirror

🎶 Notes

Need for speed

List References

About

Blacklists and whitelists that aim to promote security, safety, and sanity across the internet!

Resources

Readme

License

AGPL-3.0 license

Code of conduct

Code of conduct
Activity

Stars

1 star

Watchers

1 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Shell 81.5%
  • Dockerfile 10.3%
  • Python 5.5%
  • Awk 1.6%
  • Perl 1.1%