idea

[xxcriticxx]

@mitchellkrogza @funilrys

can you code something that finds domains on this list that aren't on any other list?

usually this way i find lots false positive

[funilrys]

@mitchellkrogza Give me 24 hours and I'll write a script unless you have a one line (I know how much you love them 😹 ) command for @xxcriticxx 👍

[mitchellkrogza]

Great idea, I'm sure I can do a one liner to achieve that, will look in the morning

[mitchellkrogza]

I'll test this in the morning grep -Fxv -f first-file.txt second-file.txt or comm -23 second-file-sorted.txt first-file-sorted.txt

[xxcriticxx]

something like Diff Checker but i dont know if it would handle 2 millions lines

[mitchellkrogza]

The comm command line should do what you need, I ran it against Ultimate Hosts and Badd Boyz and it took all of 5 seconds

[xxcriticxx]

@mitchellkrogza can you tub it against few larger listsand post output here?

[funilrys]

I think I can't do better than 5 seconds 😸

[mitchellkrogza]

@xxcriticxx send me one list for now to test against Ultimate to find any uncommon entries

[mitchellkrogza]

@xxcriticxx @funilrys I just ran Ultimate Hosts against itself to show how quick it is. So we are talking 1,602,081 domains. 2 seconds to complete ... no jokes. You can see result.txt is 0 kb which means there are no odd one's out 😄 as @funilrys says I sure do LOVE my one liners and Linux RULES !!!

[xxcriticxx]

i will play with this on the weekend

[mitchellkrogza]

@xxcriticxx great, please share your findings with us

[xxcriticxx]

comm wont work

comm -13 list1.txt list2.txt > result.txt
comm: file 2 is not in sorted order
comm: file 1 is not in sorted order

[mitchellkrogza]

comm will work, you need to make sure both lists are sorted first.

sort -u list1.txt -o list1.txt
sort -u list2.txt -o list2.txt
comm -13 list1.txt list2.txt > result.txt

[mitchellkrogza]

Unfortunately lists need to be sorted and also clean as far as funny commenting like # Whatever and empty lines too

[xxcriticxx]

if i sorted them they wont same and stripping # will take hours

[mitchellkrogza]

There's very easy scripts to remove commenting, empty lines and clean a file. Send me your list1.txt and list2.txt to my email (zip them) - [email protected] and I will show you tomorrow and also show you how quick and easy it is using command line tools

[xxcriticxx]

list1 is your list
list2 is stevenblack list

[mitchellkrogza]

Will run them against each other tomorrow

[xxcriticxx]

@mitchellkrogza send me nice picture of nature i am in need of new wallpapers

[mitchellkrogza]

What size? and what kind I have far too much stuff :) Best to check my Facebook Page and drop me an email. https://www.facebook.com/MitchellKrogPhotography

[xxcriticxx]

i have 2x Acer G277HL 1920 x 1080

[smed79]

Extract lines in file1 not found in file2

diff --new-line-format="" --unchanged-line-format="" file1 file2 > file3

Extract lines from file2 already found in file1

awk 'NR==FNR{lines[$0]++; next} $1 in lines' file1 file2 > file3

[mitchellkrogza]

Thanks @smed79 👍

[funilrys]

With the new repository structure/system, once we merged all sources, we remove all duplicates before generating the files 😸

[xxcriticxx]

The idea was to find something here that doesn’t exist on any other list or lists

[funilrys]

Because we are going to work with big lists soon, we decided that it will be great to have a script or system which can help us find the amount of domains from a list which are not already part of this repository.

So I invite you to play with https://gist.github.com/funilrys/900abd388b1f3b399a9da69e0e592fef (-h gives you the help) !!!

Have a nice day/night.

Closing.