A simple, fast command-line tool for scoring the similarity of text documents.

$ docsim --query some-file.txt --show-scores ~/documents/notes
0.000  completely-dissimilar-file.txt
0.152  somewhat-similar-file.md
0.469  pretty-similar-file.md
0.872  very-similar-file.org
1.000  potentially-identical-file.txt

Given a query document and a collection of potential matches, docsim ranks each document in the collection by its textual similarity to the query.

Check the man page for the definitive documentation, but these should get you started.

Search for similar files in a given directory:

$ docsim --query some-file.txt ~/documents/notes
[...]

If no paths are provided docsim will search the current working directory.

$ docsim --query some-file.txt
[...]

Without a provided --query document docsim takes input from STDIN:

$ echo "Here's a query to search for." | docsim ~/documents/notes
[...]

Only show the top 3 matches, with the best at the top:

$ docsim --query some-file.txt --limit 3 --best-first ~/documents/notes
potentially-identical-file.txt
very-similar-file.org
pretty-similar-file.md

Find Go files similar to main.go in the current directory. Don't use stemming or stoplists, since these aren't English documents.

$ docsim --query main.go --no-stemming --no-stoplist **/*.go
[...]

Notice that because docsim uses an English stoplist and an English stemming algorithm, you'll almost certainly want to use the --no-stoplist and --no-stemming flags if your documents are written in another language (including source code).

Optionally, you can use the --stoplist flag to provide a custom stoplist. A custom stoplist is just a text file of words to ignore, separated by whitespace.

The easiest thing is probably to grab a compiled binary appropriate to your platform.

But if you've got a Go toolchain handy, you can also either:

$ git clone [email protected]:hrs/docsim.git
$ cd docsim
$ sudo make install

Or just:

$ go install github.com/hrs/docsim/docsim@latest

Note that using go install doesn't include the man page, which you can optionally install manually by copying into e.g. /usr/local/share/man/man1.

$ make test

docsim uses TF-IDF weighting and cosine similarity to numerically score the textual similarity between the query and every other document.

"Textual similarity" roughly means "uses the same words." Each document is parsed into a big bag of words, which are passed through a common English stoplist, stemmed (so "spins," "spinner," and "spinning" might all reduce down to just "spin"), and assigned weights based on how often they appear in the document and how rare they are in the corpus as a whole.

We can think of each of these documents as a vector in term space, where each word is a dimension with its weight as a magnitude. Two documents are "similar," then, inasmuch as they point in the same direction, so we define similarity by the size of the angle between them.

docsim is still in a nascent state, so I'm happy just writing the code myself for now, but please feel free to report any issues you encounter!