The module in ntlk.model was removed in NLTK version 3, however it provides some very helpful code for text analysis. Although there are efforts to resuscitate the module, it is still not present in NLTK version 3.3 -- and the nltk/model branch has not been active in several years.

As such, I am adapting the model code from an older version of NLTK. Alternatively, one could just install and use NLTK version 2.0.4. However, hopefully, the nltk.model will return to the upstream.

This code is (barely) modified so that the imports resolve; there are no substantive changes to the code other than updating for Python 3 and referencing new NLTK method names. However, I have also included the LgramModel from even earlier versions of the NLTK, simply because it is useful in my use-case.

I have browsed the NLTK repositories in the hope of finding where the LgramModel was removed from the tree, but I had no luck (it is before the project was using git, I believe).

It retains its original copyright licence - see LICENCE.txt.

Please note that this still depends on NLTK as I am not distributing the entire NLTK package -- only a module re-adding files from the old model module. You can install the latest version of NLTK through pip: pip install nltk.

1. Install nltk through pip: pip install nltk
2. Clone this repository in your project directory: git clone https://github.com/sleepyfoxen/nltk_model.git
3. Import it and use it in your code: from nltk_model import NgramModel, LgramModel

from nltk_model import *
from nltk.corpus import brown
from nltk.probability import LidstoneProbDist


est = lambda fdist, bins: LidstoneProbDist(fdist, 0.2)
lm = NgramModel(3, brown.words(categories='news'), estimator=est)
print(lm)
print(lm._backoff)
print(lm.entropy(['The', 'Fulton', 'County', 'Grand', 'Jury', 'said',
                 'Friday', 'an', 'investigation', 'of', "Atlanta's", 'recent',
                 'primary', 'election', 'produced', '``', 'no', 'evidence',
                 "''", 'that', 'any', 'irregularities', 'took', 'place', '.']))