Extract variable-length MWE using a user-defined POS regex pattern.

[leobeeson]

Resolves #19 and closes #42

Problem

Currently, only two-word MWEs are supported by the MWE class.

Objective

Enable the extraction of higher-order Ngrams.

Results

• Created class HigherOrderMWEExtractor with workhorse method extract_higher_order_mwes() for extracting variable length MWE given a user pattern of POS tags.
• Wrote unit tests for validating class instantiation arguments. All are passing.
• Wrote unit tests for validating extraction of variable-length MWE from user-defined POS patterns. All are passing.
• Pre-commit hooks pass.

Considerations for Future Work

• The class HigherOrderMWEExtractor was designed to be decoupled from the rest of the dataset; unlike the class MWE that is coupled with the entire corpus via its df attribute.
  • HigherOrderMWEExtractor operates atomically on a single sentence input and for a specific pattern for that input. This allows for vectorising using different patterns
• We might want to refactor the class and method names, but must be done in conjunction with refactoring the name of the MWE class, as the latter is significantly broader than just a MWE. MWE could be renamed to MWEExtractor (although this is still too generic), and HigherOrderMWEExtractor could be renamed to VariableLengthMWEExtractor, POSPatternMWEExtractor, etc.
• It would be good at some point to benchmark the efficiency of HigherOrderMWEExtractor.extract_higher_order_mwes() as it uses nltk.RegexpParser, i.e. uses regex pattern matching, which could have efficiency implications.
• I can't run pre-commit hooks with language-version python3.9 (as I'm only using 3.10), but for some reason mypy would not let me run pre-commit hooks without also committing the .pre-commit-config.yaml. It feels like not committing this file should be whitelisted by mypy, but I could be wrong.

[meghdadFar]

Looks good but what do you think about renaming extract_higher_order_mwes because it returns MWE candidates (and their counts) for certain patterns? They are not exactly MWEs.

To extract valid MWEs, the calculate_am function should be first updated to be able to calculate AM scores for a variable length sequence and then it should be applied to MWE candidates returned by extract_higher_order_mwes.

We can upgrade calculate_am() in a different PR.