Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

💫 Fix loading of multiple vector models #2158

Merged
merged 10 commits into from
Mar 28, 2018
Merged

💫 Fix loading of multiple vector models #2158

merged 10 commits into from
Mar 28, 2018

Commits on Mar 28, 2018

  1. Fix loading of multiple pre-trained vectors 

    This patch addresses #1660, which was caused by keying all pre-trained
vectors with the same ID when telling Thinc how to refer to them. This
meant that if multiple models were loaded that had pre-trained vectors,
errors or incorrect behaviour resulted.

The vectors class now includes a .name attribute, which defaults to:
{nlp.meta['lang']_nlp.meta['name']}.vectors
The vectors name is set in the cfg of the pipeline components under the
key pretrained_vectors. This replaces the previous cfg key
pretrained_dims.

In order to make existing models compatible with this change, we check
for the pretrained_dims key when loading models in from_disk and
from_bytes, and add the cfg key pretrained_vectors if we find it.
    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    95a9615 View commit details
    Browse the repository at this point in the history

  2. Set pretrained_vectors in begin_training

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    9bf6e93 View commit details
    Browse the repository at this point in the history

  3. Add message noting vectors

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    17c3e7e View commit details
    Browse the repository at this point in the history

  4. Set pretrained_vectors in parser cfg

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    79dc241 View commit details
    Browse the repository at this point in the history

  5. Remove print statement

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    bc4afa9 View commit details
    Browse the repository at this point in the history

  6. Add test for #1660

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    fd9e259 View commit details
    Browse the repository at this point in the history

  7. Warn and fallback if vectors have no name

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    f8dd905 View commit details
    Browse the repository at this point in the history

  8. Dont assume pretrained_vectors cfg set in build_tagger

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    4555e3e View commit details
    Browse the repository at this point in the history

  9. Update serialization test

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    cf5fcf0 View commit details
    Browse the repository at this point in the history

  10. Avoid forcing a name on empty vectors, and remove print statement

    @honnibal
    honnibal committed Mar 28, 2018
    Configuration menu
    Copy the full SHA
    a7c5ae2 View commit details
    Browse the repository at this point in the history