Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spaCy analyzer #374

Closed
osma opened this issue Dec 20, 2019 · 2 comments · Fixed by #527
Closed

spaCy analyzer #374

osma opened this issue Dec 20, 2019 · 2 comments · Fixed by #527
Assignees
@osma
Labels
enhancement
Milestone
0.56

Comments

@osma
Copy link
Member

osma commented Dec 20, 2019

We could add an analyzer based on spaCy. It would enable support for some new languages and possibly also give better results for eg English and German than the current snowball analyzer.

Getting the full benefit of spaCy may require some internal API changes because it is more object oriented than NLTK and processes whole sentences instead of just individual words, taking some of the context into account.

This would be an optional feature as spaCy is implemented as a native code extension, not just pure Python.
@osma osma added this to the Short term milestone Dec 20, 2019
@osma
Copy link
Member Author

osma commented Aug 30, 2021

spaCy also has some support for document (text) categorization, both multiclass and multilabel. This could be supported as an Annif backend: https://spacy.io/api/textcategorizer

@osma osma self-assigned this Aug 31, 2021
@osma
Copy link
Member Author

osma commented Aug 31, 2021

Starting to look at this more seriously...

One immediate issue that comes up is that spacy (3.1.2) depends on typer (0.3.2) which at the moment depends on click==7.1.2, while Annif depends on click==8.0.1 since PR #499 was merged just before the 0.53 release.

Now this appears to be fixed with typer 0.4.0 that was released yesterday(!) and supports Click 8, but it won't help until there's a newer version of spaCy available that upgrades the typer dependency.

For now we may have to downgrade back to Click 7.1.2, which probably isn't a problem for Annif since I don't think we've started using any Click 8 features yet.

osma added a commit that referenced this issue Sep 1, 2021
@osma
Initial version of spaCy analyzer (#374)
227eb64
@osma osma mentioned this issue Sep 1, 2021
Merged
5 tasks
@osma osma mentioned this issue Oct 11, 2021
Closed
osma added a commit that referenced this issue Nov 22, 2021
@osma
Initial version of spaCy analyzer (#374)
4e18555
@osma osma mentioned this issue Nov 23, 2021
Open
@osma osma mentioned this issue Dec 17, 2021
Closed
osma added a commit that referenced this issue Jan 18, 2022
@osma
Initial version of spaCy analyzer (#374)
5c6719c
@juhoinkinen juhoinkinen modified the milestones: Short term, 0.56 Jan 21, 2022
@osma osma closed this as completed in #527 Jan 24, 2022
@osma osma mentioned this issue May 27, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@osma osma

Labels
enhancement
Projects
None yet
Milestone
0.56
Development

Successfully merging a pull request may close this issue.

Add spaCy analyzer NatLibFi/Annif
2 participants
@osma @juhoinkinen