Create demo notebook for Conllu files #198
Conversation
…p the demo notebook to use it to hopefully avoid sticky coyright situations
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
notebooks/Read_CoNLL-U_files.ipynb
Outdated
| @@ -0,0 +1,1116 @@ | |||
| { | |||
There was a problem hiding this comment.
It looks like the "head" columns ended up as floating point values somehow. They should be ints. Did some nulls sneak into the head pointers?
Reply via ReviewNB
There was a problem hiding this comment.
I believe this actually has to do with 'virtual' tokens where words were omitted from the surface representation and had to be added by the labeller, which have numbers of the form 10.1 or similar
There was a problem hiding this comment.
Do you have a suggestion of how I should handle this?
There was a problem hiding this comment.
The real root cause is upstream in Pandas. There is old code in various parts of Pandas that casts ints to floats for operations that might produce NA values. Pandas should be converting to nullable int types (see https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html) instead. I've been meaning to submit PRs to fix those problems but haven't had time.
For now, the best fix at the application level is to convert the series that Pandas has "floatified" back to an integer dtype. Use int if there aren't supposed to be NAs. Use Int64 (with a capital "I") if there might be NAs.
There was a problem hiding this comment.
Follow-up: Looking some more at the code, I think the conversion to a nullable integer dtype should happen inside conll_u_to_dataframes.
|
This is very close to ready to merge. Some minor requests:
|
Created a demo notebook that shows how the new conllu logic can be used to open and process diverse .conllu files; addresses issue #195
Shows how these can be powerfully used with pandas dataframes, and other text-extensions integrations to quickly manipulate the data and do things like displaying parse trees, and retokenizing with spaCy.