-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multilabel Live Test #9
Comments
Hi @oterrier! Thanks for creating this issue. Yes, Issue #6 added the possibility to load multi-label datasets from disk using two common file structures. Now I'm currently working on Issue #5, which, once finished, will enable PySS3 to provide full support for multi-label classification. I've just finished working on Yes, it would be great to have the feature you're suggesting. Besides, I think that
It would also be a nice feature to have, what do you think? In the case of multi-label classification, The only thing that I'm not entirely sure about yet is how to show the list of documents in the Live Test. So far, since each document belongs to only one category, documents are grouped by categories in the left panel, as shown in the following image: Also, the % of hits (recall) and the "misclassified" icon (!) should be removed or replaced by something else when loading documents with multiple labels. I think one possible solution is, divided into the following, from easiest to more complex, steps: Step 1 When loading multilabel documents, remove everything from the left panel, and only show the plain list of all documents (no categories, no % of hits, no misclassification icon). Once a document is selected, and once classified, show the list of true category labels along with its name, in the location that is marked below: There, we could even paint in green correctly predicted labels and in red misclassified ones. Step 2 For each document, add the % of the total labels correctly classified. For instance, if we have the following case:
And the predicted labels are:
In the Live Test we could show the list with these 3 documents along with the % of label hits (recall), as follows:
Step 3 When the user moves the cursor over the %, the actual list of true labels is shown along with the predicted ones. Maybe we could use a Tooltip to accomplish this. Step 4 Add a filter option at the top of the panel, allowing the user to filter the list of documents by category labels or even other options, like %0 of hits, for instance. What do you think about this? It is not necessary to implement all these steps; step 1 alone will be enough to enable the user to load multilabel documents. Further steps only improve user experience. Once I finish working on Issue #5 I'll start working on this one, however, feel free to submit any PR, since any kind of help would give me a "head start" and I'll greatly appreciate it 😎 |
Hi again @oterrier! I've just finished implementing the multi-label support for the Live Test Tool (up to step 2 included). As described above, now, in the left panel, the list of documents is shown with no categories and when selected, the true labels are shown along with the predicted labels, as shown below: Step two was also implemented, and each test document in the list is shown with a % corresponding to its label-based accuracy (aka hamming score). Besides, misclassified labels are shown colored in red, as "drama" below: Let me know Oliver if this is what you needed/wanted, or if you find something that needs to be improved/changed/fixed. I'm performing the final checks before releasing the new version, which will include all the changes and improvements regarding multi-label classification 😄 |
Wow! It looks good Thx |
Now the Live Test Tool for multi-label classification can be lunched with a default category label to be assigned when no tags could be predicted by SS3. To do that, the ``def_cat`` argument was added, therefore, ``Live_Test.run`` can now be called as follows: ``` Live_Test.run(clf, x_test, y_test, def_cat="most-probable") Live_Test.run(clf, x_test, y_test, def_cat="aCategoryLabel") Live_Test.run(clf, x_test, y_test) # default, predict no label ([]) ```
PySS3 now fully support multi-label classification! :) - The ``load_from_files_multilabel()`` function was added to the ``Dataset`` class (7ece7ce, resolved #6) - The ``Evaluation`` class now supports multi-label classification (#5) - Add multi-label support to ``train()/fit()`` (4d00476) - Add multi-label support to ``Evaluation.test()`` (0a897dd) - Add multi-label support to ``show_best and get_best()`` (ef2419b) - Add multi-label support to ``kfold_cross_validation()`` (aacd3a0) - Add multi-label support to ``grid_search()`` (925156d, 79f1e9d) - Add multi-label support to the 3D Evaluation Plot (42bbc65) - The Live Test tool now supports multi-label classification as well (15657ee, b617bb7, resolved #9) - Category names are no longer case-insensitive (4ec009a, resolved #8)
Hey @sergioburdisso,
I've noticed taht you fixed recently the Multilabel fit issue #6
But the Live_Test.run(clf, X_test, y_test) still does not accept y_test as List[List[str]]
It would be really great to have it
If you don't have time maybe I could submit a PR?
Olivier
The text was updated successfully, but these errors were encountered: