-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some questions regarding evaluations on next keyword prediction #4
Comments
Thanks for your attention,
You can also minimize the negative log-likelihood loss of every target keyword after a softmax layer. In my impression the training result is similar.
|
@squareRoot3 Thank you very much for the quick reply. I have two more questions regarding keyword prediction. Q1It seems that the test keywords are used as the vocab during training? Any reasons for this? ./config/data_config.py:
./model/neural.py:
Q2I experimented both Here is the PyTorch code to compute the two losses:
|
Q1: I had thought that the test keyword vocab contains more frequent keywords and the size is relatively smaller, which can facilicate training. But using the train keyword vocab seems more reasonable. We have fixed this in the new repository: https://github.com/James-Yip/TGODC-DKRN. Q2: It looks like that the implement of two losses are correct, so I am sorry that I have no ideas about it. The BCE loss in our repository works normally. |
Sorry to bother you again. Another strange thing happed to the retrieval-neural model. I trained a keyword prediction model and obtained around 0.08 test R@1. I also trained a retrieval baseline (without ketword conditioning) and obtained around 0.51 test R@1. However, when I train the retrieval-neural model to use predicted keywords to retrieve the next turn, the result is still around 0.51. It seems that using keywords do not improve model performance. My implementation of conditioning on keyword follows your code:
|
It seems that the word embedding of the trained keyword predictor is used in the retrieval model. I didn't implement that. I will fix this and let you know if it works. |
After reusing the word embedding from the trained keyword predictor, the retrieval-neural model achieves test R1 of 0.5235, which is still a bit below the reported 0.5395. Hmm... |
I suspect that one of the reasons is that we use different pretrained word embeddings. How is your pretrained word embedding obtained? GloVe on PersonaChat or GloVe from one of the files here https://nlp.stanford.edu/projects/glove/ ? |
The embedding file is provided in the source data. It is obtained from https://nlp.stanford.edu/projects/glove/ (seems to be glove.twitter.27B.zip) Sorry that I am busy with some deadlines and have no time to check your codes. If you still have any question about this repository, feel free to ask me. |
Any advice regarding why my model didn’t get improved after incorporating keyword? |
Any updates? |
Hi,
Thank you very much for sharing your work!
I have a few questions regarding evaluations for keyword predictions. I'm sorry that I may miss or misunderstand your code since I'm not familiar with Tensorflow.
For a given history of keywords, there can be multiple target keywords for the next turn. Do you minimize the negative log-likelihood losses for every target keyword? Is the batch loss averaged over batch size or the number of target keywords in the batch?
How did you compute the
correlation
metric? Greedy, average or max embedding? Do you just compute the correlation between the top-1 keyword with target keywords or top-k keywords? Do you average across target keywords before or after computing correlations?Any response will be appreciated.
Thanks,
Peixiang
The text was updated successfully, but these errors were encountered: