-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't reproduce Robotcar results #45
Comments
Hi, Could you please let us know the complete process that you used to obtain these results? In particular, how you map the best match to a pose? Best, Tobias |
Ok - @StephenHausler - let's sit together at some point to find where the culprit lies. |
Hi, @StephenHausler, @Tobias-Fischer , some days ago I ran the Pittsburgh_WPCA4096 and MSLS_WPCA4096 models for RobotSeasons and obtained the following results with the NetVLAD retrieval: MSLS_WPCA4096: The overall I calculate it by doing the weighted mean of both numbers based on the number of images taken at day and night: overall = ( day * 9300 + night * 2634 ) / (9300 + 2634) For the Pittsburgh model the difference with the reported numbers seems reasonable to me (like what would happen between two different trainings), so I think that the model is probably fine and the problem lies in the Patch-NetVLAD feature extraction part. I hope this info helps with the issue. |
Hi, could you please tell me the dataset you ran Pittsburgh_WPCA4096 model is RobotSeasons V1 or V2? |
Hi @HeartbreakSurvivor, I ran RobotSeasons V2 |
Hi, the question is the RobotcarV1 has 9300 + 2634 = 11934 query images and the RobotCar v2 has 1872 query images, you said you ran on RobotSeasons V2 but calculate overall use this:
I dont't know why, but it doesn't matter. What I really wonder is that how you get these result? just follow the QuickStart in ReadMe.md file? |
Hi again @HeartbreakSurvivor
Thank you very much for pointing this out, it seems that I indeed mixed the two versions. The overall should be instead calculated as: I use indeed feature_extract.py and feature_match.py and then use the NetVLAD_predictions.txt file (I have not evaluated Patch-NetVLAD yet, only NetVLAD). You have to be careful with the format of the poses, as explained in the dataset readme, but the retrieval itself should be fine. |
thank you very much for the reply, I will check my code. |
Hi again @marialeyvallina it seems that I got the same problem with you. I ran the Pittsburgh_WPCA4096 model for RobotSeasons V1 and obtained the following results with the NetVLAD retrieval:
which seems reasonable to me.
I have test Pittsburgh_WPCA4096 on RobotCar Seasons V1 for twice just in case, but got the same result, the result is as follows. So I agree with your point, the problem maybe lies in PathchNetvlad feature extraction or feature match part. |
Hi, @StephenHausler and I will be looking at this. However the holiday season is coming up and we're tied with other projects. We haven't ever checked V1 as far as I remember. I'm assuming you guys are aware that the lower scores are better for NetVLAD (distances), but higher scores for Patch-NetVLAD (number of inliers)? So it needs an argmax instead of argmin to get the top1 match. |
Hi, I tried to reproduce your result on Robotcar Seasons V2 test set by submitting to the challenge submission server. I used the released performance-focused model which is pre-trained on MSLS dataset, but I got this incorrect result:
![image](https://user-images.githubusercontent.com/50984705/142757409-f80f53a7-99f3-41c5-b42f-7641f6c8a226.png)
![image](https://user-images.githubusercontent.com/50984705/142757440-df69b2da-7cba-4771-8838-bfbd98d9d3a7.png)
And I tried the model pre-trained on Pitts30k, the results are not correct either.
Besides, the results on other datasets is normal. Is the model version that I used is wrong? Could you possibly release the model state that achieves the results on Robotcat dataset shown in the paper? Or would you provide the results on test set split by conditions like the Supplementary Table 1? Thank you so much.
Best regards,
The text was updated successfully, but these errors were encountered: