EMNLP'19 Paper: Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

Authors: Khanh Nguyen, Hal Daumé III

UPDATE Oct 15, 2019: fix a bug in the validation code that prevented the code from reproducing results in the paper.

HANNA is an interactive photo-realistic simulator that mimics an agent fulfilling object-finding tasks by leveraging natural language-and-vision assistance.

An example HANNA task.

Comparing HANNA with VLN (Anderson et al., 2018b), EQA (Wijmans et al., 2019), VNLA (Nguyen et al., 2019), CVDN (Thomason et al., 2019).

0. git clone --recursive https://github.com/khanhptnk/hanna.git (don't forget the recursive flag!)
1. Download data.
2. Setup simulator.
3. Run experiments.

If you use the code or data in this repo, please cite our paper using the following bibtex code

@inproceedings{nguyen2019hanna,
  author = {Nguyen, Khanh and Daum{\'e} III, Hal},
  title = {Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning},
  booktitle = {Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  month = {November},
  year = {2019},
}