Situated Instruction Following
So Yeon Min, Xavi Puig, Devendra Singh Chaplot, Tsung-Yen Yang, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov, Yonatan Bisk, and Roozbeh Mottaghi
Carnegie Mellon University, FAIR (Meta)

In proceedings of ECCV 2024

Project Webpage

• SIF tasks
• The code to evaluate Reasoner (LLM + mapping baseline) on the SIF task.
• We provide code of the model, and users can use their own API keys.

We DO NOT have any model parameters.
We ONLY provide code and data to run upon the user providing their own OpenAI API KEY. Please use your own OpenAI API key (instructions below).

Please run

conda create -n sif_env python=3.9 cmake=3.14.0 -y
conda activate sif_env

git clone https://github.com/soyeonm/SIF_github.git
export HOME_DIR=$(pwd)
cd SIF_github
export SIF_DIR=$(pwd)
sh setup/install.sh

Please have your huggingface datasets username and read token ready. Now, run

sh setup/download_data.sh

to download SIF data. You will be prompted to input your hugging face username and read token (as password) when

git clone https://huggingface.co/datasets/fpss/fphab

this line runs in the above file.

This will make your data directory

SIF/
  data/
    datasets/
      sif_release/
        jsons/
          pnp_val_seen.json.gz
          ...
        room_annotations/
          ...

Now, to download object assets, do the following (taken from habitat repo).

1. Download Amazon and Google object archives
2. Extract these two object datasets into habitat-lab/data as follows:

cd objects
tar -xvf ~/Downloads/google_object_dataset.tar.gz
tar -xvf ~/Downloads/amazon_berkeley.tar.gz

You will see your data directory like this:

SIF/
  data/
    armatures/
    fpss/
       hssd-hab.scene_dataset_config.json
       ...
    humanoids/
    objects/
       google_object_dataset/
       amazon_berkeley/
       ycb/
    robots/
    versioned_data/
    default.physics_config.json 
    ...

To test that your setup is correct, run

export MAGNUM_LOG=quiet   
export HABITAT_SIM_LOG=quiet 
python main.py --task_config config/s_obj/val_seen.yaml  --print_images 1 --exp_name  test --gt_sem_seg --magic_man_if_exists  --eps_to_run ''  --run_full_task --oracle_baseline

and see that it runs.

We DO NOT provide the model parameters itself. To use Reasoner with Open AI's API, please add

export OPENAI_API_KEY= #your api key

to ~/.bashrc, and run

source  ~/.bashrc

For running inference with Reasoner, with OpenAI API:

export Detic_directory=$HOME_DIR/Detic
export MAGNUM_LOG=quiet   
export HABITAT_SIM_LOG=quiet 
HABITAT_SIM_LOG=quiet python main.py --task_config config/s_obj/val_seen.yaml  --print_images 1 --exp_name reasoner_vs_s_obj  --gt_sem_seg --magic_man_if_exists    --eps_to_run ''  --run_full_task 

Argument explanation:

Arguments

--task_config: The config to use.

--exp_name: The saving directory of model output ("tmp/dump/exp_name").

--llm_type: LLM used for inference (openai/ openai_chat). Use openai for Reasoner and openai_chat for prompter.

--print_images: Show image visualization.

To run with GT semantic segmenation, use " --gt_sem_seg". To run with GT manipulation, use "--magic_man_if_exists". Use both for "Oracle Perception" in Table 6 of the paper, and neither for "Learned Perception".

To run with Prompter (instead of Reasoner), add "--prompter_baseline".

e.g.

export Detic_directory=$HOME_DIR/Detic
export MAGNUM_LOG=quiet   
export HABITAT_SIM_LOG=quiet 
HABITAT_SIM_LOG=quiet python main.py --task_config config/s_obj/val_seen.yaml  --print_images 1 --exp_name reasoner_vs_s_obj  --gt_sem_seg --magic_man_if_exists    --eps_to_run ''  --run_full_task --prompter_baseline

To run with oracle policy, add "--oracle_baseline".

Logs and visualiations are saved to "tmp/dump/exp_name":

For the example command above,

Run

python utils/evaluate_model_output.py --exp_name reasoner_vs_s_obj   --json_name  s_obj_val_seen

to obtain the SR/ SPL. Add flag "--get_amb_clear" to see results by clear/ ambiguous tasks. The "--json_name" can be accessed by looking at the "data_path" of the config file you used (e.g. config/s_obj/val_seen.yaml).

This repository uses Habitat 3.0 implementation for simulating human and spot robot trajectory.