Decoding object context

[ruinianxu]

Hi all,

I am confused about the code doing object context decoding. I attached the related code of decoder_rnn.py:

labels_to_embed = labels[start_ind:end_ind].clone()
# Whenever labels are 0 set input to be our max prediction
nonzero_pred = pred_dist[:, 1:].max(1)[1] + 1
is_bg = (labels_to_embed.data == 0).nonzero()
if is_bg.dim() > 0:
    labels_to_embed[is_bg.squeeze(1)] = nonzero_pred[is_bg.squeeze(1)]
out_commitments.append(labels_to_embed)
previous_embed = self.obj_embed(labels_to_embed+1)

First of all, based on the paper, LSTM will take the previous object class prediction as input. But based on this code, if there is no bg (which is background), nonzero_pred won't be used to update labels_to_embed which is used to extract embeddings from self.obj_embed. Meanwhile in most of cases, there is no bg category. This code seems take the object category probability distribution predicted by faster-rcnn as input.

Please let me know if I understand it correctly.

[dongxinfeng1]

Hi, I've been working on this code lately.  Can you run this code without bugs? I followed the author's requirements for environment and dataset except the Torch version is 0.3.1, but encountered some problems.
 

[ruinianxu]

Yes, I do. I will recommend you to use PyTorch under 0.4 since one of his implementations is based on that and it was officially dumped after PyTorch 0.4.

[dongxinfeng1]

Thanks for your response, I'll try it later.