sam bad result

[spacewalk01]

Hi thank you for your wonderful implementation.
The command I used:
main.exe -t 16 -i img.jpg

I am getting following problem when executing sam on windows:

main: seed = 1694161295
main: loaded image 'img.jpg' (680 x 453)
sam_image_preprocess: scale = 0.664062
main: preprocessed image (1024 x 1024)
sam_model_load: loading model from 'F:/models/sam/ggml/ggml-model-f16.bin' - please wait ...
sam_model_load: n_enc_state      = 768
sam_model_load: n_enc_layer      = 12
sam_model_load: n_enc_head       = 12
sam_model_load: n_enc_out_chans  = 256
sam_model_load: n_pt_embd        = 4
sam_model_load: ftype            = 1
sam_model_load: qntvr            = 0
operator (): ggml ctx size = 202.32 MB
sam_model_load: ...................................... done
sam_model_load: model size =   185.05 MB / num tensors = 304
embd_img
dims: 64 64 256 1 f32
First & Last 10 elements:
-0.05101 -0.06350 -0.07115 -0.06838 -0.06825 -0.06971 -0.07147 -0.07088 -0.06774 -0.05427
0.01575 0.01772 0.02244 0.01668 0.01755 0.01668 0.01795 0.02048 0.02101 0.03388
sum:  12757.156252

Skipping mask 0 with iou 0.562130 below threshold 0.880000
Skipping mask 1 with iou 0.638328 below threshold 0.880000
Skipping mask 2 with iou 0.620674 below threshold 0.880000

It yields no result image and when I lowered the iou threshold, the result image is generated but the detected mask is no good.

[YavorGIvanov]

I am going to work on this today in the https://github.com/YavorGIvanov/sam.cpp repository. This is the reason the ggml version in my repository is not updated to the latest. There is a problem when using the latest ggml version. Will update the example once I fix the problem in sam.cpp.

Meanwhile you can test it by going back a couple of commits (e.g. 7b5fcf5), but I expect to fix it today.

[spacewalk01]

I see so you mean there is an incompatibility issue between sam-cpp and ggml latest version. okay I will try it with the old ggml the one you used in your repo. Thank you for your reply

[spacewalk01]

with old version it works well. Looking forward to your update in this version.

[YavorGIvanov]

I am still trying to figure out exactly the reason why my change actually fixes the wrong output, but I still decided to fix it until I figure it out, so now the example output should be fine. May need some pointers by @slaren on this for where to look. I am trying to debug the library and allocator.

It is interesting that removing other inplace operations in the mask decoder doesn't fix the issue, but only removing this one - does fix it. More details in the commit.

[slaren]

After ggerganov/llama.cpp#2874, ggml-alloc will respect _inplace ops, ie. if you use an _inplace op, it will always happen inplace now. That wasn't the case before. I tried to look through the code to see if there are any tensors being overwritten unintentionally, but there are too many _inplace ops for me to follow exactly what is the logic. I would suggest removing all manual _inplace ops unless they are strictly necessary for correctness, and instead let ggml-alloc make them inplace automatically to save memory when possible. If that still doesn't work, it is possible that there may be issues with the auto-inplace logic in ggml-alloc, and to test that you can disable it by making ggml_op_can_inplace in ggml-alloc.c always return false.

[YavorGIvanov]

Ok. Thanks. I can remove the not needed inplaces as I added them initially only to boost the inference speed a bit. I will bench after I remove them though. Removing them all will surely fix the problem. Currently I remove only 1 add_inplace and it got fixed, but it was probably random.

[spacewalk01]

Hi, @YavorGIvanov I am looking forward to gpu implementation of sam. Do you have a plan to implement it in near future. Thanks

[YavorGIvanov]

Yes. I do. Will do first some attempts to optimize the inference as is, then quantizaiton and then CUDA/Metal support. I will try to finish these things by the end of this month.