Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start using ggml-alloc in SAM example #490

Merged
merged 1 commit into from
Aug 28, 2023
Merged

Conversation

YavorGIvanov
Copy link
Collaborator

@YavorGIvanov YavorGIvanov commented Aug 28, 2023

  • Remove scratch buffers from image encoder
  • This change makes the whole inference a bit slower. I will additionally investigate why this happens and speed it up again.
  • There are two additional ggml_allocr_alloc(..) calls, which should NOT be needed after the ggml_conv_transpose_2d_p0(..) operation. This will also be fixed additionally.

@ggerganov
Copy link
Owner

M2 Ultra speed:

 $make -j sam && time ./bin/sam -t 8 -i img.jpg
[ 28%] Built target common
[ 71%] Built target ggml
[100%] Built target sam
main: seed = 1693224265
main: loaded image 'img.jpg' (680 x 453)
sam_image_preprocess: scale = 0.664062
main: preprocessed image (1024 x 1024)
sam_model_load: loading model from 'models/sam-vit-b/ggml-model-f16.bin' - please wait ...
sam_model_load: n_enc_state      = 768
sam_model_load: n_enc_layer      = 12
sam_model_load: n_enc_head       = 12
sam_model_load: n_enc_out_chans  = 256
sam_model_load: n_pt_embd        = 4
sam_model_load: ftype            = 1
sam_model_load: qntvr            = 0
operator(): ggml ctx size = 202.32 MB
sam_model_load: ...................................... done
sam_model_load: model size =   185.05 MB / num tensors = 304
embd_img
dims: 64 64 256 1 f32
First & Last 10 elements:
-0.05117 -0.06408 -0.07154 -0.06991 -0.07212 -0.07690 -0.07508 -0.07281 -0.07383 -0.06779 
0.01589 0.01775 0.02250 0.01675 0.01766 0.01661 0.01811 0.02051 0.02103 0.03382 
sum:  12736.272313

Skipping mask 0 with iou 0.705935 below threshold 0.880000
Skipping mask 1 with iou 0.762136 below threshold 0.880000
Mask 2: iou = 0.947081, stability_score = 0.955437, bbox (371, 436), (144, 168)


main:     load time =    51.28 ms
main:    total time =  2047.49 ms

real	0m2.068s
user	0m16.343s
sys	0m0.214s

Comment on lines +316 to +321
static void ggml_disconnect_node_from_graph(ggml_tensor * t) {
t->op = GGML_OP_NONE;
for (int i = 0; i < GGML_MAX_SRC; i++) {
t->src[i] = NULL;
}
}
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future, this should become ggml_disconnect part of ggml

@ggerganov ggerganov merged commit 69bf842 into master Aug 28, 2023
2 checks passed
@ggerganov ggerganov deleted the ggml-alloc-sam.cpp branch August 28, 2023 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants