Are derived shapes possible?

[LoganDark]

As a part of implementing RWKV sequence mode, I want to do what basically amounts to ggml_get_rows(ctx, model.emb, tokens), but where tokens could possibly be any length.

ggml stops there; this isn't allowed. If the length of the tensor at computation time is different than the length at creation time, ggml throws an assert. This means that I need a new computation graph for each length of sequence, because I need to call ggml_get_rows again.

Is this something that ggml plans to, or even can, support? I know with its style of allocation, this might not be possible without bailing each tensor out into its own allocation, but still.

[LoganDark]

Actually, closing this because I can destroy and rebuild a graph when needed (they are cheap).

[ggerganov]

Yes, for now I think we should just rebuild the graph in such cases.
Maybe in the future we will come up with an alternative solution, but I think this is good since it is very cheap overall

[LoganDark]

Yes, for now I think we should just rebuild the graph in such cases. Maybe in the future we will come up with an alternative solution, but I think this is good since it is very cheap overall

I mean the recursive find-parents that takes upwards of 2 seconds would like to have a word, but yeah other than that fine