Are there plans to develop CONV_2D? this is also a VIP

[Rane2021]

No description provided.

[narain1]

its already in development have a look at this #74

[ggerganov]

#74 is not in a working condition yet.
I will get back to this soon and add CONV_2D operator

[Rane2021]

great work! expecting!

[monatis]

Hi, is this planned for any time soon, or should I give it a try myself? I want to implement clip.cpp as CLIP is frequently used in large multimodal models, and this seems to be the only blocker.

[ggerganov]

Please go ahead

This is planned, but there are a few high-priority things that will probably take me a week before I get back to this.
Note that currently we have a very limited support for operators with more than 2 arguments, so for CONV_1D I implemented a couple of hard-coded calls with fixed padding and stride:

ggml/include/ggml/ggml.h

Lines 708 to 721 in 94a24c9

	// padding = 1
	// TODO: we don't support extra parameters for now
	// that's why we are hard-coding the stride, padding, and dilation
	// not great ..
	GGML_API struct ggml_tensor * ggml_conv_1d_1s(
	struct ggml_context * ctx,
	struct ggml_tensor * a,
	struct ggml_tensor * b);
	GGML_API struct ggml_tensor * ggml_conv_1d_2s(
	struct ggml_context * ctx,
	struct ggml_tensor * a,
	struct ggml_tensor * b);

Either follow the same pattern, or if you need a more general-purpose CONV_2D, see how we pass extra parameters for example in ggml_rope() and do the same thing:

ggml/include/ggml/ggml.h

Lines 687 to 699 in 94a24c9

	// rotary position embedding
	// in-place, returns view(a)
	// if mode & 1 == 1, skip n_past elements
	// if mode & 2 == 1, GPT-NeoX style
	// TODO: avoid creating a new tensor every time
	GGML_API struct ggml_tensor * ggml_rope(
	struct ggml_context * ctx,
	struct ggml_tensor * a,
	int n_past,
	int n_dims,
	int mode);

[monatis]

Thanks for the hints and pointing me to the right direction! Let's see what I can achieve with it.

[YavorGIvanov]

I think conv_2d has been finished and its interface generalized. Further optimizations of the operations will be done in a separate issue.