Reuse gpt2_graph

[datduonguva]

On the gpt-2 example, I can see that during inference, each step invokes a gpt2_eval() function, which in turn, invokes gpt2_graph() function to recreate the graph.

Why can't we create just once and reuse it?

[ggerganov]

Even though the nodes in the graph are the same type for each invocation (there are exceptions though), the tensor dimensions do change. Mainly because the size of the tensors in the attention are a function of the number of tokens and also the input number of tokens in the batch can be different. In certain scenarios, it could be beneficial to pre-init the graphs (lets say, a graph for each possible n_past and a single new token) at the start in order to avoid creating them at runtime.

[datduonguva]

thank you!