Estimate token usage + dry run option

[taylorjdawson]

Confirm this is a feature request for the Node library and not the underlying OpenAI API.

• This is a feature request for the Node library

Describe the feature or improvement you're requesting

There a couple nodejs tokenizers laying around npm land:

• gpt-tokenizer
• tiktoken

However, they have to be kept maintained and up to date with the official tiktoken. It feels very natural that such a tool be apart of this repository and updated regularly by the openai team. I feel that pre-request token estimation is vital to ensuring hobbyist developers don't unknowingly succumb to financial demise.

Why apart of this library?
Take function calling for example:

    const response = await openai.chat.completions.create({
        model: "gpt-3.5-turbo",
        messages: messages,
        functions: functions,
        function_call: "auto",  // auto is default, but we'll be explicit
    });

I could intercept this function call above to view what data is sent to chatGPT so I can figure out how many tokens this call takes with the message + the functions, pipe them into gpt-tokenizer and see how many tokens are used. But if it were to be built into the API itself then you can have a dry run option on your nodejs api functions that estimate cost and then the developer could determine in the logic if they want to proceed with the actual run.

What else could be very helpful would be to include some parameter whether in the configuration or when calling api functions such that if the tokens and/or usd cost exceeds a certain limit the call auto fails and doesn't actually get executed.

Additional context

No response

[rattrayalex]

Thanks for the thoughtful feature request, the function-calling example is a compelling one. We'll think about this.

Can you share an example of how you'd want to code the logic to proceed or not proceed based on the token count?

[eugene-kim]

I use function calling extensively. I'd love a simple function that returned a token count given completion input, e.g. ChatCompletionCreateParamsNonStreaming.

I want to ensure that I respect rate limits before I ever hit them. Rate limiting libraries such as Bottleneck can be configured to respect both the request and token limits, but I need the token counts before I make the request

[rattrayalex]

I don't expect to get to this anytime soon, but I would be interested in PR's implementing this that add the functionality to the lib/ folder!

[NatoBoram]

If anyone is stuck at this problem...

• https://github.com/ceifa/tiktoken-node is a node binding to the official Rust implementation
• https://github.com/niieani/gpt-tokenizer is a pure TypeScript reimplementation
• There's a feature request to publish tiktoken to npm at NPM package tiktoken#22