Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check if the model param is valid for moderations endpoint #437

Merged

Conversation

MunarYesen
Copy link
Contributor

Currently there are only two models available for OpenAI /moderations endpoint, which are text-moderation-stable and text-moderation-latest. In the official python library openai-python. The method Moderation will check if the model param is in VALID_MODEL_NAMES and returns an error when it's not.

 def _prepare_create(cls, input, model, api_key):
        if model is not None and model not in cls.VALID_MODEL_NAMES:
            raise ValueError(
                f"The parameter model should be chosen from {cls.VALID_MODEL_NAMES} "
                f"and it is default to be None."
            )

        instance = cls(api_key=api_key)
        params = {"input": input}
        if model is not None:
            params["model"] = model
        return instance, params

It would be great if we can add similar checks before passing the request to /v1/moderations endpoint.

@codecov
Copy link

codecov bot commented Jul 12, 2023

Codecov Report

Merging #437 (1b8d84d) into master (f028c28) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head 1b8d84d differs from pull request most recent head 1e82071. Consider uploading reports for the commit 1e82071 to get more accurate results

@@            Coverage Diff             @@
##           master     #437      +/-   ##
==========================================
+ Coverage   97.02%   97.03%   +0.01%     
==========================================
  Files          17       17              
  Lines         705      709       +4     
==========================================
+ Hits          684      688       +4     
  Misses         15       15              
  Partials        6        6              
Impacted Files Coverage Δ
moderation.go 100.00% <100.00%> (ø)

Comment on lines 30 to 40
// TestModerationsWithIncorrectModel Tests passing an incorrect model to Moderations request.
func TestModerationsWithIncorrectModel(t *testing.T) {
client, server, teardown := setupOpenAITestServer()
defer teardown()
server.RegisterHandler("/v1/moderations", handleModerationEndpoint)
_, err := client.Moderations(context.Background(), ModerationRequest{
Model: GPT3Dot5Turbo,
Input: "I want to kill them.",
})
checks.ErrorIs(t, err, ErrModerationInvalidModel)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MunaerYesiyan Thanks for PR! How about trying to write exhaustive tests for both valid and invalid models in table driven tests?

Copy link
Contributor Author

@MunarYesen MunarYesen Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Included valid and invalid model for moderation unit testing.
e579559

@vvatanabe
Copy link
Collaborator

Confirmed not supported just in case. 👍

$ curl https://api.openai.com/v1/moderations \
                    -H "Content-Type: application/json" \
                    -H "Authorization: Bearer $OPENAI_API_KEY" \
                    -d '{
                  "input": "I want to kill them.", "model": "text-moderation-001"
                }'
{

  "error": {
    "message": "Invalid value for 'model' = text-moderation-001. Please check the OpenAI documentation and try again.",
    "type": "invalid_request_error",
    "param": "model",
    "code": null
  }
}

@MunarYesen
Copy link
Contributor Author

Confirmed not supported just in case. 👍

$ curl https://api.openai.com/v1/moderations \
                    -H "Content-Type: application/json" \
                    -H "Authorization: Bearer $OPENAI_API_KEY" \
                    -d '{
                  "input": "I want to kill them.", "model": "text-moderation-001"
                }'
{

  "error": {
    "message": "Invalid value for 'model' = text-moderation-001. Please check the OpenAI documentation and try again.",
    "type": "invalid_request_error",
    "param": "model",
    "code": null
  }
}

And it's kinda confusing as the official document uses text-moderation-005 in the response example

moderation.go Outdated
@@ -18,6 +19,15 @@ const (
ModerationText001 = "text-moderation-001"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MunaerYesiyan I would like you to add a comment indicating that this constant is deprecated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deprecating the text-moderation-001 model in moderation.go e579559

@MunarYesen MunarYesen force-pushed the chore/check_moderation_model branch from 1b8d84d to 1e82071 Compare July 13, 2023 01:02
@vvatanabe vvatanabe merged commit e22a29d into sashabaranov:master Jul 13, 2023
3 checks passed
@vvatanabe
Copy link
Collaborator

@MunaerYesiyan Thanks for the pull request!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants