-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The projection head order needs to be be relooked #17
Comments
Do you know why use GELU here? @Akshay1-6180 |
so based on experiments it was found that GELU has a significantly smoother gradient transition and its not abrupt or sharp like relu , if u look at both the functions u would understand. https://github.com/openai/gpt-2/blob/master/src/model.py |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Going through these papers
I feel the order should be this
The text was updated successfully, but these errors were encountered: