-
Notifications
You must be signed in to change notification settings - Fork 83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation of the proximal operator #383
Comments
@AnchorBlues it's been a while that I dug into the math. Could you share a screenshot where it shows that the learning rate matters in the link you shared? Let me also solicit opinions from @pavanramkumar and @titipata I think what is happening is that |
The following site may be easier to understand. https://www.stat.cmu.edu/~ryantibs/convexopt-S15/scribes/08-prox-grad-scribed.pdf The target section is 8.1.3.( As the formula (8.11) and (8.12) shows, the optimization of parameters of the Lasso is executed as follows (The same method applies to the Elastic-Net):
I updated the version of Converged in 443 iterations. |
humm ... indeed you are right. Have you verified that the fix solves the comparison problem with |
Yes, I have verified it.
I made a pull request. |
Thanks @AnchorBlues for the careful verification and fix! |
closed by #384 |
In the source code, the proximal operator is defined as follows:
The second argument is the threshold of the proximal operator.
As this site, the threshold should be the product of the learning rate and the L1 regularization parameter.
However, this method is called with feeding the second argument with only the L1 regularization parameter(
reg_lambda * alpha
) as follows.I think
reg_lambda * alpha
must be replaced withlearning_rate * reg_lambda * alpha
. Otherwise, the model with L1 regularization will not be correctly trained whenlearning_rate
is not1
.In fact, the trained result of
GLM
could be quite different from the one ofsklearn
when the learning rate ofGLM
is NOT1
.The text was updated successfully, but these errors were encountered: