[pytorch] Add strong Wolfe line search for lbfgs #8824
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fehiepsi
requested review from
apaszke,
colesbury,
ezyang,
gchanan,
soumith and
zdevito
as code owners
fehiepsi
commented
Jun 23, 2018
|
|
||
| # store new direction/step | ||
| old_dirs.append(y) | ||
| old_stps.append(s) | ||
| ro.append(1. / ys) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
fehiepsi
commented
Jun 23, 2018
| else: | ||
| t = lr | ||
|
|
||
| # directional derivative | ||
| gtd = flat_grad.dot(d) # g * d | ||
|
|
||
| # directional derivative is below tolerance | ||
| if gtd > -tolerance_change: | ||
| break |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
fehiepsi
commented
Jun 23, 2018
| break | ||
|
|
||
| if d.mul(t).abs_().sum() <= tolerance_change: | ||
| # lack of progress | ||
| if d.mul(t).abs().max() <= tolerance_change: |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
fehiepsi
commented
Jun 23, 2018
| @@ -106,16 +278,18 @@ def step(self, closure): | |||
| state['func_evals'] += 1 | |||
|
|
|||
| flat_grad = self._gather_flat_grad() | |||
| abs_grad_sum = flat_grad.abs().sum() | |||
| opt_cond = flat_grad.abs().max() <= tolerance_grad | |||
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
fehiepsi
commented
Jun 23, 2018
| min_pos = x1 - (x1 - x2) * ((g1 + d2 - d1) / (g1 - g2 + 2 * d2)) | ||
| return min(max(min_pos, xmin_bound), xmax_bound) | ||
| else: | ||
| return (xmin_bound + xmax_bound) / 2. |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
fehiepsi
commented
Jun 23, 2018
test/test_optim.py
Outdated
| lambda params: optim.LBFGS(params, lr=5e-2, max_iter=5), | ||
| wrap_old_fn(old_optim.lbfgs, learningRate=5e-2, maxIter=5) | ||
| lambda params: optim.LBFGS(params, lr=1, max_iter=5), | ||
| wrap_old_fn(old_optim.lbfgs, learningRate=1, maxIter=5) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This was referenced Jun 23, 2018
|
I'll review this soon. |
|
cc: @ssnl |
jeffreyksmithjr
added
triaged
vincentqb
reviewed
Jun 11, 2019
vincentqb
reviewed
Jun 11, 2019
vincentqb
reviewed
Jun 11, 2019
vincentqb
reviewed
Jun 11, 2019
vincentqb
reviewed
Jun 11, 2019
|
Thanks for reviewing @vincentqb ! |
vincentqb
approved these changes
Jun 12, 2019
There was a problem hiding this comment.
@vincentqb is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@vincentqb merged this pull request in ad73ea2. |
nghorbani
pushed a commit
to nghorbani/human_body_prior
that referenced
this pull request
Aug 9, 2019
This has been copied from pytorch/pytorch#8824 PyTorch 1.2.0 has already merged this pull-request and we will incorporate it from the official repository in our code in near future.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds a line search for lbfgs. "strong Wolfe" is the default line search method in minFunc and it is also recommended in the Numerical Optimization book.
The implementation is based on four sources:
The 'lua' version is based on an old version of
minFunc, which has been updated in 2012. I made a couple of small changes based on the updated version. Due to that, the test of comparing with.luaversion is not consistent (that's is the reason I changed a learning rate in the test).Differential Revision: D15740107