add cosine decay #184
Merged
add cosine decay #184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LDOUBLEV
reviewed
Jun 11, 2020
| @@ -41,3 +41,7 @@ Optimizer: | |||
| base_lr: 0.0005 | |||
| beta1: 0.9 | |||
| beta2: 0.999 | |||
| decay: | |||
| function: cosine_decay | |||
There was a problem hiding this comment.
config.md中可以考虑加decay参数的说明，以及文档中加decay的使用方法，默认有没有使用，能提升多少
LDOUBLEV
reviewed
Jun 11, 2020
| @@ -28,6 +28,16 @@ def AdamDecay(params, parameter_list=None): | |||
| base_lr = params['base_lr'] | |||
| beta1 = params['beta1'] | |||
| beta2 = params['beta2'] | |||
| if 'decay' in params: | |||
There was a problem hiding this comment.
配置参数文件里加了decay， 默认是使用decay的是吗？
There was a problem hiding this comment.
默认不使用，配置文件里添加了才会进行decay
LDOUBLEV
reviewed
Jun 12, 2020
| base_lr = fluid.layers.cosine_decay( | ||
| learning_rate=base_lr, | ||
| step_each_epoch=step_each_epoch, | ||
| epochs=total_epoch) |
There was a problem hiding this comment.
看这里是否要加个提示啥的，如果decay mode不是cosine_decay 采用的是不加decay的优化器
doc里的文档里也可以加一下支持哪些decay方式，可以加在备注里
| function(decay) | 设置decay方式 | cosine_decay | \ |
There was a problem hiding this comment.
decay_list = ["cosine_decay"]
if decay_mode not in decay_list:
print("decay_mode can only be one of {}, but get [{}], the training will running without decay".format(decay_list, decay_mode))
LDOUBLEV
approved these changes
Jun 15, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.