Add recursive CCS #57
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
norabelrose
requested changes
Feb 13, 2023
| and data[0].dtype == data[1].dtype == self.dtype | ||
| ), "Data must be a tuple of two tensors of the same shape and dtype" | ||
|
|
||
| def correct_dtypes( |
There was a problem hiding this comment.
Why is this necessary at all? I don't see why we need to cast to a single dtype
There was a problem hiding this comment.
When removing this cast I get
Traceback (most recent call last):
File "/home/ubuntu/elk/elk/extensions/recursive_ccs/train.py", line 110, in <module>
train(args)
File "/home/ubuntu/elk/elk/extensions/recursive_ccs/train.py", line 74, in train
probe, train_loss = rccs.fit_next_probe(
File "/home/ubuntu/elk/elk/extensions/recursive_ccs/rccs.py", line 33, in fit_next_probe
train_loss = ccs.fit(data, **train_params)
File "/home/ubuntu/elk/elk/training/ccs.py", line 142, in fit
loss = self.train_loop_lbfgs(x0, x1, num_epochs, weight_decay)
File "/home/ubuntu/elk/elk/training/ccs.py", line 238, in train_loop_lbfgs
optimizer.step(closure)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/optim/lbfgs.py", line 312, in step
orig_loss = closure()
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/elk/elk/training/ccs.py", line 224, in closure
logit0, logit1 = self(x0), self(x1)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/elk/elk/training/ccs.py", line 103, in forward
return self.probe(x)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/nn/modules/container.py", line 204, in forward
input = module(input)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/miniconda3/envs/elk/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype
There was a problem hiding this comment.
If you just call .float() on the input before calling fit this should work
| @@ -0,0 +1,110 @@ | |||
| import csv | |||
There was a problem hiding this comment.
It's not clear why this entire file is necessary. It seems to be mostly copied over from the primary train.py. Could we add a flag or a subparser to the main command instead?
There was a problem hiding this comment.
It makes training more complicated. I think this should not happen before the first release.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.