-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CrabNet matbench results - possibly neglecting 25% of the training data it could have used #19
Comments
Email response from @MahamadSalah74
|
@anthony-wang,
In the CrabNet matbench notebook, it does train/val/test splits. However, if #15 (comment) is correct such that the validation data (i.e.
val.csv
) doesn't contribute to hyperparameter tuning, then that 25% of the training data is essentially getting thrown away, correct?In other words, the CrabNet results are based on only 75% of the training data compared to what the other
![](https://camo.githubusercontent.com/e5c2f19319199eb51c5d8f41fa8ac9c19c5627a43faf81b16c53102367b4e330/68747470733a2f2f6861636b696e676d6174657269616c732e6c626c2e676f762f6175746f6d61746d696e65722f5f696d616765732f63765f6e65737465642e706e67)
matbench
models use for training. From what I understand, the train/val/test split in the context ofmatbench
only really makes sense if you're doing hyperparameter optimization in a nested CV scheme, as follows:(Source: https://hackingmaterials.lbl.gov/automatminer/advanced.html)
To correct this, I think all that needs to be done is change:
to
which makes it so there's data bleeding between
train_df
andval_df
, butval_df
ends up being essentially just a dummy dataset so that CrabNet doesn't error out when aval.csv
isn't available.Sterling
The text was updated successfully, but these errors were encountered: