-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Validation procedure #17
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
According to your notebook, you create your validation set based on a sample of 20% of the initial data set:
df_ml_conso_for_model, df_ml_conso_validation = train_test_split(df_ml_conso, test_size=0.2, random_state=42)
If this is the case, you validate your final models on the same subjects you trained it on. While not exactly the same samples are used for validation, the samples come from the same subject and would therefore bias your results.
Validating your model on unseen subjects from the same dataset, I got a mean precision of 0.63 (SD=0.19), mean recall of 0.97 (SD=0.06) and mean F1 of 0.75 (SD=0.15). See also image below. The low precision indicates that a lot of windows will be marked good when they are not.
![image](https://private-user-images.githubusercontent.com/52130886/270631024-0b23c883-9ea9-4587-8f6b-57e4c1ff6d8b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxNDc2ODIsIm5iZiI6MTcyMjE0NzM4MiwicGF0aCI6Ii81MjEzMDg4Ni8yNzA2MzEwMjQtMGIyM2M4ODMtOWVhOS00NTg3LThmNmItNTdlNGMxZmY2ZDhiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI4VDA2MTYyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU0NjUyYzhlNWI3ZjUwYzhmODcyYTM0ZDgxZjdjODIxNjZmNzU4MmU1MzNjYWY3MjFmMTY0MjJlMmMyM2ZiZGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ajBRaxLiZZjpYIBaJ0qa6jCIlOd_0iIFBIUoBAkzerc)
The text was updated successfully, but these errors were encountered: