Replies: 3 comments 3 replies
-
These benchmarks are great, but how do these benchmarks compare to other state-of-the-art models? For example, your ViT has a 98% validation accuracy on EuroSAT, but Google reported a 99.2% accuracy on EuroSAT using a simple ResNet-50 model 5 years ago. I would recommend against using saturated benchmarks like EuroSAT for which even simple ImageNet weights can easily achieve 98%+ accuracy. There are far better, larger, and newer benchmark suites out there (see Table 2a of this paper). |
Beta Was this translation helpful? Give feedback.
-
@srmsoumya is it documented what resources were used for fine-tuning, similar to the Training Card for training the Clay model from scratch? My assumption is that this was done single node with one p5.48x instance but not sure if the memory footprint of the fine-tuning is smaller because the encoder is frozen and if smaller instances can be used. https://clay-foundation.github.io/model/release-notes/specification.html#training-card |
Beta Was this translation helpful? Give feedback.
-
@rbavery For fine-tuning, we don't need larger VM instances like We can also use smaller instances by adjusting the batch size accordingly. |
Beta Was this translation helpful? Give feedback.
-
Experiment Overview
We have conducted experiments with the Clay model on various downstream tasks, specifically focusing on classification and segmentation. Note that in all cases, the Clay encoder remains frozen, and only the additional layers are trained.
Initial Observations
The Clay model shows strong learning capabilities, with most tasks being learned effectively within the first epoch, after which performance plateaus.
Classification Task
For the classification task, we added a fully connected (FC) block on top of the Clay encoder. After 5 minutes of training, the model achieved a training accuracy of 0.985 and a validation accuracy of 0.98. The loss curves indicate that most of the learning occurs within the first epoch.
Training statistics
![image](https://private-user-images.githubusercontent.com/8049519/337409204-34287f8c-2cb7-4947-8b0f-9a987fb1cf22.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTAzODcsIm5iZiI6MTcyMzk5MDA4NywicGF0aCI6Ii84MDQ5NTE5LzMzNzQwOTIwNC0zNDI4N2Y4Yy0yY2I3LTQ5NDctOGIwZi05YTk4N2ZiMWNmMjIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQwODA3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Njg5YjY2MzdiNWZjMTA0MjhlM2YwN2IyOTExMWI0NzZhMGUxZWRiMWU2MTU2NjdhN2UzZWM3MmUwNmE5N2JlZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ye9juxNJoAnuZFzUmh2A8fhwhIcdba6FfJmCN_-hrhk)
Validation statistics
![image](https://private-user-images.githubusercontent.com/8049519/337409611-448b79dd-490f-41bb-b670-216d3cc040db.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTAzODcsIm5iZiI6MTcyMzk5MDA4NywicGF0aCI6Ii84MDQ5NTE5LzMzNzQwOTYxMS00NDhiNzlkZC00OTBmLTQxYmItYjY3MC0yMTZkM2NjMDQwZGIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQwODA3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTMzYTFkMzMzNDZmYjBlYzIzYmRkMzEwYzI3YmVmN2FhZjRlYjk0MzRmM2VmY2M3ZjJhNGRhMmVkNDgzZGE2OSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.6BR5w_i8vnWIVWlM9qJvdor-RE-2KU3KbGY5s9EkTI8)
Dataset Used: EuroSAT
Dataset Citation:
Segmentation Task
For the segmentation task, we tested the model on the Chesapeake Bay CVPR dataset. We attached a decoder similar to Segformer, which extracts features from intermediate layers and fuses them to predict segmentation masks.
We used a subset of the dataset consisting of 2000 random samples for training and validation. After 10 minutes of training (10 epochs), the model showed a similar learning pattern, with most learning occurring in the first epoch. The validation scores were a weighted IOU of 0.875 and an F1 score of 0.93.
Training statistics
![image](https://private-user-images.githubusercontent.com/8049519/337407710-c240f5bf-86d1-4fd8-81a0-af63d4efc2e4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTAzODcsIm5iZiI6MTcyMzk5MDA4NywicGF0aCI6Ii84MDQ5NTE5LzMzNzQwNzcxMC1jMjQwZjViZi04NmQxLTRmZDgtODFhMC1hZjYzZDRlZmMyZTQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQwODA3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NWQxM2Q2YmMwNWNkOWYyYjYxOTJlNjU5MDYxYzI4MWJjNDgxYzMzY2E4Nzk1Y2ViZTA0YTlmMjc2ZTYzYTU5MiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.wDkEsWWMhibZaZcoJT-uWqTBMuYLFXBGneZRYFpvRwk)
Validation statistics
![image](https://private-user-images.githubusercontent.com/8049519/337408110-ecb7fc72-b703-42bb-9bb7-8a338157e911.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTAzODcsIm5iZiI6MTcyMzk5MDA4NywicGF0aCI6Ii84MDQ5NTE5LzMzNzQwODExMC1lY2I3ZmM3Mi1iNzAzLTQyYmItOWJiNy04YTMzODE1N2U5MTEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQwODA3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9M2Q3MjQzMGQ3NjIzOWUyOGY0N2U3MWRhYjNlZTA4ZWE0M2FhZmNkMTU0MTMwYTc1Y2RjZTA2YTc2ODAyMGFhZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.SkXRcjPin4J7XnByETE3QeP4rEIAFrQSFi6tSbx2Kws)
Prediction on sample masks: Image / Actual Mask / Prediction Mask
![image](https://private-user-images.githubusercontent.com/8049519/337406945-7a914f04-e869-4e20-aec6-74fcf667a7d8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTAzODcsIm5iZiI6MTcyMzk5MDA4NywicGF0aCI6Ii84MDQ5NTE5LzMzNzQwNjk0NS03YTkxNGYwNC1lODY5LTRlMjAtYWVjNi03NGZjZjY2N2E3ZDgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQwODA3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjY0MzFkMWVmZmMxNzdmZGMyYzM1MTQxZmI4MWI1NmM3OGI0ZWIwYjJmN2ZkOTdkMGQ1ZjkyYjdiODBmMzliMyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.nptuK3tYIAEQzKJ_rXo3OYMkGSNxkeiu__Y3FRPk_XQ)
Dataset Used: Chesapeake Bay CVPR
Dataset Citation:
Decoder Reference:
Next Steps
Beta Was this translation helpful? Give feedback.
All reactions