You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I have a question about how to reproduce the model on mmdetection 3.x. The model returns slightly different outputs even after fixing all seeds through mmdetection:
Setting "randomness.seed" and "randomness.deterministic" will invoke the function defined in the mmengine library:
def set_random_seed(seed: Optional[int] = None,
deterministic: bool = False,
diff_rank_seed: bool = False) -> int:
"""Set random seed.
Args:
seed (int, optional): Seed to be used.
deterministic (bool): Whether to set the deterministic option for
CUDNN backend, i.e., set `torch.backends.cudnn.deterministic`
to True and `torch.backends.cudnn.benchmark` to False.
Defaults to False.
diff_rank_seed (bool): Whether to add rank number to the random seed to
have different random seed in different threads. Defaults to False.
"""
if seed is None:
seed = sync_random_seed()
if diff_rank_seed:
rank = get_rank()
seed += rank
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
# torch.cuda.manual_seed(seed)
if is_cuda_available():
torch.cuda.manual_seed_all(seed)
elif is_musa_available():
torch.musa.manual_seed_all(seed)
# os.environ['PYTHONHASHSEED'] = str(seed)
if deterministic:
if torch.backends.cudnn.benchmark:
print_log(
'torch.backends.cudnn.benchmark is going to be set as '
'`False` to cause cuDNN to deterministically select an '
'algorithm',
logger='current',
level=logging.WARNING)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
if digit_version(TORCH_VERSION) >= digit_version('1.10.0'):
torch.use_deterministic_algorithms(True)
return seed
However, this model and BARON show its Faster-RCNN model returns different rpn results upon each run. TwoStageDetector in MMDetection contains a function:
def extract_feat(self, batch_inputs: Tensor) -> Tuple[Tensor]:
"""Extract features.
Args:
batch_inputs (Tensor): Image tensor with shape (N, C, H ,W).
Returns:
tuple[Tensor]: Multi-level features that may have
different resolutions.
"""
x = self.backbone(batch_inputs)
if self.with_neck:
x = self.neck(x)
return x
The "self.backbone" is a resnet model saved in mmdet/models/backbones/resnet.py. The ResNet model produces different results during forward():
def forward(self, x):
"""Forward function."""
if self.deep_stem:
x = self.stem(x)
else:
x = self.conv1(x)
x = self.norm1(x)
x = self.relu(x)
x = self.maxpool(x)
outs = []
for i, layer_name in enumerate(self.res_layers):
res_layer = getattr(self, layer_name)
x = res_layer(x)
if i in self.out_indices:
outs.append(x)
return tuple(outs)
I checked that the "x" was the same every time.
x = self.conv1(x)
I checked the log and the conv1 produced different results for the same inputs. The model does calculate the same total losses every time, but the backward process changes the parameters differently even from the same loss.
How can I guarantee the rpn results to be consistent?
Thanks for your response in advance!
The text was updated successfully, but these errors were encountered:
[First run]
![image](https://private-user-images.githubusercontent.com/39285147/333828122-72d645e8-7c76-482e-848e-e22f5a2fcec7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5MDA3OTgsIm5iZiI6MTcyMzkwMDQ5OCwicGF0aCI6Ii8zOTI4NTE0Ny8zMzM4MjgxMjItNzJkNjQ1ZTgtN2M3Ni00ODJlLTg0OGUtZTIyZjVhMmZjZWM3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE3VDEzMTQ1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBiN2JiNTRiMDQyMjc4NjFjZDBiYjBmOGM2MWM4NDVjZDk0NTBkMDRjZDcxMzZiMjkzZTI5Yzg1ZjZiMzQzNDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.7VPao2Q8ZpWVU6L2HsKe1oaoP6Vb-FfyVT4qGf3vnvM)
[Second run]
![image](https://private-user-images.githubusercontent.com/39285147/333828133-aa50633b-860f-4584-837e-fba745e565d5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5MDA3OTgsIm5iZiI6MTcyMzkwMDQ5OCwicGF0aCI6Ii8zOTI4NTE0Ny8zMzM4MjgxMzMtYWE1MDYzM2ItODYwZi00NTg0LTgzN2UtZmJhNzQ1ZTU2NWQ1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE3VDEzMTQ1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcyOWFkYmQ5ZWE5ODJiYjc3YTlkODk3NzU4ZGNhNzljN2E3YzM5NWU0ZGUyZTE4OTczN2RkYzY1NjhiODIzZmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.YwJ7UzNnye7hP_Ktkk7_HfKHqTHp-k_-Gmp6Q4B1JFU)
Hello, I have a question about how to reproduce the model on mmdetection 3.x. The model returns slightly different outputs even after fixing all seeds through mmdetection:
Setting "randomness.seed" and "randomness.deterministic" will invoke the function defined in the mmengine library:
However, this model and BARON show its Faster-RCNN model returns different rpn results upon each run. TwoStageDetector in MMDetection contains a function:
The "self.backbone" is a resnet model saved in mmdet/models/backbones/resnet.py. The ResNet model produces different results during forward():
I checked that the "x" was the same every time.
I checked the log and the conv1 produced different results for the same inputs. The model does calculate the same total losses every time, but the backward process changes the parameters differently even from the same loss.
How can I guarantee the rpn results to be consistent?
Thanks for your response in advance!
The text was updated successfully, but these errors were encountered: