IndexError: index 909 is out of bounds for dimension 0 with size 900

[WanFeng123313]

我在我自己的数据集上训练，完整报错如下：
Traceback (most recent call last):
File "/mnt/lvm_data/project/xyguo/code_dmx/Open-GroundingDino/main.py", line 372, in
main(args)
File "/mnt/lvm_data/project/xyguo/code_dmx/Open-GroundingDino/main.py", line 284, in main
train_stats = train_one_epoch(
File "/mnt/lvm_data/project/xyguo/code_dmx/Open-GroundingDino/engine.py", line 48, in train_one_epoch
loss_dict = criterion(outputs, targets, cap_list, captions)
File "/mnt/lvm_data/public/package/anaconda3/envs/groundingdino_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/lvm_data/project/xyguo/code_dmx/Open-GroundingDino/models/GroundingDINO/groundingdino.py", line 596, in forward
tgt_ids[i]=tgt_ids[i][indices[i][1]]
IndexError: index 909 is out of bounds for dimension 0 with size 900

通过翻阅代码，发现应该下图中的两行代码产生的索引值超出了for循环中tgt_ids[i]的长度，请问应该怎么改呢?

[Tibbers888]

I met the same problem. Have you found a solution？

[longzw1997]

Could you please provide more details? For example, the specific values of tgt_ids and indices.

[WanFeng123313]

Could you please provide more details? For example, the specific values of tgt_ids and indices.

i printed the recent output of the error line "tgt_ids[i]=tgt_ids[i][indices[i][1]]", and I found that this only happens when I use the SOTA dataset in SAMRS.

********before tgt_ids[i]: tensor([ 3, 5, 3, 17, 3, 3, 3, 3, 5, 5, 5, 5, 5, 5, 17, 3, 3, 5,
5, 3, 5, 3, 3, 3, 3]) 25
********indices[i][1]: tensor([ 2, 20, 22, 5, 21, 11, 0, 23, 3, 12, 17, 16, 18, 14, 19, 24, 1, 7,
4, 6, 13, 9, 15, 8, 10]) 25
********after tgt_ids[i]: tensor([ 3, 5, 3, 3, 3, 5, 3, 3, 17, 5, 5, 3, 5, 17, 3, 3, 5, 3,
3, 3, 5, 5, 3, 5, 5])
********before tgt_ids[i]: tensor([14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 14, 14, 14, 14, 14, 0, 14,
0, 14, 0]) 21
********indices[i][1]: tensor([17, 10, 3, 8, 9, 19, 7, 2, 4, 6, 20, 5, 1, 18, 14, 0, 11, 15,
16, 13, 12]) 21
********after tgt_ids[i]: tensor([14, 0, 0, 0, 0, 14, 0, 0, 0, 0, 0, 0, 0, 0, 14, 14, 14, 14,
0, 14, 14])
********before tgt_ids[i]: tensor([15, 15, 15, 15, 15, 15, 15, 15, 15, 15]) 10
********indices[i][1]: tensor([6, 0, 5, 3, 7, 8, 9, 2, 4, 1]) 10
********after tgt_ids[i]: tensor([15, 15, 15, 15, 15, 15, 15, 15, 15, 15])
********before tgt_ids[i]: tensor([11, 7, 11, 11, 11, 11, 11, 12, 11, 12, 11, 11, 11, 11, 7, 11, 11, 11,
11, 11, 11, 11, 11, 11, 12, 11, 7, 7, 11, 11, 11, 11, 7, 11, 11, 11,
11, 7, 11, 11, 11, 11, 11, 11, 11, 1, 11, 11, 11, 11, 7, 11, 7, 11,
11, 11, 11, 11, 7, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 7, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 7, 11, 11, 7, 11,
11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11,
11, 11, 7, 11, 11, 7, 11, 11, 11, 7, 11, 7, 11, 7, 11, 11, 11, 11,
11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11,
11, 7, 11, 11, 11, 7, 11, 7, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 7, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 12, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11,
11, 11, 11, 11, 11, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 1,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11,
11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 7, 7, 11, 11, 11,
11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
7, 7, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 1, 7, 7, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 0, 11, 11, 7, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11,
11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
7, 11, 11, 0, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 7, 11, 11, 11, 11, 7, 11,
11, 11, 11, 7, 7, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 7,
11, 11, 11, 11, 7, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7,
11, 11, 11, 11, 7, 11, 11, 11, 7, 0, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 12, 11, 11, 11, 11, 7, 7, 11, 11, 11, 7, 7,
11, 11, 11, 7, 11, 11, 11, 7, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 7, 11,
11, 11, 7, 11, 11, 1, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 7, 11, 7, 7, 11, 7,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11,
11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11,
7, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 7, 11, 1, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11,
1, 7, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 7, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11]) 900
********indices[i][1]: tensor([1066, 1126, 336, 763, 747, 616, 887, 1115, 1001, 1113, 941, 1082,
862, 824, 1182, 774, 620, 630, 632, 269, 432, 524, 529, 625,
1114, 914, 1204, 1165, 321, 234, 888, 535, 1170, 105, 445, 308,
1123, 1187, 1026, 363, 935, 437, 615, 722, 416, 0, 923, 745,
556, 430, 1136, 665, 1186, 570, 902, 919, 820, 718, 1210, 1167,
837, 901, 474, 441, 917, 860, 574, 839, 909, 324, 739, 555,
843, 859, 1178, 929, 957, 327, 730, 1201, 446, 855, 856, 873,
520, 1185, 866, 955, 1176, 568, 627, 440, 666, 849, 646, 517,
1199, 921, 948, 672, 522, 236, 911, 936, 613, 651, 1161, 240,
274, 912, 1191, 772, 806, 1203, 864, 729, 939, 1132, 86, 1131,
231, 711, 263, 918, 187, 167, 886, 673, 786, 643, 528, 563,
767, 879, 537, 882, 557, 813, 145, 80, 1118, 614, 602, 152,
265, 1179, 634, 795, 951, 1171, 1028, 1190, 1151, 1073, 34, 962,
1080, 769, 1062, 52, 924, 562, 197, 543, 1124, 371, 658, 47,
1211, 325, 748, 1081, 113, 62, 259, 454, 1015, 1012, 1046, 580,
724, 91, 310, 851, 273, 930, 1112, 968, 958, 301, 137, 773,
937, 583, 657, 154, 746, 412, 881, 534, 842, 1068, 799, 1111,
223, 810, 961, 821, 144, 305, 1091, 938, 394, 482, 65, 2,
143, 794, 932, 444, 459, 792, 71, 807, 1063, 840, 686, 974,
509, 894, 1140, 906, 549, 876, 618, 278, 922, 1069, 982, 1128,
90, 582, 865, 815, 68, 558, 132, 1142, 1155, 1032, 92, 898,
351, 834, 1196, 356, 1060, 717, 647, 115, 168, 1105, 478, 891,
872, 205, 427, 897, 1023, 609, 1205, 1147, 1035, 857, 1189, 829,
601, 830, 652, 853, 989, 954, 4, 1144, 1148, 675, 418, 715,
959, 507, 138, 525, 828, 316, 158, 102, 165, 809, 277, 428,
994, 8, 697, 693, 1146, 684, 1043, 884, 377, 972, 933, 151,
450, 889, 586, 885, 101, 1108, 802, 266, 1134, 505, 705, 159,
24, 120, 270, 783, 971, 1173, 925, 194, 104, 967, 515, 123,
1017, 1018, 1040, 944, 1052, 527, 1206, 947, 841, 7, 1004, 67,
946, 659, 288, 1153, 15, 623, 504, 328, 1079, 576, 875, 135,
340, 38, 905, 956, 1130, 920, 690, 172, 93, 496, 716, 607,
950, 952, 890, 96, 157, 591, 185, 40, 904, 791, 931, 367,
816, 109, 628, 1162, 1058, 1120, 217, 554, 694, 698, 1209, 291,
1102, 635, 83, 1158, 1197, 710, 668, 112, 983, 927, 1163, 531,
709, 319, 538, 238, 499, 986, 521, 1005, 510, 117, 566, 706,
1064, 1125, 704, 1045, 660, 55, 267, 468, 893, 775, 685, 501,
1042, 692, 530, 419, 338, 439, 714, 674, 610, 146, 999, 1157,
357, 980, 976, 243, 648, 1143, 14, 84, 603, 926, 1149, 1129,
663, 420, 497, 1056, 465, 1048, 516, 184, 826, 48, 98, 827,
861, 486, 812, 323, 426, 735, 1152, 908, 1054, 650, 654, 1101,
1109, 508, 913, 979, 1085, 87, 114, 1159, 801, 544, 222, 819,
1031, 1100, 519, 1070, 30, 845, 670, 964, 741, 1110, 642, 1198,
97, 467, 832, 329, 777, 776, 315, 713, 1154, 6, 464, 1057,
541, 848, 878, 533, 629, 245, 272, 449, 294, 218, 513, 577,
523, 1207, 390, 953, 589, 404, 561, 1084, 1208, 260, 587, 42,
386, 626, 271, 1160, 782, 548, 823, 103, 703, 598, 1041, 581,
365, 547, 653, 836, 896, 617, 1098, 413, 330, 753, 1071, 139,
619, 280, 678, 838, 526, 565, 780, 539, 279, 106, 600, 756,
689, 369, 760, 770, 779, 512, 1169, 216, 198, 858, 1177, 868,
189, 506, 414, 476, 696, 750, 1014, 903, 443, 1193, 1039, 966,
1116, 943, 31, 311, 141, 1117, 1156, 411, 451, 915, 1141, 1164,
708, 289, 320, 621, 604, 681, 781, 1192, 803, 252, 595, 818,
1133, 662, 366, 874, 883, 1074, 1094, 1008, 518, 12, 688, 536,
987, 1067, 765, 863, 728, 796, 224, 200, 334, 347, 676, 422,
111, 228, 498, 1181, 487, 721, 869, 771, 469, 916, 1194, 671,
907, 1050, 854, 970, 1200, 125, 1051, 895, 1195, 60, 532, 5,
768, 572, 1137, 1103, 99, 1104, 719, 720, 636, 633, 835, 622,
934, 784, 202, 1107, 171, 82, 1150, 738, 559, 805, 354, 131,
1172, 212, 1135, 1127, 169, 1184, 494, 949, 590, 173, 667, 43,
10, 605, 847, 701, 1175, 1180, 965, 712, 790, 335, 737, 852,
79, 1168, 314, 342, 945, 130, 758, 584, 814, 800, 303, 1188,
287, 910, 460, 594, 754, 88, 844, 1024, 317, 969, 442, 702,
585, 170, 1065, 121, 731, 639, 175, 811, 1119, 644, 833, 16,
1145, 514, 759, 57, 1122, 552, 778, 493, 567, 326, 241, 402,
290, 831, 362, 564, 871, 638, 645, 1121, 119, 462, 78, 637,
899, 503, 793, 129, 1006, 1183, 597, 850, 1138, 682, 1, 755,
13, 438, 997, 664, 942, 649, 641, 608, 817, 174, 870, 176,
214, 85, 981, 963, 457, 502, 553, 892, 732, 990, 804, 127,
164, 985, 1139, 118, 304, 752, 787, 749, 661, 825, 417, 655,
3, 1166, 1055, 29, 683, 373, 734, 261, 656, 726, 571, 742,
1033, 345, 624, 977, 789, 677, 472, 177, 1077, 785, 699, 588,
725, 147, 358, 415, 744, 1106, 1202, 798, 761, 877, 551, 631,
1013, 108, 797, 1061, 500, 153, 606, 867, 124, 77, 669, 107,
1088, 491, 764, 302, 17, 1047, 282, 100, 546, 463, 353, 788,
940, 1174, 573, 542, 928, 1099, 66, 116, 808, 346, 23, 388]) 900
Traceback (most recent call last):
File "/home/wafeng/code/Open-GroundingDino/main.py", line 372, in
main(args)
File "/home/wafeng/code/Open-GroundingDino/main.py", line 284, in main
train_stats = train_one_epoch(
File "/home/wafeng/code/Open-GroundingDino/engine.py", line 50, in train_one_epoch
loss_dict = criterion(outputs, targets, cap_list, captions)
File "/home/wafeng/anaconda3/envs/groundingdino_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wafeng/code/Open-GroundingDino/models/GroundingDINO/groundingdino.py", line 601, in forward
elastic_launch(
File "/home/wafeng/anaconda3/envs/groundingdino_env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/wafeng/anaconda3/envs/groundingdino_env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

main.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2023-12-04_09:06:49
host : DESKTOP-87GSUUI.
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 87291)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[WanFeng123313]

I met the same problem. Have you found a solution？

I suspected that the number of instances in the image could not be more than the number of queries (is 900), so I deleted all the images with more than 900 instances in the dataset and can now train normally

[xiyangyang99]

I met the same problem. Have you found a solution？

I suspected that the number of instances in the image could not be more than the number of queries (is 900), so I deleted all the images with more than 900 instances in the dataset and can now train normally

this is my train logs：
Some weights of BertModel were not initialized from the model checkpoint at bert_base_uncased and are newly initializedense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
File "main.py", line 372, in
main(args)
File "main.py", line 144, in main
model, criterion, postprocessors = build_model_main(args)
File "main.py", line 81, in build_model_main
model, criterion, postprocessors = build_func(args)
File "/home/quchunguang/003-large-model/Open-GroundingDino/models/GroundingDINO/groundingdino.py", line 804, in buil
postprocessors = {'bbox': PostProcess(num_select=args.num_select , text_encoder_type=args.text_encoder_type,nms_i,args=args)}
File "/home/quchunguang/003-large-model/Open-GroundingDino/models/GroundingDINO/groundingdino.py", line 665, in __in
new_pos_map[v] = pos_map[k]
IndexError: index 1 is out of bounds for dimension 0 with size 1

i have only 1 class ，but i get the same question with you ，but my datasets have no object 。

[VolkaJ]

I've encountered the same problem.

Open-GroundingDino/models/GroundingDINO/groundingdino.py", line 665, in init
new_pos_map[v] = pos_map[k]
IndexError: index 4 is out of bounds for dimension 0 with size 4