Poor training performance - Partitioned detector

[Mobius1D]

I have trained the individual model and the partitioned detectors as per the configuration file (shown below) after some minor fixes to the code. (nn.CrossEntropyLoss as classification loss etc.)

Please check and let me know on potential causes and fixes. I am also attaching the performance of COCO Partitioned Detector performance vs Individual performance. (Partitioned detector not performing as expected while individual detector works.)

_BASE_: "Base-CRCNN-COCO.yaml"
MODEL:
  META_ARCHITECTURE: "SplitClassifierRCNN"
  ROI_HEADS:
    NUM_CLASSES: -1
    NAME: "MultiDatasetCascadeROIHeads"
  ROI_BOX_HEAD:
    USE_SIGMOID_CE: False
    # USE_SIGMOID_CE: True
    # USE_EQL_LOSS: True
    HIERARCHY_IGNORE: True
    HIERARCHY_POS_PARENTS: True
DATASETS:
  TRAIN: ("coco_2017_train","objects365_train",)
  TEST: ('coco_2017_val','objects365_val',)
DATALOADER:                                                                                                                                                                                                                                                                                                                                                                 
  SAMPLER_TRAIN: "MultiDatasetSampler"
  NUM_WORKERS: 6
MULTI_DATASET:
  ENABLED: True
  DATASETS: ['coco', 'objects365']
  NUM_CLASSES: [80, 365]
  DATA_RATIO: [1, 1]
  USE_CAS: [False, True]
SOLVER:
  IMS_PER_BATCH: 32
  STEPS: (60000, 80000)
  MAX_ITER: 90000
  CHECKPOINT_PERIOD: 10000

COCO base RCNN.pdf
COCO partitioned detector.pdf