SA-DNet is an end-to-end Semantic-Aware on-Demand registration-fusion network. SA-DNet can specify registration and fusion regions based on semantic information, achieving on-demand conceptualization. Furthermore, the sam, reg, and fuse modules are separated, allowing integration of new functionalities. The current architecture implemented in this project is as follows:

In this project, the HOL operators in reg have not been implemented. When SAM perceives regions that are too large, it leads to excessive computation in HOL, causing the system to freeze.

• Fixed bug.
• Update the new sam, reg, and fuse modules.
• To optimize and implement the HOL operator

git clone [email protected]:Meng-Sang/SA-DNet.git
cd SA-DNet

This project requires a runtime environment with Python=3.8 or higher. Additionally, it depends on the libraries specified in the requirements.txt file and 2 GitHub repositories (Grounding DINO and Segment Anything). The environment configuration is as follows:

conda create -n sa-dnet  python=3.8
pip install -r requirement.txt

For Grounding DINO installation, please refer to Grounding DINO.

# if model/sam/grounding_dino/weight not exist.
# please run: mkdir -p model/sam/grounding_dino/weight
# then:
cd model/sam/grounding_dino/weight
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth

If the network is unavailable, the following error may occur:

OSError: Can't load tokenizer

git clone [email protected]:facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

or

pip install git+git:https://github.com/facebookresearch/segment-anything.git

# if model/sam/grounding_dino/weight not exist.
# please run: mkdir -p model/sam/segment_anything/weight
# then:
cd model/sam/segment_anything/weight
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

import cv2
from utils.predict import sa_dnet

ir_image = cv2.imread("assets/ir/006393.jpg")
vi_image = cv2.imread("assets/vi/006393.jpg")
_, fuse_image = sa_dnet(ir_image, vi_image, "car", is_mask=True)
cv2.imwrite("assets/demo.jpg", fuse_image)

IR Image	VI Image
Matching Image (Car)
Unmask Fuse Image (Car)	Mask Fuse Image (Car)

This project depends on the following repositories:

• Grounding DINO
• Segment Anything