multi labels of same object class

[soans1994]

Dear BlenderProc team,

Currently I have managed to get output for different object classes in COCO format.
But, I want (all the annotations like bbox, segmentation mask) of all the smaller meshes that are present inside the main object.
For example, there are texts and symbols that i managed to create as different meshes in blender. I want to save the annotations of these texts and symbols in the coco output. Output attributes like
sub_text_category_id_1, bbox_text_1, segmentation_text_1
sub_symbol_category_id_1, bbox_symbol_1, segmentation_symbol_1
sub_text_category_id_2, bbox_text_2, segmentation_text_2

My goal is to create a multi branch output cnn model which detects main object and its inner features like text and symbols separately.

Thank You
Soans Rahul

[cornerfarmer]

But, I want (all the annotations like bbox, segmentation mask) of all the smaller meshes that are present inside the main object.

What do you mena with smaller meshes? Are these separate objects or not? You will only get separate annotations if they are different objects. So I would recommend to separate the objects and then implement the logic (which objects is the sub objects of which other object) outside of blenderproc.

[soans1994]

@cornerfarmer

actually I have a flat object model, on which there are some text and arrows (which are separate meshes before joining). like a sign board. I want to get the annotations of the text and arrows (like pixels inside each text letter character and arrows/ or at least the centre point of each letter).

Previously you have suggested me a script which saves all grease pencil points as 2D points in this issue.
#134
Also you have told that adding empty objects in blender or iterating over the vertices of an object an use their locations as keypoints is better than grease pencil.

However, in my case what do you think is best since there are many letters to annotate. Should I use the above approach.
Or Is there a method where i can get all mid points of the letters before joining the meshes of the sign board. But then I guess I cannot get annotations of the whole object without joining. Please suggest me a solution.

edit: blender model like below (right side object is before joining the meshes)

[cornerfarmer]

So if I understand you correctly, a segmentation mask for each letter would be enough, right? You can calculate the keypoints then based on that.

If that is the case, I would suggest to simply not join the objects and then use the default segmentation renderer

[soans1994]

Is it possible to do along with the coco annotations. I want the bbox values of the entire sign board. Is it possible wihtout joining the mesh.

[cornerfarmer]

The blue part of the object is one whole object, right? Then the bbox of the blue part should be equal to the bbox of the whole sign.

[soans1994]

Yes,
is it better to use coco annotations modules instead of segment since I need bbox of blue part

[cornerfarmer]

You need both, the segmentation renderer and the coco annotations writer. See https://github.com/DLR-RM/BlenderProc/tree/main/examples/advanced/coco_annotations
You can also calculate the bbox manually based on the segmentation output, but the coco annotations do that out of the box.

[soans1994]

Okay. I will try it tomorrow and update with you

[soans1994]

@cornerfarmer

i tried the coco annotations module to get bbox and segmentation values for each blender curves differently.

Below is the part of output coco json file for 2 images.

#########################################################

The first part "categories" contains id,supercategory and names
"categories": [{"id": 1, "supercategory": "coco_annotations", "name": "Curve.121"}, {"id": 3, "supercategory": "coco_annotations", "name": "Curve.119"}
The second part "images" contains image names
"images": [{"id": 0, "file_name": "images/000000.jpg", "width": 512, "height": 512
The third part contains id, image_id, category_id
"annotations": [{"id": 1, "image_id": 0, "category_id": 1, "iscrowd": 0, "area": 6, "bbox": [246, 277, 3, 2], "segmentation": {"counts": [126229, 2, 510, 2, 510, 2, 134889], "size": [512, 512]}, "width": 512, "height": 512},

Since I have many sign boards of different total number of mesh/paths, the ids will change at each run

To get the bbox of main object,
I will change the name of the outermost mesh whos bbox i am interested, for all sign boards as "Curve_Outer" . Then i can write condition while reading the json file that if "categories""name"=="Curve_Outer":
- then get "id" of that from "categories" section
- then in "annotations" section, if "image_id"=="id", then read annotations of that part as main object bbox where id matches.
And for the texts inside the sign board i am only interested in the segmaps and center point, which i will calculate from the bbox of all by iterating rest of the ids.

Is this correct?

Currently, I have rendered only one sign board in a given scene as shown in picture.
I can render multiple objects. but i am worried that some images get more samples. What do you think?
Which is better?

[cornerfarmer]

Hey @soans1994,

one thing you could do to make this easier and also work with multiple signs is to use the category_id a bit more:
So for example: Use id=1 for "Curve_Outer" of sign 1, then use id=2 for all letters on sign 1. Use id=3 for "Curve_Outer" of sign 2, use id=4 for all letters on sign 2... and so on.
In this way you dont have to work with the name and there should be no confusion between different signs.

[soans1994]

@cornerfarmer

same id for all the text characters possible? wont it be considered as same object group?

[soans1994]

First i should name paths inn blender.
Then I have to set category_id according names right?

for j, obj in enumerate(objs):
    if obj.get_attr("name") == "Outermost":
         obj.set_cp("category_id", 1)
    else:
        obj.set_cp("category_id", j+1)

Should i rename texts too?

[cornerfarmer]

You can use the same category id for multiple objects. There will still be multiple annotations in the coco output.
What do you mean with texts?

[soans1994]

@cornerfarmer
okay i understand.
i mean renaming the meshes of the text characters inside the sign board

[cornerfarmer]

I assume this is solved, if not, feel free to reopen.

[Misbah-Said]

@cornerfarmer

i tried the coco annotations module to get bbox and segmentation values for each blender curves differently.

Below is the part of output coco json file for 2 images.

#########################################################

The first part "categories" contains id,supercategory and names "categories": [{"id": 1, "supercategory": "coco_annotations", "name": "Curve.121"}, {"id": 3, "supercategory": "coco_annotations", "name": "Curve.119"} The second part "images" contains image names "images": [{"id": 0, "file_name": "images/000000.jpg", "width": 512, "height": 512 The third part contains id, image_id, category_id "annotations": [{"id": 1, "image_id": 0, "category_id": 1, "iscrowd": 0, "area": 6, "bbox": [246, 277, 3, 2], "segmentation": {"counts": [126229, 2, 510, 2, 510, 2, 134889], "size": [512, 512]}, "width": 512, "height": 512},

Since I have many sign boards of different total number of mesh/paths, the ids will change at each run

To get the bbox of main object, I will change the name of the outermost mesh whos bbox i am interested, for all sign boards as "Curve_Outer" . Then i can write condition while reading the json file that if "categories""name"=="Curve_Outer": - then get "id" of that from "categories" section - then in "annotations" section, if "image_id"=="id", then read annotations of that part as main object bbox where id matches. And for the texts inside the sign board i am only interested in the segmaps and center point, which i will calculate from the bbox of all by iterating rest of the ids.

Is this correct?

Currently, I have rendered only one sign board in a given scene as shown in picture. I can render multiple objects. but i am worried that some images get more samples. What do you think? Which is better?

hello....

My json file look like this. and how you added name like curve etc.? and my segmentation mask having a lot of value. could you help me hoe to solve this problem??

[Misbah-Said]

@cornerfarmer
i tried the coco annotations module to get bbox and segmentation values for each blender curves differently.
Below is the part of output coco json file for 2 images.
#########################################################
The first part "categories" contains id,supercategory and names "categories": [{"id": 1, "supercategory": "coco_annotations", "name": "Curve.121"}, {"id": 3, "supercategory": "coco_annotations", "name": "Curve.119"} The second part "images" contains image names "images": [{"id": 0, "file_name": "images/000000.jpg", "width": 512, "height": 512 The third part contains id, image_id, category_id "annotations": [{"id": 1, "image_id": 0, "category_id": 1, "iscrowd": 0, "area": 6, "bbox": [246, 277, 3, 2], "segmentation": {"counts": [126229, 2, 510, 2, 510, 2, 134889], "size": [512, 512]}, "width": 512, "height": 512},
Since I have many sign boards of different total number of mesh/paths, the ids will change at each run
To get the bbox of main object, I will change the name of the outermost mesh whos bbox i am interested, for all sign boards as "Curve_Outer" . Then i can write condition while reading the json file that if "categories""name"=="Curve_Outer": - then get "id" of that from "categories" section - then in "annotations" section, if "image_id"=="id", then read annotations of that part as main object bbox where id matches. And for the texts inside the sign board i am only interested in the segmaps and center point, which i will calculate from the bbox of all by iterating rest of the ids.
Is this correct?
Currently, I have rendered only one sign board in a given scene as shown in picture. I can render multiple objects. but i am worried that some images get more samples. What do you think? Which is better?

hello.... My json file look like this. and how you added name like curve etc.? and my segmentation mask having a lot of value. could you help me hoe to solve this problem??

my segmented .npy file is looking like this and I think its wrong.