Camera to ignore world & extracting image pixel coordinates.

[Varatharajan-Raja]

Describe the issue

Thank you for the wonderful tool. @cornerfarmer

1. I'm trying to snap images from top of a plane(soil in my case) by hovering my camera over it. I'm sampling locations on the top with bproc.sampler.upper_region() Could you please help me how I can restrict camera from capturing the world background?

2. I wish to extract the location of the objects(plants) placed in a plane(soil) in image coordinates? I know in coco annotation one could extract the bbox as pixel coords. But I'm looking for exact location(point of contact with the plane(soil) and different objects(plants)) in the rendered image.

Thanks.

Minimal code example

No response

Files required to run the code

No response

Expected behavior

None

BlenderProc version

v2.6.1

[cornerfarmer]

Hey @Varatharajan-Raja,

regarding 1: You can use bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"no_background": True}, bvh_tree) to check whether the background is visible from a given camera pose. (See e.g.: https://github.com/DLR-RM/BlenderProc/blob/main/examples/datasets/matterport3d/main.py#L35)

regarding 2: I have added a few more helper functions in #1045. The function bproc.camera.project_points() might be what you need.

[Varatharajan-Raja]

Thank you @cornerfarmer . I added "no_background" : True in the function which checks for obstacle as suggested. 1st of all I see that the function only takes "no_background" along with obstacle distance of min/max values given to it. I'm unsure why it cannot be only 'no_background'. However, I set 1.0 m min proximity for obstacles along with 'no_background' to skip thoses frames. For this, I see that not even one frame was registered to render. This was the code and its error.

       for try_counter in range(1000):
            # Sample location
            location=bproc.sampler.upper_region(objects_to_sample_on=[soil],
                                                min_height=19,
                                                max_height=20,
                                                use_ray_trace_check=True)
            rotation_matrix = bproc.camera.rotation_from_forward_vec(poi)
            cam2world_matrix = bproc.math.build_transformation_mat(location,rotation_matrix)
  
            # Check that obstacles are at least 1 meter away from the camera and make sure the view has no background
            if not bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 1.0, "no_background": True}, bvh_tree):
                continue

            #Light properties start
            light_location = bproc.sampler.shell(
                center=[1, 2, 21],
                radius_min=4,
                radius_max=7,
                elevation_min=15,
                elevation_max=70
            )
            light.set_rotation_euler([0, random.uniform(-math.pi/2, math.pi/2), 0],poses)
            light.set_energy(random.uniform(1,10), poses)
            light.set_color(np.random.uniform([1, 0.4, 0.3], [1, 0.6, 0.6]), poses)
            #light.set_energy(1, poses)
            light.set_location(light_location, poses)

            
            poses +=1
            if poses == 5:
                break

Error: Python: Traceback (most recent call last):
File "/home/raja/00_projects/00_E-Terry/00_dev/synthetic_farm_v3.py", line 157, in
main()
File "/home/raja/00_projects/00_E-Terry/00_dev/synthetic_farm_v3.py", line 151, in main
data = bproc.renderer.render()
File "/home/raja/00_projects/00_E-Terry/BlenderProc/blenderproc/python/renderer/RendererUtility.py", line 704, in render
raise RuntimeError("No camera poses have been registered, therefore nothing can be rendered. A camera "
RuntimeError: No camera poses have been registered, therefore nothing can be rendered. A camera pose can be registered via bproc.camera.add_camera_pose().

Next for,

            if not bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"no_background": True}, bvh_tree):
                continue

Error Traceback:

Exception has occurred: KeyError (note: full exception trace is shown but execution is paused at: )
'min'
File "/home/raja/00_projects/00_E-Terry/BlenderProc/blenderproc/python/camera/CameraValidation.py", line 71, in perform_obstacle_in_view_check
range_distance = proximity_checks["min"]
File "/home/raja/00_projects/00_E-Terry/00_dev/synthetic_farm_v3.py", line 113, in main
if not bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"no_background": True}, bvh_tree):
File "/home/raja/00_projects/00_E-Terry/00_dev/synthetic_farm_v3.py", line 159, in (Current frame)
main()
KeyError: 'min'

[cornerfarmer]

Hey @Varatharajan-Raja,

thanks for the hint regarding the min param. This should be fixed.

regarding the fact that no camera poses are marked as valid: this might be caused by a bug I fixed in #1045. Could you try to apply the fix this bvh_tree fix: https://github.com/DLR-RM/BlenderProc/pull/1045/files#diff-86c0fcf5b7503bc5d0aa08d93b78a9f794597a2984785da24f051e9d094f1b74 and check if that solves your problem?

If not, could you post your full code and check whether there really are camera poses sampled which have no background in there?

[Varatharajan-Raja]

@cornerfarmer No valid camera poses even after applying the fix to bvh_tree in MeshObjectUtility.py

[cornerfarmer]

If not, could you post your full code and check whether there really are camera poses sampled which have no background in there?

[Varatharajan-Raja]

Hey @cornerfarmer,

1. Sorry, yesterday I was deeply engrossed in the loop, and unfortunately, I missed to add camera pose. Now it works fine. However, I feel in is necessary to enhance the function bproc.camera.perform_obstacle_in_view_check() to also only take no_background.

2. Regarding extracting the position of fallen object on the plane(soil) in pixel coordinates, I wrote a separate function to extract the point of contact with the soil in camera view. So for every frames, the position of the fallen objects(plants) are extracted.

Thanks. I'm closing this issue and the associated issue #1051