Metashape project unable to load photos in multiple subfolders

[youngdjn]

Most of my photogrammetry (Metashape) projects involve processing photos that are nested within subfolders; for example:

images/
├── 100MEDIA/
│   ├── DJI_0001.jpg
│   ├── DJI_0002.jpg
│   ├── DJI_0003.jpg
│   └── ...
├── 101MEDIA/
│   ├── DJI_0001.jpg
│   ├── DJI_0002.jpg
│   ├── DJI_0003.jpg
│   └── ...
└── 102MEDIA/
    ├── DJI_0001.jpg
    ├── DJI_0002.jpg
    ├── DJI_0003.jpg
    └── ...

But when I try to load such a Metashape project in EasyIDP, I get: IndexError: Index [0] out of range (0, 0).

To reproduce it, you can download this example metashape project file (which is based on this image set) and run:

import easyidp as idp
from pathlib import Path
msproj = Path("path/to/project/example-multifolder.psx")
ms = idp.Metashape(msproj, chunk_id=0)

The full error returned is:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_481076/3594889881.py in <module>
----> 1 ms = idp.Metashape(msproj, chunk_id=0)

~/Downloads/EasyIDP-2.0/easyidp/metashape.py in __init__(self, project_path, chunk_id)
    105         self.photos = self.photos
    106 
--> 107         self.open_project(project_path, chunk_id)
    108 
    109     def __repr__(self) -> str:

~/Downloads/EasyIDP-2.0/easyidp/metashape.py in open_project(self, project_path, chunk_id)
    172         if project_path is not None:
    173             self._open_whole_project(project_path)
--> 174             self.open_chunk(self.chunk_id)
    175         else:
    176             if chunk_id is not None:

~/Downloads/EasyIDP-2.0/easyidp/metashape.py in open_chunk(self, chunk_id, project_path)
    244 
    245         if chunk_id in self._project_chunks_dict.keys():
--> 246             chunk_content_dict = read_chunk_zip(
    247                 self.project_folder,
    248                 self.project_name,

~/Downloads/EasyIDP-2.0/easyidp/metashape.py in read_chunk_zip(project_folder, project_name, chunk_id, skip_disabled, return_label_only)
   1176         camera_meta, marker_meta = _decode_frame_xml(frame_xml_str)
   1177         for camera_idx, camera_path in camera_meta.items():
-> 1178             chunk_dict["photos"][camera_idx]._path = camera_path
   1179             # here need to resolve absolute path
   1180             # <photo path="../../../../source/220613_G_M600pro/DSC06035.JPG">

~/Downloads/EasyIDP-2.0/easyidp/__init__.py in __getitem__(self, key)
     72                 return self.id_item[key]
     73             else:
---> 74                 raise IndexError(f"Index [{key}] out of range (0, {len(self.item_label)})")
     75         elif isinstance(key, str):  # index by photo name
     76             if key in self.item_label.keys():

IndexError: Index [0] out of range (0, 0)

The error occurs even if the photo file names are not duplicated across the folders (which is true of the example I ran and linked to here). Ideally, EasyIDP would tolerate multiple subfolders and an arbitrary folder nesting depth (at least to 5 levels).

[HowcanoeWang]

This has been fixed, but please notice, the camera label will become "100MEDIA-DJI_0001" rather than "DJI_0001"

[youngdjn]

Thank you! This works now for the example project I provided -- great!

However, I tested this for a more common use case for me -- multiple layers of nested folders -- and it didn't work (same error as before). Here is an example folder structure:

images/
├── 80m/
│   ├── n-s/
│   │   ├── card01/
│   │   │   ├── 100MEDIA/
│   │   │   │   ├── DJI_0001.jpg
│   │   │   │   └── DJI_0002.jpg
│   │   │   └── 101MEDIA/
│   │   │       ├── DJI_0001.jpg
│   │   │       └── DJI_0002.jpg
│   │   └── card02/
│   │       ├── 100MEDIA/
│   │       │   ├── DJI_0001.jpg
│   │       │   └── DJI_0002.jpg
│   │       └── 101MEDIA/
│   │           ├── DJI_0001.jpg
│   │           └── DJI_0002.jpg
│   └── e-w/
│       ├── 100MEDIA/
│       │   ├── DJI_0001.jpg
│       │   └── DJI_0002.jpg
│       └── 101MEDIA/
│           ├── DJI_0001.jpg
│           └── DJI_0002.jpg
└── 120m/
    ├── 100MEDIA/
    │   ├── DJI_0001.jpg
    │   └── DJI_0002.jpg
    └── 101MEDIA/
        ├── DJI_0001.jpg
        └── DJI_0002.jpg

To reproduce it, you can use this example metashape project file (which is based on this image set).

If you have time, I wonder if you would consider updating the method to tolerate an arbitrary level of nesting (or at least 5 levels).

[HowcanoeWang]

Actually, what I did for this fix, it just supported an extra layer about the "camera group" in the metashape (the previous version did not support camera group)

project without camera group	project with camera group
<cameras next_id="218" > <camera id="0" label="DJI_0003" ... >...</camera> <camera id="1" label="DJI_0004"... >...</camera> <camera id="2" label="DJI_0005"... >...</camera> ... </cameras>	<cameras next_id="218" next_group_id="3"> <group id="0" label="100MEDIA" type="folder"> <camera id="0" label="DJI_0003" ...>...</camera> <camera id="1" label="DJI_0004" ...>...</camera> <camera id="2" label="DJI_0005" ...>...</camera> ... </group> ... </cameras>

the EasyIDP package doesn't care about how the raw image folder is structured, it just read the path data stored in metashape project (*.files/chunk_id/frame.zip/doc.xml).

<?xml version="1.0" encoding="UTF-8"?>
<frame version="1.2.0">
  <cameras>
    <camera camera_id="0">
      <photo path="../../../../../../tnc-yuba-2021/example-nested-folders/80m/e-w/card01/100MEDIA/DJI_0001.JPG">

And parse the relative path to absolute path.

The reason why you got the "same" (actually not exactly the same) error is that in your given project:

The camera-group name and image name are duplicated, hence the later image label will overwrite the previous image label, and cause the image-id not matching the total number.

In this 211f185 updates, I handled such case, if the project has the duplicated camera-group name, it will return the label like [0]100MEDIA-DJI_0001

Hoping this fix helps for your project.

[youngdjn]

Thank you! This works now and it allows me to process my projects. I really appreciate you building in this tolerance for nested folder structures.