-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding multi-camera for the side grayscale fisheye lens and multi-user training support on the Aria glasses #2932
base: main
Are you sure you want to change the base?
Conversation
11f03c5
to
3a539ac
Compare
…xel_sampler.py, nerfacto_field.py have changes
…with aria multi-cam data
input_image: np.ndarray, | ||
input_calib: calibration.CameraCalibration, | ||
output_focal_length: int, | ||
) -> [np.ndarray, calibration.CameraCalibration]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this a valid annotation? should it be Tuple[np.ndarray, calibration.CameraCalibration]
?
|
||
return output_image, updated_calib | ||
|
||
def generate_circular_mask(numRows: int, numCols: int, radius: float): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return annotation would be nice!
@@ -76,11 +78,10 @@ class TimedPoses: | |||
t_world_devices: List[SE3] | |||
|
|||
|
|||
def get_camera_calibs(provider: VrsDataProvider) -> Dict[str, AriaCameraCalibration]: | |||
def get_camera_calibs(provider: VrsDataProvider, name="camera-rgb") -> AriaCameraCalibration: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
super nit: can we annotate the name
with a Literal
type? (also below)
if pinhole: | ||
f_length = 500 if camera_name == "camera-rgb" else 170 | ||
image_array, src_calib = undistort_image_and_calibration(image_array, src_calib, f_length) | ||
intrinsic = [f_length, f_length, image_array.shape[1] // 2, image_array.shape[0] // 2] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: can we use tuples instead if things don't need to be mutated?
@@ -68,6 +69,7 @@ class AriaImageFrame: | |||
file_path: str | |||
t_world_camera: SE3 | |||
timestamp_ns: float | |||
pinhole_intrinsic: List[int] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe tuple[float, float, float, float]
?
"""Path to the second VRS file if provided""" | ||
mps_data_dir2: Optional[Path] = None | ||
"""Path to the second MPS attachments if provided""" | ||
points_file2: Optional[Path] = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it feels a little weird to me to hardcode things to only support 1 or 2 VRS files, instead of being able to do multiple
would it make sense to replace the original vrs_file: Path
with vrs_file: Tuple[Path, ...]
? as well as mps_data_dir
with mps_data_dir: Tuple[Path, ...]
?
@@ -184,59 +292,126 @@ class ProcessProjectAria: | |||
"""Path to Project Aria Machine Perception Services (MPS) attachments.""" | |||
output_dir: Path | |||
"""Path to the output directory.""" | |||
points_file: Optional[Path] = None | |||
"""Path to the point cloud file (usually called semidense_points.csv.gz) if not in the mps_data_dir""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just wondering, when is the point cloud file to not in the MPS dir?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I downloaded the Aria Everyday Activities Dataset, the pointcloud was in its own separate folder so i went ahead and added support for both cases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
aha that makes sense!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it one shared point cloud across all users?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually it's multiple point cloud files, so rn main() will combine all of them since they should be using the same point of reference
…, also added more annotations
…Macaronio/nerfstudio into aria-side-camera-support
…b.com/AntonioMacaronio/nerfstudio into aria-side-camera-support
… with big datasets
Background
process_project_aria.py
only works for the main camera, and only supports 1 user. Since the Aria glasses can sync multiple users by timestamp with the same point of reference for MPS, it is a nice functionality to support.Changes