Add Joint Gravity Estimation with Rigid Rig Constraints for Multi-Camera Rigs

[sarlinpe]

Overview

This pull request introduces support for joint gravity estimation across multiple perspective images in a camera rig. By utilizing known relative rotations between sensors in a camera rig (such as the multi-camera perspectives in a 360 panorama), GeoCalib can now jointly optimize for a single gravity vector defined in the rig coordinate system, rather than estimating gravity independently for each perspective.

Enforcing a shared gravity vector acts as a strong regularizer and significantly improves the accuracy and robustness of gravity estimation, reducing both angular errors and the rate of outliers.

Key Changes

1. Jacobians Transformation in Perspective Fields:

   • Modified geocalib/perspective_fields.py to accept an optional camera_R_rig tensor of shape (B, 3, 3).
   • Updated derivatives of perspective fields w.r.t. gravity to use the chain rule, transforming Jacobians from individual camera frames into the shared rig coordinate frame:
     $J_{\text{rig}} = J_{\text{camera}} \cdot R_{\text{rig}}$
   • This ensures that updates are correctly accumulated on the shared rig gravity vector.

2. Optimizer Updates for Rig Constraint:

   • Updated geocalib/lm_optimizer.py to support camera_R_rig in the optimization loop.
   • Implemented rig-based gravity initialization by averaging the initial gravity estimated for each image.
   • Allowed fixing intrinsics during optimization by detecting the presence of a focal length prior (prior_focal in input data dict), disabling intrinsics optimization and focusing solely on the shared gravity vector.

3. Inference Scripts in COLMAP (colmap repository):

   • Added python/examples/geocalib_panorama.py to read rigs, cameras, and images from a COLMAP database, run joint gravity estimation via the updated GeoCalib model, and write estimated gravity vectors as PosePrior objects back to the database, leaving intrinsic parameters fixed.

Evaluation Results

We evaluated joint gravity estimation with rigid rig constraints versus independent single-image estimation on the campus_parterre dataset ($36$ panoramas = $144$ perspective renderings). In this evaluation, cameras' focal lengths were fixed to their known ground truth priors ($f=705.0$) from the beginning (has_focal_length_prior = True).

Results with Fixed Intrinsics Priors ($f=705.0$ locked)

Method	Mean Error (deg)	Median Error (deg)	Max Error (deg)	Recall@1°	Recall@5°	Recall@10°	Evaluated Images
Baseline (Independent)	$1.0911$	$0.9170$	$5.0687$	$55.86%$	$99.77%$	$100.00%$	$444$
Joint (Rig Constrained)	$\mathbf{0.4239}$	$\mathbf{0.4220}$	$\mathbf{0.8633}$	$\mathbf{100.00%}$	$\mathbf{100.00%}$	$\mathbf{100.00%}$	$444$

Discussion of Results

• The Joint (Rig Constrained) estimation achieves a mean error of $\mathbf{0.4239^\circ}$ compared to $1.0911^\circ$ for the Baseline, demonstrating superior accuracy.
• Robustness: The maximum error is reduced from $5.0687^\circ$ to $0.8633^\circ$, suppressing outliers.
• Recall Metrics: All estimates in the joint configuration fall within $1^\circ$ of the ground truth (Recall@$1^\circ = 100.00%$), whereas independent estimations only achieve $55.86%$ within $1^\circ$.

Verification Plan

1. Run python/examples/geocalib_panorama.py --database_path <db> --image_path <images> --baseline on a test database and verify independent estimates.
2. Run python/examples/geocalib_panorama.py --database_path <db> --image_path <images> (joint) and verify gravities are written.
3. Evaluate both databases using python/examples/geocalib_evaluate.py --database_path <db> --reconstruction_path <aligned_model> and compare error metrics and recalls.