Core Concepts
Sparse multi-view methods can improve hand-object reconstruction quality compared to single-view methods, while requiring less data than dense multi-view approaches.
Abstract
The paper proposes a sparse multi-view method for hand-object reconstruction, called SVHO, that takes as input multiple RGB images and corresponding global hand poses. The method predicts the hand and object shapes independently from each view and combines them to form a final reconstruction.
Key highlights:
The authors train autoencoders to encode hand and object shapes independently in a canonical coordinate space using Patchwise VQ-VAE (P-VQ-VAE). This provides a compact representation to train the hand-object shape prior.
During test time, the model obtains 2D features from the input images, forms a 3D feature grid by projecting the 3D points to the image space using the global hand pose, and reconstructs the hand and object shapes in the canonical coordinate space.
The authors evaluate the proposed method on the DexYCB dataset with unseen objects, and show that while reconstruction of unseen hands and objects from RGB is challenging, additional views can help improve the reconstruction quality.
The authors observe that increasing the number of views can negatively impact the object reconstruction quality in cluttered scenes, and suggest the need for a hand-object segmentation model to better leverage the multiple views.
Stats
The paper reports the following key metrics:
Chamfer distance (CD) and F-score (FS) for evaluating the reconstruction quality of the predicted hand and object meshes.
The authors report the CD and FS for hand and object reconstruction when varying the number of input views from 1 to 8.
Quotes
"Sparse multi-view methods provide a balanced approach between single-view and dense multi-view methods but has not been investigated in the hand-object reconstruction task."
"We show that while reconstruction of unseen hands and objects from RGB is challenging, additional views can help improve the reconstruction quality."