SeMv-3D: Achieving Semantic and Multi-view Consistency in General Text-to-3D Generation Using Triplane Priors

Concepts de base

SeMv-3D is a novel framework that leverages triplane priors and a two-step learning process to generate semantically consistent and multi-view coherent 3D objects from text descriptions.

Résumé

• Bibliographic Information: Cai, X., Zeng, P., Gao, L., Zhu, J., Zhang, J., Su, S., Shen, H. T., & Song, J. (2024). SEMV-3D: TOWARDS SEMANTIC AND MUTIL-VIEW CONSISTENCY SIMULTANEOUSLY FOR GENERAL TEXT-TO-3D GENERATION WITH TRIPLANE PRIORS. arXiv preprint arXiv:2410.07658v1.

• Research Objective: This paper introduces SeMv-3D, a novel framework designed to address the challenges of achieving both semantic and multi-view consistency in general text-to-3D generation tasks.

• Methodology: SeMv-3D consists of two primary components:

  • Triplane Prior Learner (TPL): This component learns a triplane prior by first retaining the main object from the input text (Object Retention) and then capturing the spatial correspondence within the triplane space (Triplane Orthogonalization) using a novel orthogonal attention mechanism.
  • Semantic-aligned View Synthesizer (SVS): This component transforms the triplane prior into latent space while aligning it with semantic information extracted from the text. It utilizes a Triplane Latents Transformation module to enhance the interaction between textual and visual features. Finally, it employs a batch sampling and rendering strategy to generate arbitrary views of the 3D object in a single feed-forward step.

• Key Findings: SeMv-3D demonstrates superior performance in generating 3D objects from text descriptions compared to existing state-of-the-art methods. It effectively addresses the limitations of previous approaches, such as multi-view inconsistency in fine-tuning-based methods and semantic inconsistency in prior-based methods.

• Main Conclusions: The authors conclude that SeMv-3D offers a promising solution for general text-to-3D generation by effectively integrating semantic and multi-view consistency. The proposed framework leverages the strengths of triplane priors, orthogonal attention, and a novel batch rendering strategy to achieve high-quality 3D object generation from text.

• Significance: This research significantly contributes to the field of text-to-3D generation by introducing a novel framework that effectively addresses the long-standing challenges of semantic and multi-view consistency. The proposed SeMv-3D framework has the potential to advance various applications, including content creation for games, movies, virtual/augmented reality, and robotics.

• Limitations and Future Research: The authors acknowledge the limitations posed by the current lack of high-quality, large-scale text-3D paired datasets. Future research could focus on developing more robust and comprehensive datasets to further enhance the performance and generalization capabilities of SeMv-3D. Additionally, exploring more efficient training strategies and incorporating advanced rendering techniques could lead to further improvements in the quality and realism of generated 3D objects.

Stats

The authors trained their model on a subset of approximately 500,000 objects from the Objaverse dataset.
The training process for the Triplane Prior Learner (TPL) involved two stages: 150,000 steps for object retention with a learning rate of 5 x 10^-4, and 60,000 steps for triplane orthogonalization with a learning rate of 5 x 10^-5.
The Semantic-aligned View Synthesizer (SVS) was trained for 100,000 steps with a learning rate of 5 x 10^-4.
All experiments and training were conducted on eight NVIDIA A6000 GPUs.
The authors used the AdamW optimizer for all training stages with β1 = 0.9, β2 = 0.95, and weight decay 0.03.
For objective evaluation, the authors used Clip Score and Aesthetic Score to assess the semantic alignment and aesthetic quality of the generated objects, respectively.
The user study involved 48 participants who evaluated the generated results based on user preference, semantic consistency, and multi-view consistency.

Citations

"To achieve semantic and multi-view consistency simultaneously, we propose SeMv-3D, a novel framework for general text-to-3d generation."
"Specifically, we propose a Triplane Prior Learner (TPL) that learns triplane priors with 3D spatial features to maintain consistency among different views at the 3D level, e.g., geometry and texture."
"Moreover, we design a Semantic-aligned View Synthesizer (SVS) that preserves the alignment between 3D spatial features and textual semantics in latent space."
"Extensive experiments present our SeMv-3D’s superiority over state-of-the-art performances with semantic and multi-view consistency in any view."

Idées clés tirées de

SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors

by Xiao Cai, Pe... à arxiv.org 10-11-2024

https://arxiv.org/pdf/2410.07658.pdf