Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships

We present Open3DSG, the first approach for learning to predict open-vocabulary 3D scene graphs from 3D point clouds. Our method can predict not only explicit open-vocabulary object classes, but also open-set relationships that are not limited to a predefined label set, enabling the expression of rare and specific objects and relationships in the predicted 3D scene graph.