ConceptFactory: Facilitate 3D Object Knowledge Annotation with Object Conceptualization

Jianhua Sun^*, Yuxuan Li^*, Longfei Xu^**, Nange Wang^**, Jiude Wei^**, Yining Zhang, Cewu Lu,^†

Shanghai Jiao Tong University
NeurIPS 2024 Datasets and Benchmarks Track
^* and ^** Indicate Equal Contribution
^† Denotes Corresponding Author

Paper Supplementary Template Codes (Coming Soon...) Conceptualization Platform (Coming Soon...)

Abstract

We present ConceptFactory, a novel scope to facilitate more efficient annotation of 3D object knowledge by recognizing 3D objects through generalized concepts (i.e. object conceptualization), aiming at promoting machine intelligence to learn comprehensive object knowledge from both vision and robotics aspects. This idea originates from the findings in human cognition research that the perceptual recognition of objects can be explained as a process of arranging generalized geometric components (e.g. cuboids and cylinders). ConceptFactory consists of two critical parts: i) ConceptFactory Suite, a unified toolbox that adopts Standard Concept Template Library (STL-C) to drive a web-based platform for object conceptualization, and ii) ConceptFactory Asset, a large collection of conceptualized objects acquired using ConceptFactory suite. Our approach enables researchers to effortlessly acquire or customize extensive varieties of object knowledge to comprehensively study different object understanding tasks. We validate our idea on a wide range of benchmark tasks from both vision and robotics aspects with state-of-the-art algorithms, demonstrating the high quality and versatility of annotations provided by our approach.

Motivation & Concept Template Design

Fig.1 [Left] Illustration of the relationship between human cognition (a-b) and our approach (c-e), exemplified by handle as object and affordable interaction as knowledge. (a) Human recognizes objects as an arrangement of geometric components. (b) Abstract commonsense information are induced from the geometries in human mind. (c) Explicitly model the abstract information as a regular geometry concept with specific knowledge. (d) Generalize the concept towards different objects. (e) Propagate the knowledge from the concept to objects as annotations. [Right] Example of parameters and the constructor of a concept template.

Fig.2 Shape instances of geometry (Top) and concept (Bottom) templates with specific parameters. [Bottom] The figures on the left side of the arrows display each geometry component of a concept template individually, whereas those on the right side are example instances of concept templates with various parameters. The instance at bottom-right is the result of modifying discrete parameters.

Conceptualization Platform

Fig.3 An overview to our conceptualization interface and the workflow (blue arrow). The interface is divided into four components: work space, target view, template rendering, and mixed view. In work space, users first select best-match templates for each part of the target object, then parameterize each concept template with the help of the optimizer, and finally save the conceptualization result. Target view illustrates the shape of the target object, template rendering displays instances of concept templates with current parameters, while mixed view visualizes the integration between target view (gray) and template rendering (blue), helping users perform the conceptualization efficiently.

Video demo of the conceptualization platform.

Conceptualization & Knowledge Annotation

Fig.4 [Left] Conceptualization results of a KitchenPot, geometric details and certain parameters are omitted for simplicity. [Right] Procedural annotation for different types of knowledge. (a-b) Region-based knowledge like semantics and affordable area is implemented through region discrimination function. (c-d) Pose-based knowledge like part pose and grasp pose is implemented with transformations from local to world coordinates. Please refer to the codes in our website for detailed implementations.

Visualization of Knowledge Annotation

Fig.5 Visualization of different types of knowledge annotations including affordance (Push-Row.1-3, Pull-Row.4-5), semantic segmentation and part pose. [Left] Annotations acquired by conventional approaches. [Right] Annotations acquired by our approach.

BibTeX

Coming Soon...