We present a real-time 3D reconstruction system using an RGB-D sensor on a hand-held tablet. The main novelty of the system is a simultaneous localization and mapping (SLAM) algorithm that uses both point and plane features as primitives. Planes are the most common structures in man-made indoor and outdoor scenes. In contrast to existing SLAM algorithms that use only point features, our algorithm that also exploits plane features has the following advantages: (1) it enables faster correspondence search and registration, since the number of planes is typically much smaller than the number of points; (2) planes generated by many points are more robust to noise than individual points, leading to more accurate registration; and (3) it produces plane-based 3D models that are more compact than point-based ones. As the core of the algorithm, we show that it is possible to register 3D data in two different coordinate systems using any combination of three point/plane features (3 planes, 2 planes and 1 point, 1 plane and 2 points, and 3 points). We use the minimal set of features in a RANSAC framework to robustly compute correspondences and estimate the camera pose. We also use the point and plane features in bundle adjustment to jointly refine all the camera poses and point/plane parameters. We show large-scale indoor reconstruction results as point-based and plane-based 3D models. We also describe several applications of the system, including collision avoidance, augmented reality (AR), and calibration of non-overlapping cameras.

[External Website] http://www.merl.com/demos/point-plane-slam