Abstract:

Aiming at the 3D objects detection based on multi-sensor information fusion， a high-accuracy real-time two-stage deep neural network PointRGBNet is proposed. In the first stage with regional proposal network， 3D point clouds are firstly projected onto 2D image to generate 6D RGB point clouds， then feature extraction is performed on the 6D RGB point clouds input to obtain low-dimensional feature map and high-dimensional feature map which are then fused for generating a large number of proposals with high confidence. In the second stage with object detection network， the proposals generated in the first stage are used for RoI pooling to obtain the feature collection corresponding to each proposal from feature map， and more accurate 3D object detection is achieved by targetedly learning the feature collection of proposals. The results of open test on KITTI dataset show that PointRGBNet is not only better than the object detection networks using only 2D images or 3D point clouds in detection accuracy， even better than some advanced multi-sensor information fusion networks， but also has a high object detection speed of the entire network up to 12 frame/s， meeting the real-time requirements.

Key words: object detection, 2D images, 3D point clouds, deep neural networks