面向自动驾驶道路场景的相机与毫米波融合的多目标检测算法

doi:10.19562/j.chinasae.qcgc.2025.05.004

Abstract

Abstract:

To meet the demand of efficient and accurate perception in autonomous driving systems， relying solely on cameras makes it challenging to achieve high-precision and robust 3D object detection. An effective solution to address this issue is to combine cameras with cost-effective millimeter-wave radar sensors， enabling more reliable multimodal 3D object detection. An effective approach to address this problem is to combine cameras with cost-effective millimeter-wave radar sensors， enabling more reliable multimodal 3D object detection， which not only improves the accuracy of environmental perception but also enhances the system's robustness and safety. In this paper， an autonomous driving perception algorithm based on the fusion of millimeter-wave radar and cameras， named HPR-Det （historical pillar of ray camera-radar fusion bird’s eye view for 3D object detection） is proposed. Specifically， a radar BEV （bird's eye view） feature extraction module called Radar-PRANet （radar point RCS attention net） is designed firstly. It comprises a dual-stream radar backbone that extracts radar features with two representations， and an RCS-aware BEV encoder that distributes radar features into the BEV space based on radar-specific RCS characteristics. Secondly， Historical radar of Object Prediction paradigm is adopted， designing both long-term and short-term decoders that operate only during training， thus avoiding additional inference overhead. Due to the sparsity of the input data in this network， multimodal historical multi-frame input is introduced to facilitate more accurate BEV feature learning. Lastly， the millimeter-wave-optimized ray denoising method is proposed， which utilizes the information from the current frame’s millimeter-wave radar point cloud as prior knowledge to assist in proposal generation， thereby enhancing the query feature representation for the camera. The proposed algorithm is trained and validated on the large-scale public dataset nuScenes， with the NDS reaching 56.7% on the backbone of Resnet50.

Key words: autonomous driving, deep learning, object detection, multi-sensor fusion

Chenyu Liu,Hai Wang,Yingfeng Cai,Long Chen. Multi-object Detection Algorithm Based on Camera and Radar Fusion for Autonomous Driving Scenarios[J].Automotive Engineering, 2025, 47(5): 829-838.

Figures/Tables 11

References 24

1	王海，张桂荣，罗彤，等. 面向自动驾驶道路场景中异常案例的多模态数据挖掘算法［J］. 汽车工程， 2024， 46（7）： 1239-1248.
	WANG H， ZHANG G， LUO T， et al. A multi-modal data mining algorithm for corner case of automatic driving road scene［J］. Automotive Engineering， 2024， 46（7）： 1239-1248.
2	王海，李建国，蔡英凤，等. 基于激光雷达点云的动态驾驶场景多任务分割网络［J］. 汽车工程， 2024， 46（9）： 1608-1616.
	WANG H， LI J， CAI Y， et al. A LiDAR-based dynamic driving scene multi-task segmentation network［J］. Automotive Engineering， 2024， 46（9）： 1608-1616.
3	陶乐，王海，蔡英凤，等. 面向自动驾驶场景的多目标点云检测算法［J］. 汽车工程， 2024， 46（7）： 1208-1218.
	TAO L， WANG H， CAI Y， et al. Multi-object detection algorithm based on point cloud for autonomous driving scenarios［J］. Automotive Engineering， 2024， 46（7）： 1208-1218.
4	PHILION J， FIDLER S. Lift， splat， shoot： encoding images from arbitrary camera rigs by implicitly unprojecting to 3D［M］∥VEDALDI A， BISCHOF H， BROX T， et al. Computer vision-ECCV 2020： Vol. 12359. Cham： Springer International Publishing， 2020： 194-210.
5	LI Z， WANG W， LI H， et al. BEVFormer： learning bird’s-eye-view representation from lidar-camera via spatiotemporal transformers［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2024.
6	HUANG J， HUANG G， ZHU Z， et al. BEVDet： high-performance multi-camera 3D object detection in bird-eye-view［J］. arXiv， 2022.
7	ZHENG L， LI S， TAN B， et al. Rcfusion： fusing 4D radar and camera with bird’s-eye view features for 3-d object detection［J］. IEEE Transactions on Instrumentation and Measurement， 2023， 72： 1-14.
8	ZHOU T， CHEN J， SHI Y， et al. Bridging the view disparity between radar and camera features for multi-modal fusion 3D object detection［J］. IEEE Transactions on Intelligent Vehicles， 2023， 8（2）： 1523-1535.
9	WU D， LIAO M W， ZHANG W T， et al. YOLOP： you only look once for panoptic driving perception［J］. Machine Intelligence Research， 2022， 19（6）： 550-562.
10	LANG A H， VORA S， CAESAR H， et al. Pointpillars： fast encoders for object detection from point clouds［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2019： 12697-12705.
11	YANG C， CHEN Y， TIAN H， et al. BEVFormer v2： adapting modern image backbones to bird’s-eye-view recognition via perspective supervision［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2023： 17830-17839.
12	QI C R， SU H， MO K， et al. PointNet： deep learning on point sets for 3D classification and segmentation［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 652-660.
13	LIN Z， LIU Z， XIA Z， et al. RCBEVDet： radar-camera fusion in bird’s eye view for 3D object detection［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2024： 14928-14937.
14	DAO T， FU D， ERMON S， et al. FlashAttention： fast and memory-efficient exact attention with IO-awareness［J］. Advances in Neural Information Processing Systems， 2022， 35： 16344-16359.
15	ZHU X， SU W， LU L， et al. Deformable DETR： deformable transformers for end-to-end object detection［J］. arXiv， 2021.
16	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［J］. Advances in Neural Information Processing Systems， 2017， 30.
17	LIU F， HUANG T， ZHANG Q， et al. Ray denoising： depth-aware hard negative sampling for multi-view 3D object detection［M］∥LEONARDIS A， RICCI E， ROTH S， et al. Computer Vision - ECCV 2024： Vol. 15107. Cham： Springer Nature Switzerland， 2025： 200-217.
18	DAO T. FlashAttention-2： faster attention with better parallelism and work partitioning［J］. arXiv， 2023.
19	ZHU B， JIANG Z， ZHOU X， et al. Class-balanced grouping and sampling for point cloud 3D object detection［J］. arXiv， 2019.
20	KIM Y， SHIN J， KIM S， et al. CRN： camera radar net for accurate， robust， efficient 3D perception［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2023： 17615-17626.
21	UHRIG J， SCHNEIDER N， SCHNEIDER L， et al. Sparsity invariant CNNS［C］. 2017 International Conference on 3D Vision （3DV）. IEEE， 2017： 11-20.
22	PARK D， AMBRUS R， GUIZILINI V， et al. Is pseudo-lidar needed for monocular 3D object detection？［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 3142-3152.
23	LI Y， GE Z， YU G， et al. BEVDepth： acquisition of reliable depth for multi-view 3D object detection［C］. Proceedings of the AAAI Conference on Artificial Intelligence： p37. 2023： 1477-1485.
24	WANG J， GAO Z， ZHANG Y， et al. Real-time detection and location of potted flowers based on a ZED camera and a YOLO V4-tiny deep learning algorithm［J］. Horticulturae， 2021， 8（1）： 21.

Method	Input	Backbone	Image Size	NDS	mAP	mATE	mASE	mAOE	mAVE	mAAE
CenterPoint-P	L	Pillars		59.8	49.4	0.320	0.262	0.377	0.334	0.198
CenterPoint-V	L	Voxel		65.3	56.9	0.285	0.253	0.323	0.272	0.186
BEVDet	C	R50	256 × 704	39.2	31.2	0.691	0.272	0.523	0.909	0.247
CenterFusion	C+R	DLA34	256 × 704	47.5	35.1	0.649	0.263	0.455	0.540	0.142
BEVDepth	C	R50	256 × 704	47.3	35.0	0.639	0.267	0.479	0.438	0.160
RCBEV4d	C+R	Swin-T	448 × 800	49.7	38.1	0.645	0.265	0.455	0.406	0.176
CRAFT	C+R	DLA34	448 × 800	51.7	41.1	0.611	0.240	0.410	0.282	0.168
SOLOFusion	C+R	R50	256 × 704	53.4	42.2	0.580	0.246	0.411	0.252	0.188
CRN	C+R	R18	256 × 704	54.3	44.8	0.518	0.283	0.552	0.279	0.180
CRN	C+R	R50	256 × 704	56.0	49.0	0.487	0.277	0.542	0.344	0.197
ours	C+R	R50	256 × 704	57.1	45.5	0.488	0.274	0.211	0.190	0.188

Baseline	Radar-Input	Radar-PRANet	Radar-Ray Denoise	HrOP	NDS
√					47.3%
√	√				56.0%
√	√	√			55.8%
√	√	√		√	56.7%
√	√	√	√	√	57.1%

[1]	Kai Gao,Xinyu Liu,Lin Hu,Xiangming Huang,Tiefang Zou,Peng Liu. Vehicle Trajectory Prediction with Spatial-Temporal Interaction Based on Sparse Attention [J]. Automotive Engineering, 2025, 47(5): 809-819.
[2]	Qirui Qin,Hai Wang,Yingfeng Cai,Long Chen,Yicheng Li. Real-Time Instance Segmentation Algorithm for Autonomous Driving Based on Instance Activation Maps [J]. Automotive Engineering, 2025, 47(4): 614-624.
[3]	Yihong Zhang,Mingen Zhong,Jiawei Tan,Kang Fan,Zhengfeng Li. Dense Traffic Object Detection Based on Histogram Feature Distillation [J]. Automotive Engineering, 2025, 47(4): 636-644.
[4]	Peicheng Shi,Runshuai Ge,Chakir Chadia,Xinlong Dong,Taonian Liang,Aixi Yang. PolarDet: An End-to-End 3D Object Detection Algorithm in Polar Coordinates Based on Position and Semantic Information Weighting [J]. Automotive Engineering, 2025, 47(3): 430-439.
[5]	Jinhui Suo, Xiaowei Wang, Peiwen Jiang, Chi Ding, Ming Gao, Yougang Bian. Domain Adaptive Visual Object Detection for Autonomous Driving Based on Multi-granularity Relation Reasoning [J]. Automotive Engineering, 2025, 47(2): 201-210.
[6]	Jiangkun Li,Ruixue Zong,Weiwen Deng,Ying Wang,Juan Ding. Directed Graph-Based Method for Evaluating Similarity in Urban Intersection Scenarios [J]. Automotive Engineering, 2025, 47(1): 23-34.
[7]	Daofei Li,Hao Pan. Application of Scenario Complexity Evaluation in Trajectory Prediction and Automated Driving Decision-Making [J]. Automotive Engineering, 2024, 46(9): 1556-1563.
[8]	Hai Wang,Jianguo Li,Yingfeng Cai,Long Chen. A LiDAR-Based Dynamic Driving Scene Multi-task Segmentation Network [J]. Automotive Engineering, 2024, 46(9): 1608-1616.
[9]	Jianan Zhang,Zhaozheng Hu,Jie Meng,Huahua Hu,Jie Zuo. Distributed Simulation Platform Architecture and Application of Autonomous Driving for Vehicle-Road-Map Collaboration [J]. Automotive Engineering, 2024, 46(8): 1335-1345.
[10]	Le Tao,Hai Wang,Yingfeng Cai,Long Chen. Multi-object Detection Algorithm Based on Point Cloud for Autonomous Driving Scenarios [J]. Automotive Engineering, 2024, 46(7): 1208-1218.
[11]	Linhui Li,Yifan Fu,Ting Wang,Xuecheng Wang,Jing Lian. Trajectory Prediction Method Enhanced by Self-supervised Pretraining [J]. Automotive Engineering, 2024, 46(7): 1219-1227.
[12]	Luxing Li,Chao Wei. Multi-sensor Data Fusion for Intelligent Vehicles Based on Tripartite Graph Matching [J]. Automotive Engineering, 2024, 46(7): 1228-1238.
[13]	Hai Wang,Guirong Zhang,Tong Luo,Meng Qiu,Yingfeng Cai,Long Chen. A Multi-modal Data Mining Algorithm for Corner Case of Automatic Driving Road Scene [J]. Automotive Engineering, 2024, 46(7): 1239-1248.
[14]	Zhigang Yang,Yujing Li,Chao Xia,Mengjia Wang,Lei Yu. A High Time-Resolution Reconstruction on the Automotive Turbulent Wake Based on LSTM-POD [J]. Automotive Engineering, 2024, 46(7): 1302-1313.
[15]	Lisheng Jin,Hongyu Zhang,Baicang Guo. Semi Solid-State LiDAR Object Detection Algorithm Enhanced by Feature Stability Enhancement [J]. Automotive Engineering, 2024, 46(6): 1015-1024.

Multi-object Detection Algorithm Based on Camera and Radar Fusion for Autonomous Driving Scenarios

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 24

Related Articles 15

Metrics

Comments

Recommended 0