基于三分图匹配的智能车辆多传感器数据融合

doi:10.19562/j.chinasae.qcgc.2024.07.010

Abstract

Abstract:

Multi-sensor fusion is an effective way to improve intelligent vehicle perception. For the data-matching problem of the three types of sensors of LiDAR， millimeter-wave radar， and camera， traditional methods such as bipartite graph matching can’t achieve high precision， with poor matching robustness. Therefore， a multi-sensor data fusion algorithm for intelligent vehicles based on tripartite graph matching is proposed in this paper. The problem of data matching of the three sensors is abstracted as a weighted tripartite graph-matching problem. By using Lagrange relaxation， the original problem space is decomposed into subspaces， the weights of vertices and edge inside which are determined then by the cost matrix model. Furthermore， combining the perceptual error model and likelihood estimation， the posterior distribution of perceptual errors is determined. Ultimately the Lagrange Multiplier （LM） model is used for data matching. Finally， the effectiveness of the proposed matching algorithm is validated by the nuScenes training dataset and real-world vehicle tests. On the dataset， the proposed algorithm improves F1 scores by 7.2% compared to common algorithms. In various real-world vehicle scenarios， the proposed algorithm shows excellent perceptual accuracy and robustness across.

Key words: Lagrange relaxation, multi-sensor fusion, perception error model, tripartite graph matching

Luxing Li,Chao Wei. Multi-sensor Data Fusion for Intelligent Vehicles Based on Tripartite Graph Matching[J].Automotive Engineering, 2024, 46(7): 1228-1238.

Figures/Tables 13

项目	噪声类型	标准差	极值	P值
$σ u$	乘法	0.015	1.07	0.196
$σ ν$	乘法	0.020	1.07	0.238
$σ d$	乘法	0.017	1.05	0.053 2
$σ α$	加法	0.021	2	0.425
$σ θ$	加法	0.025	2	0.294
$σ w$	加法	0.019	0.1	0.162
$σ h$	加法	0.029	0.1	0.423
$σ l$	加法	0.018	0.1	0.239
$σ ω$	加法	0.027	2	0.091

References 29

1	BADUE C， GUIDOLINI R， CARNEIRO R V， et al. Self-driving cars： a survey［J］. Expert Systems with Applications， 2021， 165（1）： 1-34.
2	TEETI I， KHAN S， SHAHBAZ A， et al. Vision-based intention and trajectory prediction in autonomous vehicles： a survey［C］. International Joint Conference on Artificial Intelligence， 2022： 5630-5637.
3	WANG Z， WU Y， NIU Q. Multi-sensor fusion in automated driving： a survey［J］. IEEE Access， 2019， 8： 2847-2868.
4	LI Y， DENG J， ZHANG Y， et al. EZFusion： a close look at the integration of LiDAR， millimeter-wave radar， and camera for accurate 3D object detection and tracking［J］. IEEE Robotics and Automation Letters， 2022， 7（4）： 11182-11189.
5	CAESAR H， BANKITI V， LANG A H， et al. Nuscenes： a multimodal dataset for autonomous driving［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 11621-11631.
6	SUN P， KRETZSCHMAR H， DOTIWALLA X， et al. Scalability in perception for autonomous driving： waymo open dataset［C］.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 2446-2454.
7	SINGH A. Vision-radar fusion for robotics bev detections： a survey［C］. 2023 IEEE Intelligent Vehicles Symposium （IV）. IEEE， 2023： 1-7.
8	TANG Q， LIANG J， ZHU F. A comparative review on multi-modal sensors fusion based on deep learning［J］. Signal Processing， 2023： 109165.
9	LIN C， TIAN D， DUAN X， et al. CL3D： camera-LiDAR 3D object detection with point feature enhancement and point-guided fusion［J］. IEEE Transactions on Intelligent Transportation Systems， 2022， 23（10）： 18040-18050.
10	CHEN W， TIAN W， XIE X， et al. RGB image-and LIDAR-based 3D object detection under multiple lighting scenarios［J］. Automotive Innovation， 2022， 5（3）： 251-259.
11	CHEN X， ZHANG T， WANG Y， et al. Futr3D： a unified sensor fusion framework for 3D detection［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2023： 172-181.
12	LI H， ZHANG Z， ZHAO X， et al. Enhancing multi-modal features using local self-attention for 3D object detection［C］. European Conference on Computer Vision. Cham： Springer Nature Switzerland， 2022： 532-549.
13	YIN T， ZHOU X， KRÄHENBÜHL P. Multimodal virtual point 3D detection［J］. Advances in Neural Information Processing Systems， 2021， 34： 16494-16507.
14	JIANG K， SHI Y， ZHOU T， et al. PTMOT： a probabilistic multiple object tracker enhanced by tracklet confidence for autonomous driving［J］. Automotive Innovation， 2022， 5（3）： 260-271.
15	ZHOU Y， DONG Y， HOU F， et al. Review on millimeter-wave radar and camera fusion technology［J］. Sustainability， 2022， 14（9）： 5114.
16	赵树廉，吴思宇，赵鹏云，等. 基于最小临近点迹和航迹关联的多源目标融合方法［J］. 汽车工程学报， 2022， 12（5）： 593-603.
	ZHAO S L， WU S Y， ZHAO P Y， et al. Multi-source target fusion method based on the nearest positioning and trajectory data association［J］. Chinese Journal of Automotive Engineering， 2022， 12（5）： 593-603.
17	刘志强，张光林，郑曰文，等. 基于检测无迹信息融合算法的多传感融合方法［J］. 汽车工程， 2020， 42（7）： 854-859.
	LIU Z Q， ZHANG G L， ZHENG Y W， et al. Multi-sensor fusion method based on checking unscented information fusion algorithm［J］. Automotive Engineering， 2020， 42（7）： 854-859.
18	HAO X， XIA Y， YANG H， et al. Asynchronous information fusion in intelligent driving systems for target tracking using cameras and radars［J］. IEEE Transactions on Industrial Electronics， 2022， 70（3）： 2708-2717.
19	文礼杰，谢荣，许军立，等. 基于相容系数的多传感器融合目标识别方法［J］. 电子测量与仪器学报， 2023， 37（4）： 142-153.
	WEN L J， XIE R， XU J L， et al. Multi-sensor fusion target recognition method based on compatibility coefficient ［J］. Journal of Electronic Measurement and Instrumentation， 2023， 37（4）： 142-153.
20	ZHANG B， SMALL D S， LASATER K B， et al. Matching one sample according to two criteria in observational studies［J］. Journal of the American Statistical Association， 2023， 118（542）： 1140-1151.
21	LIU Y， WANG Z， PENG L， et al. A detachable and expansible multisensor data fusion model for perception in level 3 autonomous driving system［J］. IEEE Transactions on Intelligent Transportation Systems， 2022， 24（2）： 1814-1827.
22	宋强，熊伟，何友. 多传感器多目标系统误差融合估计算法［J］. 北京航空航天大学学报， 2012， 38（6）： 835-841.
	SONG Q， XIONG W， HE Y. Multi-sensor multi-target systematic bias fusion estimation algorithm［J］. Journal of Beijing University of Aeronautics and Astronautics， 2012， 38（6）： 835-841.
23	LIU Q， ZHOU W， ZHANG Y， et al. Multi-target detection based on multi-sensor redundancy and dynamic weight distribution for driverless cars［C］. 2021 International Conference on Communications， Information System and Computer Engineering （CISCE）. IEEE， 2021： 229-234.
24	IGNATIOUS H A， EL-SAYED H， KULKARNI P. Multilevel data and decision fusion using heterogeneous sensory data for autonomous vehicles［J］. Remote Sensing， 2023， 15（9）： 2256.
25	IVANOV A， TONCHEV K， POULKOV V， et al. Graph-based resource allocation for integrated space and terrestrial communications［J］. Sensors， 2022， 22（15）： 5778.
26	YANG K， TANG X， LI J， et al. Uncertainties in onboard algorithms for autonomous vehicles： challenges， mitigation， and perspectives［J］. IEEE Transactions on Intelligent Transportation Systems， 2023， 24（9）： 8963-8987.
27	PIAZZONI A， CHERIAN J， SLAVIK M， et al. Modeling perception errors towards robust decision making in autonomous vehicles［C］. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence， 2021： 3494-3500.
28	XU L， QIAN B， HU R， et al. Lagrange heuristic algorithm incorporated with decomposition strategy for green multi-depot heterogeneous-fleet vehicle routing problem［C］. International Conference on Intelligent Computing. Singapore： Springer Nature Singapore， 2023： 537-548.
29	ULTRALYTICS. YOLOv5［DB/OL］. （2024-01-22）. https：//github.com/ultralytics/yolov5/tree/v5.0.

元素	传感器
元素	激光雷达	毫米波雷达	相机
顶点元素	3D坐标	BEV坐标	2D坐标

子空间	正态检验	P值	均值	标准差
1	Shapiro-wilk	0.343	0.948	0.016
2	Shapiro-wilk	0.111	0.956	0.015
3	Shapiro-wilk	0.070	0.920	0.026

子空间	均值	标准差	最小值	最大值
1	0.917	0.017	0.729	0.964
2	0.927	0.015	0.750	0.937
3	0.873	0.025	0.623	0.945

序号	算法	F1/%
1	（LR）+C^［24］	88.1
2	（CR）+L^［25］	88.2
3	（LC）+R^［22］	86.8
4	LM-μ₁	90.3
5	LM-μ₂	90.2
6	LM-μ₃	88.9
7	LM+	91.7
8	LM++	94.0

[1]	Zhenfeng Pu, Liang Tang, Wenbin Shangguan, Weiwei Wang, Kaihong Jiang. Research on the Estimation of Vehicle Speed Under Low-Speed Conditions Based on Multi-sensor Information [J]. Automotive Engineering, 2023, 45(7): 1235-1243.
[2]	Jingwei Zhang,Tiejun Liu,Rengang Li,Dan Liu,Jinglin Zhan,Hongwei Kan. A Temporal Calibration Method for Multi-Sensor Fusion of Autonomous Vehicles [J]. Automotive Engineering, 2022, 44(2): 215-224.
[3]	Yang Mingliang, Kou Shengjie, Lu Yong, Yu Chunlei, Jiang Kun & Yang Diange. Research on Lane Change Warning Model Based on Multi-sensor Fusion [J]. Automotive Engineering, 2019, 41(10): 1197-1203.

Multi-sensor Data Fusion for Intelligent Vehicles Based on Tripartite Graph Matching

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 29

Related Articles 3

Metrics

Comments

Recommended 0