基于D-S证据理论的多模态结果级融合框架研究

doi:10.19562/j.chinasae.qcgc.2023.10.004

Abstract

Abstract:

Multi-modal fusion perception is one of the research hotspots of automatic driving. However， in complex traffic environment， due to the interference of weather， illumination and other external factors， the target recognition may be wrong， leading to inevitable classification conflict during fusion. Therefore， this paper proposes a multi-modal late fusion framework based on D-S Evidence Theory. The confidence score of deep neural network is output and used as the probability density function of D-S evidence theory. By modifying the classification result of conflict through evidence combination， this framework can solve the classification conflict problem of fusion between any mode. The framework is verified by experiments based on KITTI data set. The results show that the fusion result of the framework output can increase by about 8% compared with the mAP value of a single sensing network， with the fusion result of Yolov3 and Pointpillar increasing by 32% compared with the single sensing result of Pointpillar， which can effectively solve the classification conflict after multi-mode fusion in the complex traffic environment.

Key words: D-S evidence theory, multimodal fusion, object recognition, conflict classification

Teng Cheng,Dengchao Hou,Qiang Zhang,Qin Shi,Ligang Guo. Research on Multi-modal Late Fusion Framework Based on D-S Evidence Theory[J].Automotive Engineering, 2023, 45(10): 1815-1823.

Figures/Tables 12

References 30

1	张新钰，邹镇洪，李志伟，等.面向自动驾驶目标检测的深度多模态融合技术［J］.智能系统学报，2020，15（4）：758-771.
	ZHANG Xinyu， ZOU Zhenhong， LI Zhiwei， et al. Deep multi-modal fusion in object detection for autonomous driving［J］. CAAI Transactions on Intelligent Systems， 2020， 15（4）： 758–771.
2	ZHOU T， JIANG K， XIAO Z， et al. Object detection using multi-sensor fusion based on deep learning［C］.19th COTA International Conference of Transportation Professionals， 2019.
3	VORA S， LANG A H， HELOU B， et al. Pointpainting： sequential fusion for 3D object detection［C］.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 4604-4612.
4	CHEN X， MA H， WAN J， et al. Multi-view 3D object detection network for autonomous driving［C］.Proceedings of the IEEE conference on Computer Vision and Pattern Recognition， 2017： 1907-1915.
5	QI C R， LIU W， WU C， et al. Frustum pointnets for 3D object detection from RGB-D data［C］.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2018： 918-927.
6	PANG S， MORRIS D， RADHA H. CLOCs： camera-LiDAR object candidates fusion for 3D object detection［C］.2020 IEEE/RSJ International Conference on Intelligent Robots and Systems （IROS）. IEEE， 2020： 10386-10393.
7	纪嘉树. 基于多传感器融合的无人驾驶环境感知技术研究［D］.济南：山东大学，2022.
	JI Jiashu. Research on environment perception technology of unmanned driving based on multi-sensor fusion［D］. Jinan：Shandong University， 2022.
8	甘耀东，郑玲，张志达，等.融合毫米波雷达与深度视觉的多目标检测与跟踪［J］.汽车工程，2021，43（7）：1022-1029，1056.DOI：10.19562/j.chinasae.qcgc.2021.07.009.
	GAN Yaodong， ZHENG Ling， ZHANG Zhida， et al. Multi-target detection and tracking with fusion of millimeter-wave radar and deep vision［J］. Automotive Engineering， 2021，43（7）：1022-1029，1056.DOI：10.19562/j.chinasae.qcgc.2021.07.009.
9	李哲，于梦茹.基于多种LBP特征集成学习的车标识别［J］.计算机工程与应用，2019，55（20）：134-138.
	LI Zhe， YU Mengru. Vehicle-logo recognition based on ensemble learning with multiple LBP features［J］. Computer Engineering and Applications， 2019， 55（20）：134-138.
10	王钦民，李宽，杨灿群.一种基于分类器投票的车牌定位方法［J］.计算机工程与科学，2016，38（6）：1200-1206.
	WANG Qinmin， LI Kuan， YANG Canqun. A license plate location method based on classifier voting［J］. Computer Engineering & Science， 2016， 38（6）： 1200-1206.
11	刘丽丽，周绍光，丁倩，等.基于最大投票融合的高光谱影像半监督分类［J］.地理空间信息，2020，18（5）：20-25，6.
	LIU Lili， ZHOU Shaoguang， DING Qian，et al. Semi-supervised classification of hyperspectral images based on maximum voting fusion［J］. Geospatial Information， 2020， 18（5）： 20-25，6.
12	李悦. 多传感器信息融合在刀具磨损在线监测中的应用研究［D］.太原：太原科技大学，2020.
	LI Yue. Research on application of multi-sensor information fusion in tool wear condition monitoring［D］.Taiyuan ：Taiyuan University of Science and Technology， 2020.
13	赵宏伟，何劲松.基于贝叶斯框架融合深度信息的显著性检测［J］.光电工程，2018，45（2）：13-20.
	ZHAO Hongwei， HE Jinsong. Saliency detection method fused depth information based on Bayesian framework［J］. Opto-Electronic Engineering， 2018， 45（2）： 13-20.
14	陈雪敏. 基于贝叶斯融合的图像显著性检测［D］.天津：河北工业大学，2019.
	CHEN Xuemin. Image saliency detection based on baytes integration［D］. Tianjin：Hebei University of Technology， 2019.
15	李伟，周靖，杜秀梅，等.基于D-S证据信息融合方法的全地形车行驶工况辨识［J］.重庆大学学报，2022，45（3）：1-11.
	LI Wei， ZHOU Jing， DU Xiumei， et al.Driving condition identification of all-terrain vehicles based on D-S evidence information fusion method［J］. Journal of Chongqing University， 2022， 45（3）： 1-11.
16	姬晓飞，石宇辰，王昱，等.D-S理论多分类器融合的光学遥感图像多目标识别［J］.电子测量与仪器学报，2020，34（5）：127-132.
	JI Xiaofei， SHI Yuchen， WANG Yu， et al. D-S theory based multi-classifier fusion optical remote sensing image target recognition［J］. Journal of Electronic Measurement and Instrument， 2020， 34（5）： 127-132.
17	SHAFER G. Dempster-shafer theory［J］. Encyclopedia of Artificial Intelligence， 1992， 1： 330-331.
18	周文文. 多分类器融合算法研究与仿真系统实现［D］.南京：南京航空航天大学，2021.
	ZHOU Wenwen. Research on multi-classifier fusion algorithm and realization of simulation system［D］. Nanjing ：Nanjing University Of Aeronautics And Astronautics， 2021.
19	TURHAN H I， DEMIREKLER M， GUNAY M. A novel methodology for target classification based on dempster-shafer theory［C］.Belief Functions： Theory and Applications： Third International Conference， BELIEF 2014， Oxford， UK， September 26-28， 2014. Proceedings 3. Springer International Publishing， 2014： 393-402.
20	CHAVEZ-GARCIA R O， AYCARD O. Multiple sensor fusion and classification for moving object detection and tracking［J］. IEEE Transactions on Intelligent Transportation Systems， 2015， 17（2）： 525-534.
21	CHAVEZ-GARCIA R O， VU T D， AYCARD O. Fusion at detection level for frontal object perception［C］.2014 IEEE Intelligent Vehicles Symposium Proceedings. IEEE， 2014： 1225-1230.
22	ZHU C， QIN B， XIAO F， et al. A fuzzy preference-based Dempster-Shafer evidence theory for decision fusion［J］. Information Sciences， 2021， 570： 306-322.
23	DRISS M， KOUBAA A， ATITALLAH S B， et al. Fusion of convolutional neural networks based on Dempster–Shafer theory for automatic pneumonia detection from chest X‐ray images［J］. International Journal of Imaging Systems and Technology， 2022， 32（2）：658-672.
24	REN S， HE K， GIRSHICK R， et al. Faster R-CNN： towards real-time object detection with region proposal networks［J］. Advances in Neural Information Processing Systems， 2015， 28.
25	SHI S， GUO C， JIANG L， et al. PV-RCNN： point-voxel feature set abstraction for 3D object detection［C］.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 10529-10538.
26	REDMON J， DIVVALA S， GIRSHICK R， et al. You only look once： unified， real-time object detection［C］.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2016： 779-788.
27	REDMON J， FARHADI A. Yolov3： an incremental improvement［J］. arXiv preprint arXiv：， 2018.
28	BOCHKOVSKIY A， WANG C Y， LIAO H Y M. Yolov4： optimal speed and accuracy of object detection［J］. arXiv preprint arXiv：， 2020.
29	DUAN K， BAI S， XIE L， et al. Centernet： keypoint triplets for object detection［C］.Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 6569-6578.
30	MURPHY C K. Combining belief functions when evidence conflicts［J］. Decision Support Systems， 2000， 29（1）：1-9.

[1]	HAO Han, WANG Si-南, LI Xiao, LIU Zong-Wei, ZHAO Fu-Quan. [J]. , 2017, 39(1): 1 -8 .
[2]	ZHANG Lei, ZHANG Jin-Qiu, LUO Tao, BI Zhan-Dong, YAO Jun. [J]. , 2016, 38(12): 1494 -1499 .
[3]	ZHOU Kai, ZANG Jing-Lun. [J]. , 2017, 39(2): 127 -132 .
[4]	WANG Guo-Chun, LI Qi-Qi, DAI Jiang-Lu, CHENG Ai-Guo, LI Tie-Zhu. [J]. , 2017, 39(2): 168 -173 .
[5]	RONG Bing, XIAO Pan, ZHOU Jian-Wen, LIU Yi-Lu. [J]. , 2017, 39(2): 214 -219 .
[6]	CAO Li-Bo, CHEN Zheng, YAN Ling-Bo, QIN Qin, ZHANG Rui-Feng. [J]. , 2017, 39(2): 225 -231 .
[7]	HONG Qiu-Cai, LIU Wei-Guo, ZHOU Da-Yong. [J]. , 2017, 39(2): 232 -236 .
[8]	XU Feng-Xiang, TIAN Xuan-Yi. [J]. , 2017, 39(2): 237 -242 .
[9]	WANG Bin, LI Tie, ZHANG Xiao-Qing, GUO Tao, SHI Qi. [J]. , 2017, 39(3): 256 -261 .
[10]	CHEN Yong, ZHANG Zhen-Dong, CHENG Qiang. [J]. , 2017, 39(3): 262 -268 .

感知网络	AP/％			mAP@0.5/%
感知网络	机动车	非机动车	行人	mAP@0.5/%
Second	84.18	57.94	58.92	67.01
PointPillar	82.62	51.59	52.55	62.25
PointRCNN	80.58	55.95	58.66	65.06
Yolov3	92.49	92.91	87.79	91.06
FasterRCNN	85.83	58.68	61.67	68.73
CenterNet	83.57	66.82	52.29	67.56
Voxel-RCNN	85.30	59.20	53.13	65.88

检测目标	检测工具	置信度/%
检测目标	检测工具	车	人	非机动车
非机动车1	相机	0.06	73.77	9.49
	激光雷达	0.04	0.04	66.43
	本文融合方法	0.000 4	0.5	99.48

检测目标	检测工具	置信度/%
检测目标	检测工具	车	人	非机动车
非机动车1	相机	0.008 988	0.036 51	84.652 1
	激光雷达	0.013 549	40.912 1	0.014 193
	D-S融合	0.021 8	49.990 4	49.987 6
	改进D-S融合	0.000 9	49.664 4	49.891 6

多模态感知网络组合	AP/%						mAP@0.5/%
	机动车		非机动车		行人		mAP@0.5/%
	本文融合方法	投票法融合	本文融合方法	投票法融合	本文融合方法	投票法融合	本文融合方法	投票法融合
FasterRCNN、Second	90.58	90.10	70.36	68.91	69.37	66.45	76.77	75.16
FasterRCNN、PointRCNN	90.95	90.44	68.96	68.26	68.09	66.87	76.00	75.19
FasterRCNN、Pointpillar	90.33	90.10	65.86	64.31	67.91	66.06	74.49	73.49
CenterNet、PointRCNN	90.59	90.53	72.13	71.38	66.06	64.82	76.26	75.38
CenterNet、Second	90.94	90.94	71.93	70.92	66.54	65.33	76.47	75.73
CenterNet、Pointpillar	90.33	90.30	69.43	67.40	62.75	61.27	74.17	72.99
Yolov3、Second	97.35	96.90	94.86	94.34	92.05	91.21	94.75	94.15
Yolov3、Pointpillar	97.25	96.85	94.86	94.46	92.05	90.62	94.75	93.98
Yolov3、PointRCNN	97.19	96.79	95.11	94.90	91.16	90.48	94.49	94.06
Voxel-RCNN、Yolov3	97.43	97.05	95.12	94.67	91.33	90.52	94.62	94.08
Voxel-RCNN、CenterNet	91.62	91.55	74.73	73.87	64.07	63.95	76.81	76.46
Voxel-RCNN、FasterRCNN	90.64	90.43	69.67	68.94	67.36	67.01	75.89	75.46

Research on Multi-modal Late Fusion Framework Based on D-S Evidence Theory

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 30

Related Articles 5

Metrics

Comments

Recommended 10

[1]	Jie Hu,Xiao Zhang,Min Wei,Lin Chen,Haihua Qing,Changbin Gao. Vehicle Fault Location Method Based on Fuzzy BN and Improved Evidence Theory [J]. Automotive Engineering, 2023, 45(10): 1975-1983.
[2]	Runze Feng,Kun Jiang,Weiguang Yu,Diange Yang. Chinese Traffic Sign Recognition Based on Two-stage Classification Algorithm [J]. Automotive Engineering, 2022, 44(3): 434-441.
[3]	Zhangpeng Gong,Guoye Wang,Shi Yu. The Algorithm of Multi⁃Category Object Recognition in Road Scene Based on Voxel Network [J]. Automotive Engineering, 2021, 43(4): 469-477.
[4]	Teng Cheng,Lei Sun,Dengchao Hou,Qin Shi,Junning Zhang,Jiong Chen,He Huang. Multi-level and Multi-modal Target Detection Based on Feature Fusion [J]. Automotive Engineering, 2021, 43(11): 1602-1610.
[5]	Huang Wei, Liu Haijiang & Li Min. Identification Method for Feature Points of Drivability Evaluation Indicators in Starting Condition [J]. , 2019, 41(3): 259-265.

1.将目标数量较少的模态初始化为左顶标，另一模态初始化为右顶标
2.迭代所有目标，交并比小于阈值连成边，建立带权二分图，权值为交并比，初始化邻接矩阵
3.根据边权初始化左顶标与右顶标
4.使用匈牙利算法获得完备匹配
5.若未找到完备匹配则修改可行顶标的值
6.重复步骤（4）和步骤（5）直到找到相等子图的完备匹配为止
7.将未匹配的目标存入结果集
8.将匹配的目标融合后存入结果集