基于多信息融合网络的行人轨迹预测方法

doi:10.19562/j.chinasae.qcgc.2024.11.004

Abstract

Abstract:

With the continuous development of autonomous driving technology， accurately predicting the future trajectories of pedestrians has become a critical element in ensuring system safety and reliability. However， most existing studies on pedestrian trajectory prediction rely on fixed camera perspectives， which limits the comprehensive observation of pedestrian movement and thus makes them unsuitable for direct application to pedestrian trajectory prediction under the ego-vehicle perspective in autonomous vehicles. To solve the problem， in this paper a pedestrian trajectory prediction method under the ego-vehicle perspective based on the Multi-Pedestrian Information Fusion Network （MPIFN） is proposed， which achieves accurate prediction of pedestrians' future trajectories by integrating social information， local environmental information， and temporal information of pedestrians. In this paper， a Local Environmental Information Extraction Module that combines deformable convolution with traditional convolutional and pooling operations is constructed， aiming to more effectively extract local information from complex environment. By dynamically adjusting the position of convolutional kernels， this module enhances the model’s adaptability to irregular and complex shapes. Meanwhile， the pedestrian spatiotemporal information extraction module and multimodal feature fusion module are developed to facilitate comprehensive integration of social and environmental information. The experimental results show that the proposed method achieves advanced performance on two ego-vehicle driving datasets， JAAD and PSI. Specifically， on the JAAD dataset， the Center Final Mean Squared Error （CF_MSE） is 4 063， and the Center Mean Squared Error （C_MSE） is 829. On the PSI dataset， the Average Root Mean Square Error （ARB） and Final Root Mean Square Error （FRB） also achieve outstanding performance with values of 18.08/29.21/44.98 and 25.27/54.62/93.09 for prediction horizons of 0.5 s， 1.0 s， and 1.5 s， respectively.

Key words: autonomous driving, pedestrian trajectory prediction, multiple pedestrian information fusion network, ego-vehicle

Song Gao,Jianglin Zhou,Bolin Gao,Jian Lu,He Wang,Yueyun Xu. Pedestrian Trajectory Prediction Method Based on Multi-information Fusion Network[J].Automotive Engineering, 2024, 46(11): 1973-1982.

Figures/Tables 10

方法	Soc	Env	Soc-Env	PSI
方法	Soc	Env	Soc-Env	MSE↓ （0.5 s/1.0 s/1.5 s）	$C F M S E$ ↓ （1.5 s）	$C M S E$ ↓ （1.5 s）
Bayesian-LSTM^［9］	√			159/539/1 535	5 615	1 447
FOL-X^［26］		√		147/484/1 374	4 924	1 290
PIE_traj^［13］		√		110/399/1 280	4 780	1 183
BiTraP^［4］	√			93/378/1 206	4 565	1 105
eP2P^［12］	√	√
SGNet^［15］	√			82/328/1 049	4 076	996
CVTF^［27］		√		98/314/1 190	4 520	1 022
Pedformer^［28］	√	√	√	93/364/1 134	4 364	1 080
VOSTN^［29］	√			94/364/1 134	3 980	947
MPIFN （Ours）	√	√	√	81/307/1 106	4 063	829

References 35

1	MARCHETTI F， BECATTINI F， SEIDENARI L， et al. Smemo： social memory for trajectory forecasting［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2024， 46（6）： 4410-4425.
2	SUN J， LI Y， CHAI L， et al. Modality exploration， retrieval and adaptation for trajectory prediction［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 45（12）： 15051-15064.
3	SHI L， WANG L， LONG C， et al. Representing multimodal behaviors with mean location for pedestrian trajectory prediction［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 45（9）： 11184-11202.
4	YAO Y， ATKINS E， JOHNSON-ROBERSON M，et al.BiTraP： bi-directional pedestrian trajectory prediction with multi-modal goal estimation［J］. IEEE Robotics and Automation Letters，2021，6（2）：1463-1470.
5	郭景华，何智飞，罗禹贡，等. 人机混驾环境下基于深度学习的车辆切入轨迹预测［J］. 汽车工程， 2022， 44（2）： 153-160.
	GUO J H， HEI Z F， LUO Y G， et al. Vehicle cut-in trajectory prediction based on deep learning in a human-machine mixed driving environment ［J］. Automotive Engineering， 2022， 44（2）： 153-160.
6	郭景华，肖宝平，王靖瑶，等. 基于 Residual BiLSTM 网络的车辆切入意图预测研究［J］. 汽车工程， 2021， 43（7）： 971-977.
	GUO J H， XIAO B P， WANG J Y， et al. Study on vehicle cut⁃in intention prediction based on residual BiLSTM network ［J］. Automotive Engineering， 2021， 43（7）： 971-977.
7	GODARD C， MAC AODHA O， BROSTOW G J. Unsupervised monocular depth estimation with left-right consistency［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 270-279.
8	CHEN Y， SCHMID C， SMINCHISESCU C. Self-supervised learning with geometric constraints in monocular video： connecting flow， depth， and camera［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 7063-7072.
9	BHATTACHARYYA A， FRITZ M， SCHIELE B. Long-term on-board prediction of people in traffic scenes under uncertainty［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2018： 4194-4202.
10	YAO Y， XU M， CHOI C， et al. Egocentric vision-based future vehicle localization for intelligent driving assistance systems［C］. 2019 International Conference on Robotics and Automation （ICRA）. IEEE， 2019： 9711-9717.
11	YAGI T， MANGALAM K， YONETANI R， et al. Future person localization in first-person videos［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2018： 7593-7602.
12	CHEN T， JING T， TIAN R， et al. Psi： a pedestrian behavior dataset for socially intelligent autonomous car［J］. arXiv preprint arXiv：， 2021.
13	RASOULI A， KOTSERUBA I， KUNIC T， et al. Pie： a large-scale dataset and models for pedestrian intention estimation and trajectory prediction［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 6262-6271.
14	WANG C， WANG Y， XU M， et al. Stepwise goal-driven networks for trajectory prediction［J］. IEEE Robotics and Automation Letters， 2022， 7（2）： 2716-2723.
15	NEUMANN L， VEDALDI A. Pedestrian and ego-vehicle trajectory prediction from monocular camera［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2021： 10204-10212.
16	SU Y， LI Y， WANG W， et al. A unified environmental network for pedestrian trajectory prediction［C］. Proceedings of the AAAI Conference on Artificial Intelligence， 2024， 38（5）： 4970-4978.
17	FU Z， JIANG K， XIE C， et al. Summary and reflections on pedestrian trajectory prediction in the field of autonomous driving［J］. IEEE Transactions on Intelligent Vehicles， 2024.
18	HASAN F， HUANG H. MALS-Net： a multi-head attention-based LSTM sequence-to-sequence network for socio-temporal interaction modelling and trajectory prediction［J］. Sensors， 2023， 23（1）： 530.
19	桑海峰，赵梓杉，王金玉，等. 基于车辆轨迹预测对抗性攻击与鲁棒性研究［J］. 汽车工程， 2024， 46（3）： 407-417.
	SANG H F， ZHAO Z S， WANG J Y， et al. Research on adversarial attacks and robustness in vehicle trajectory prediction ［J］. Automotive Engineering， 2024， 46（3）： 407-417.
20	DAI J， QI H， XIONG Y， et al. Deformable convolutional networks［C］. Proceedings of the IEEE International Conference on Computer Vision， 2017： 764-773.
21	DU J， WANG S， MIAO H， et al. Multi-channel pooling graph neural networks［C］. IJCAI. 2021： 1442-1448.
22	GROSSBERG S. Recurrent neural networks［J］. Scholarpedia， 2013， 8（2）： 1888.
23	GRAVES A， GRAVES A. Long short-term memory［J］. Supervised Sequence Labelling with Recurrent Neural Networks， 2012： 37-45.
24	RASOULI A， KOTSERUBA I， TSOTSOS J K. Are they going to cross？ a benchmark dataset and baseline for pedestrian crosswalk behavior［C］. Proceedings of the IEEE International Conference on Computer Vision Workshops， 2017： 206-213.
25	KOSARAJU V， SADEGHIAN A， MARTÍN-MARTÍN R， et al. Social-bigat： multimodal trajectory forecasting using bicycle-gan and graph attention networks［J］. Advances in Neural Information Processing Systems， 2019， 32.
26	YAO Y， XU M， WANG Y， et al. Unsupervised traffic accident detection in first-person videos［C］. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems （IROS）. IEEE， 2019： 273-280.
27	HE Y， YANG Y， CAI Y， et al. Predicting pedestrian tracks around moving vehicles based on conditional variational transformer［J］. Proceedings of the Institution of Mechanical Engineers， Part D： Journal of Automobile Engineering， 2023： 09544070231175536.
28	RASOULI A， KOTSERUBA I. PedFormer： pedestrian behavior prediction via cross-modal attention modulation and gated multitask learning［C］. 2023 IEEE International Conference on Robotics and Automation （ICRA）. IEEE， 2023： 9844-9851.
29	WANG J， SANG H， CHEN W， et al. VOSTN： variational one-shot transformer network for pedestrian trajectory prediction［J］. Physica Scripta， 2024， 99（2）： 026002.
30	LABRíN C， URDINEZ F. Principal component analysis［M］. R for Political Data Science. Chapman and Hall/CRC， 2020： 375-393.
31	SANAGA K P， YANG M S. Unsupervised K-means clustering algorithm ［J］. IEEE Access， 2020， 8： 80716-80727.
32	HAN K， XIAO A， WU E， et al. Transformer in transformer［J］. Advances in Neural Information Processing Systems， 2021， 34： 15908-15919.
33	OKK AN U， SERBES Z A. Rainfall-runoff modeling using least squares support vector machines［J］. Environmetrics， 2012， 23（6）： 549-564.
34	TREIBER M， HENNECKE A， HELBING D. Congested traffic states in empirical observations and microscopic simulations［J］. Physical Review E， 2000， 62（2）： 1805.
35	LIU L， FENG S， FENG Y， et al. Learning-based stochastic driving model for autonomous vehicle testing［J］. Transportation Research Record， 2022， 2676（1）： 54-64.

[1]	ZHOU Wei, ZHANG Cheng-Ning, LI Jun-Qiu. [J]. , 2016, 38(12): 1407 -1414 .
[2]	LI Huan, HUANG Ying, ZHANG Fu-Jun, ZHAO Yu, GE Yan-Wu. [J]. , 2016, 38(12): 1420 -1426 .
[3]	HAO Han, WANG Si-南, LI Xiao, LIU Zong-Wei, ZHAO Fu-Quan. [J]. , 2017, 39(1): 1 -8 .
[4]	ZONG Yi-Qi, GU Zheng-Qi, LUO Ze-Min, JIANG Cai-Mao, ZHANG Qi-Dong, YANG Zhen-Dong. [J]. , 2016, 38(12): 1427 -1433 .
[5]	XU Cheng-Shan, JIANG Fa-Chao, SONG Sen-Nan, TIAN Guang-Yu. [J]. , 2017, 39(1): 9 -14 .
[6]	JIANG Ting, DAI Bing. [J]. , 2016, 38(12): 1459 -1466 .
[7]	ZHANG Yi-Bing, 吕Jian-Jun , YUAN Guang-Hui. [J]. , 2016, 38(12): 1467 -1470 .
[8]	LIU Hui, WANG Xiao-Jie, XIANG Chang-Le. [J]. , 2016, 38(12): 1483 -1487 .
[9]	CHEN Wu-Wei, DENG Shu-Chao, HUANG He, XIE You-Hao. [J]. , 2016, 38(12): 1488 -1493 .
[10]	PENG Qian-Lei, GUI Liang-Jin, FAN Zi-Jie. [J]. , 2016, 38(12): 1500 -1507 .

项目	输入	时空信息提取模块 + 局部环境信息提取模块 + 多模态特征融合模块	解码器	MSE （1.5 s）
Variant1	Bbox	LSTM + - + -	FC	2 458
Variant2	Bbox+Image	LSTM + 传统卷积 + Concat	FC	2 103
Variant3	Bbox+Image	时空注意力 + 传统卷积 + Concat	FC	1 502
Variant4	Bbox+Image	时空注意力 + 可形变卷积 + Concat	FC	1 435
Variant5	Bbox+Image	时空注意力 + 可形变卷积 + 交叉注意力	FC	1 307
本文	Bbox+Image	时空注意力 + 可形变卷积 + 交叉注意力	LSTM	1 106

[1]	Daofei Li,Hao Pan. Application of Scenario Complexity Evaluation in Trajectory Prediction and Automated Driving Decision-Making [J]. Automotive Engineering, 2024, 46(9): 1556-1563.
[2]	Hai Wang,Jianguo Li,Yingfeng Cai,Long Chen. A LiDAR-Based Dynamic Driving Scene Multi-task Segmentation Network [J]. Automotive Engineering, 2024, 46(9): 1608-1616.
[3]	Jianan Zhang,Zhaozheng Hu,Jie Meng,Huahua Hu,Jie Zuo. Distributed Simulation Platform Architecture and Application of Autonomous Driving for Vehicle-Road-Map Collaboration [J]. Automotive Engineering, 2024, 46(8): 1335-1345.
[4]	Le Tao,Hai Wang,Yingfeng Cai,Long Chen. Multi-object Detection Algorithm Based on Point Cloud for Autonomous Driving Scenarios [J]. Automotive Engineering, 2024, 46(7): 1208-1218.
[5]	Linhui Li,Yifan Fu,Ting Wang,Xuecheng Wang,Jing Lian. Trajectory Prediction Method Enhanced by Self-supervised Pretraining [J]. Automotive Engineering, 2024, 46(7): 1219-1227.
[6]	Hai Wang,Yuxuan Ding,Tong Luo,Meng Qiu,Yingfeng Cai,Long Chen. A Multi-class Multi-target Tracking Algorithm Combining Motion Speed and Appearance Features in Driving Scenarios [J]. Automotive Engineering, 2024, 46(6): 956-964.
[7]	Jing Huang,Xiangzhen Liu,Xiaoyang Deng,Ran Chen. Research on Intelligent Vehicle Trajectory Planning Based on Multimodal Trajectory Prediction [J]. Automotive Engineering, 2024, 46(6): 965-974.
[8]	Mengfan Li,Zhongxiang Feng,Weihua Zhang,Jingyu Li. Study on Driver's Visual Transfer Characteristics During the Takeover Process of Human-Computer Co-driving Mode [J]. Automotive Engineering, 2024, 46(5): 795-804.
[9]	Fuxing Yao,Chao Sun,Yungang Lan,Bing Lu,Bo Wang,Haiyang Yu. A Lane Change Decision Method for Intelligent Connected Vehicles Based on Mixture of Expert Model [J]. Automotive Engineering, 2024, 46(5): 882-892.
[10]	Ting Chikit,Yafei Wang,Yichen Zhang,Mingyu Wu,Yile Wang. Energy-Saving Planning Method for Autonomous Driving Mining Trucks Based on Composite Dynamic Sampling [J]. Automotive Engineering, 2024, 46(4): 588-595.
[11]	Yiwei Zhou,Mo Xia,Bing Zhu. Multimodal Vehicle Trajectory Prediction Methods Considering Multiple Traffic Participants in Urban Road Scenarios [J]. Automotive Engineering, 2024, 46(3): 396-406.
[12]	Xiaocong Zhao,Shiyu Fang,Zirui Li,Jian Sun. Extraction and Application of Key Utility Term for Social Driving Interaction [J]. Automotive Engineering, 2024, 46(2): 230-240.
[13]	Xiaolin Tang,Lu Gan,Guofa Li,Keqiang Li,Wenbo Chu. Large Model Alignment Technology for Autonomous Driving: A Review [J]. Automotive Engineering, 2024, 46(11): 1937-1951.
[14]	Hailun Zhang,Guangwei Wang,Qingwen Meng,Qing Xu,Jianqiang Wang,Keqiang Li. An Online Semi-supervised Hybrid Approach for Vehicle Behavior Perception at Intersections [J]. Automotive Engineering, 2024, 46(11): 1993-2004.
[15]	Zhijun Chen,Chaowei Wang,Chaozhong Wu,Chuang Qian,Huaizhu Wu,Guangjun Shen. Algorithm for Detecting Free Space in Underground Mine Tunnels for Autonomous Vehicles [J]. Automotive Engineering, 2024, 46(11): 2017-2027.

Pedestrian Trajectory Prediction Method Based on Multi-information Fusion Network

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 35

Related Articles 15

Metrics

Comments

Recommended 10

方法	Soc	Env	Soc-Env	PSI
方法	Soc	Env	Soc-Env	ADE↓ （0.5 s/1.0 s/1.5 s）	FDE↓ （0.5 s/1.0 s/1.5 s）	ARB↓ （0.5 s/1.0 s/1.5 s）	FRB↓ （0.5 s/1.0 s/1.5 s）
Bayesian-LSTM^［9］	√
FOL-X^［26］		√
PIE_traj^［13］		√
BiTraP^［4］	√
eP2P^［12］	√	√		22.67/31.07/44.90	27.76/52.03/93.14	27.12/35.03/48.51	31.59/55.08/95.97
SGNet^［15］	√
本文	√	√	√	10.00/17.13/27.67	14.66/34.56/62.21	18.08/29.21/44.98	25.27/54.62/93.99