基于动作条件交互的高效行人过街意图预测

doi:10.19562/j.chinasae.qcgc.2024.01.004

摘要/Abstract

摘要：

城市化的进程不断加速，人车冲突问题已成为现代社会亟待解决的重大难题。复杂交通场景下，行人横穿马路行为导致交通事故频发，准确、实时地预测行人过街意图对避免人车冲突、提高驾驶安全系数和保障行人安全至关重要。本文提出基于动作条件交互的高效行人过街意图预测框架（efficient action-conditioned interaction pedestrian crossing intention anticipation framework，EAIPF）来预测行人过街意图。EAIPF引入行人动作编码模块增强多模态动作模式下的表征能力，挖掘深层骨架上下文信息。同时，引入场景对象交互模块挖掘与对象交互信息，理解交通场景中高级语义线索。最后，意图预测模块融合行人动作特征和对象交互特征，实现行人过街意图的鲁棒预测。所提出的方法在两个公共数据集JAAD和PIE上验证算法性能，准确率分别达到了89%和90%，表明本文方法可以在复杂交通场景下准确预测行人穿越意图。

关键词: 人车冲突, 行人过街意图预测, 图卷积网络, 行人动作编码, 场景理解

Abstract:

With acceleration of the urbanization process， pedestrian-vehicle conflicts have become a significant issue that modern society urgently needs to solve. In complex traffic scenarios， pedestrian crossing behavior leads to frequent traffic accidents. Accurately and timely anticipating pedestrian crossing intentions is crucial for avoiding pedestrian-vehicle conflicts， improving driving safety， and ensuring pedestrian safety. An Efficient Action-Conditioned Interaction Pedestrian Crossing Intention Anticipation Framework （EAIPF） is proposed in this paper to anticipate pedestrian crossing intention. EAIPF introduces in a pedestrian action encoding module to enhance the representation ability of multimodal action patterns and discover deep skeletal context information. At the same time， the scene object interaction module is introduced to explore interaction information with objects and understand advanced semantic clues in traffic scenes. Finally， the intention anticipation module fuses pedestrian action and object interaction features to achieve robust anticipation of pedestrian crossing intentions. The proposed method is verified on two public datasets， JAAD and PIE， achieving the accuracy of 89% and 90%， respectively， indicating that the proposed method can accurately anticipate pedestrian crossing intentions in complex traffic scenarios.

Key words: pedestrian-vehicle conflict, crossing intention anticipation, graph convolution network, pedestrian action encoding, scene understanding

杨彪, 韦智文, 倪蓉蓉, 王海, 蔡英凤, 杨长春. 基于动作条件交互的高效行人过街意图预测[J]. 汽车工程, 2024, 46(1): 29-38.

Biao Yang, Zhiwen Wei, Rongrong Ni, Hai Wang, Yingfeng Cai, Changchun Yang. Efficient Pedestrian Crossing Intention Anticipation Based on Action-Conditioned Interaction[J]. Automotive Engineering, 2024, 46(1): 29-38.

图/表 11

图1

图2

图3

图4

图5

表1

表2

表3

表4

图6

图7

参考文献 34

1	CHEN B， SUN D， ZHOU J， et al. A future intelligent traffic system with mixed autonomous vehicles and human-driven vehicles［J］. Information Sciences， 2020， 529： 59-72.
2	连静，王欣然，李琳辉，等. 基于人-车交互的行人轨迹预测［J］. 中国公路学报， 2021， 34（5）： 215.
	LIAN J， WANG X R， LI L H， et al. Pedestrian trajectory prediction based on human-vehicle interaction［J］. China Journal of Highway and Transport， 2021， 34（5）： 215.
3	LI J， SU W， WANG Z. Simple pose： rethinking and improving a bottom-up approach for multi-person pose estimation［C］. Proceedings of the AAAI Conference on Artificial Intelligence， 2020： 11354-11361.
4	LUO Z， WANG Z， HUANG Y， et al. Rethinking the heatmap regression for bottom-up human pose estimation［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2021： 13264-13273.
5	LIU J， ROJAS J， LI Y， et al. A graph attention spatio-temporal convolutional network for 3D human pose estimation in video［C］. 2021 IEEE International Conference on Robotics and Automation （ICRA）. IEEE， 2021： 3374-3380.
6	LI Y， YANG S， LIU P， et al. SimCC： a simple coordinate classification perspective for human pose estimation［C］. Computer Vision-ECCV 2022： 17th European Conference， Tel Aviv， Israel， October 23-27， 2022， Proceedings， Part VI. Cham： Springer Nature Switzerland， 2022： 89-106.
7	SUN K， XIAO B， LIU D， et al. Deep high-resolution representation learning for human pose estimation［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2019： 5693-5703.
8	YU B， YIN H， ZHU Z. Spatio-temporal graph convolutional networks： a deep learning framework for traffic forecasting［J］. arXiv preprint arXiv：， 2017.
9	SHI L， ZHANG Y， CHENG J， et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2019： 12026-12035.
10	YE F， PU S， ZHONG Q， et al. Dynamic GCN： context-enriched topology learning for skeleton-based action recognition［C］. Proceedings of the 28th ACM International Conference on Multimedia， 2020： 55-63.
11	杨彪，范福成，杨吉成，等. 基于动作预测与环境条件的行人过街意图识别［J］. 汽车工程， 2021， 43（7）： 1066-1076.
	YANG Biao， FAN Fucheng， YANG Jicheng， et al. Recognizing pedestrians’ crossing intentions based on action prediction and environment context ［J］. Automotive Engineering， 2021， 43（7）： 1066-1076.
12	CADENA P R G， QIAN Y， WANG C， et al. Pedestrian Graph+： a fast pedestrian crossing prediction model based on graph convolutional networks［J］. IEEE Transactions on Intelligent Transportation Systems， 2022.
13	YANG D， ZHANG H， YURTSEVER E， et al. Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention［J］. IEEE Transactions on Intelligent Vehicles， 2022， 7（2）： 221-230.
14	NI R， YANG B， WEI Z， et al. Pedestrians crossing intention anticipation based on dual‐channel action recognition and hierarchical environmental context［J］. IET Intelligent Transport Systems， 2023， 17（2）： 255-269.
15	RASOULI A， KOTSERUBA I， TSOTSOS J K. Are they going to cross？ a benchmark dataset and baseline for pedestrian crosswalk behavior［C］. Proceedings of the IEEE International Conference on Computer Vision Workshops. 2017： 206-213.
16	RASOULI A， KOTSERUBA I， TSOTSOS J K. Pedestrian action anticipation using contextual feature fusion in stacked RNNs［J］. arXiv preprint arXiv：， 2020.
17	SHI J， LIU C， ISHI C T， et al. Skeleton-based emotion recognition based on two-stream self-attention enhanced spatial-temporal graph convolutional network［J］. Sensors， 2020， 21（1）： 205.
18	STERGIOU A， POPPE R. Adapool： exponential adaptive pooling for information-retaining downsampling［J］. IEEE Transactions on Image Processing， 2022， 32： 251-266.
19	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2016： 770-778.
20	CHEN Y， ZHANG Z， YUAN C， et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 13359-13368.
21	GIRSHICK R. Fast R-CNN［C］. Proceedings of the IEEE International Conference on Computer Vision， 2015： 1440-1448.
22	LIN T Y， DOLLÁR P， GIRSHICK R， et al. Feature pyramid networks for object detection［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 2117-2125.
23	RASOULI A， KOTSERUBA I， KUNIC T， et al. PIE： a large-scale dataset and models for pedestrian intention estimation and trajectory prediction［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 6262-6271.
24	ZHOU Z， FAN X， SHI P， et al. R-MSFM： recurrent multi-scale feature modulation for monocular depth estimating［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 12777-12786.
25	BOUAZIZI A， HOLZBOCK A， KRESSEL U， et al. Motionmixer： MLP-based 3D human body pose forecasting［J］. arXiv preprint arXiv：， 2022.
26	KOTSERUBA I， RASOULI A， TSOTSOS J K. Benchmark for evaluating pedestrian action prediction［C］. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision， 2021： 1258-1268.
27	SHI X， CHEN Z， WANG H， et al. Convolutional LSTM network： a machine learning approach for precipitation nowcasting［J］. Advances in Neural Information Processing Systems， 2015， 28.
28	KOTSERUBA I， RASOULI A， TSOTSOS J K. Do they want to cross？ understanding pedestrian intention for behavior prediction［C］. 2020 IEEE Intelligent Vehicles Symposium （IV）. IEEE， 2020： 1688-1693.
29	BHATTACHARYYA A， FRITZ M， SCHIELE B. Long-term on-board prediction of people in traffic scenes under uncertainty［C］.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2018： 4194-4202.
30	NG J Y， HAUSKNECHT M， VIJAYANARASIMHAN S， et al. Beyond short snippets： deep networks for video classification［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2015： 4694-4702.
31	LIN R， LIU S， YANG M， et al. Hierarchical recurrent neural network for document modeling［C］. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing， 2015： 899-907.
32	TRAN D， BOURDEV L， FERGUS R， et al. Learning spatiotemporal features with 3D convolutional networks［C］. Proceedings of the IEEE International Conference on Computer Vision， 2015： 4489-4497.
33	LI J， WANG C， ZHU H， et al. CrowdPose： efficient crowded scenes pose estimation and a new benchmark［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2019： 10863-10872.
34	CARREIRA J， ZISSERMAN A. Quo vadis， action recognition？ a new model and the kinetics dataset［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 6299-6308.

2D 动作编码	3D 动作编码	预测模块	场景对象交互模块	Acc	AUC	F1	Pre	Rec
√				84.15	72.66	55.04	55.22	54.87
	√			84.04	74.59	57.07	54.43	59.98
√		√		86.27	82.88	66.43	58.03	77.66
	√	√		86.01	79.95	63.85	54.27	77.57
√			√	85.47	76.63	60.27	57.75	63.02
	√		√	84.61	77.23	59.96	55.01	65.88
√		√	√	88.96	84.10	78.81	85.66	72.98
	√	√	√	88.01	79.23	65.70	65.67	65.73

2D 动作编码	3D 动作编码	预测模块	场景对象交互模块	Acc	AUC	F1	Pre	Rec
√				63.76	55.48	75.12	86.76	66.18
	√			63.85	55.72	75.62	86.79	63.85
√		√		67.86	67.89	72.51	77.98	67.76
	√	√		68.05	67.74	72.93	77.49	68.97
√			√	64.15	61.77	71.31	71.41	71.21
	√		√	65.69	63.53	72.44	72.81	72.08
√		√	√	68.64	67.72	74.01	76.86	71.37
	√	√	√	69.10	65.03	76.69	72.64	81.21

2D 动作编码	3D 动作编码	预测模块	场景对象交互模块	Acc	AUC	F1	Pre	Rec
√				78.57	66.69	50.92	71.81	39.45
	√			79.80	69.60	56.34	72.07	46.24
√		√		87.75	81.26	75.31	86.98	66.41
	√	√		88.86	82.92	77.79	88.64	69.31
√			√	82.62	74.06	63.82	77.05	54.47
	√		√	84.10	78.98	70.43	73.91	67.26
√		√	√	88.93	82.59	77.58	90.16	68.08
	√	√	√	90.33	86.57	81.95	86.35	77.97

模型名称	模型变体	PIE					JAAD_beh					JAAD_all
模型名称	模型变体	Acc	AUC	F1	Pre	Rec	Acc	AUC	F1	Pre	Rec	Acc	AUC	F1	Pre	Rec
ATGC^［14］	VGG16	0.71	0.60	0.41	0.49	0.36	0.59	0.52	0.71	0.63	0.82	0.82	0.75	0.55	0.49	0.63
ATGC^［14］	ResNet50	0.70	0.59	0.38	0.47	0.32	0.46	0.45	0.54	0.58	0.51	0.81	0.72	0.52	0.47	0.56
ConvLSTM^［27］	VGG16	0.58	0.55	0.39	0.32	0.49	0.53	0.49	0.64	0.64	0.64	0.63	0.57	0.32	0.24	0.48
ConvLSTM^［27］	ResNet50	0.54	0.46	0.26	0.23	0.29	0.59	0.55	0.69	0.68	0.70	0.63	0.58	0.33	0.25	0.49
SingleRNN^［28］	GRU	0.83	0.77	0.67	0.70	0.64	0.58	0.54	0.67	0.67	0.68	0.65	0.59	0.34	0.26	0.49
SingleRNN^［28］	LSTM	0.81	0.75	0.64	0.67	0.61	0.51	0.48	0.61	0.63	0.59	0.78	0.75	0.54	0.44	0.70
MultiRNN^［29］	GRU	0.83	0.80	0.71	0.69	0.73	0.61	0.50	0.74	0.64	0.86	0.79	0.79	0.58	0.45	0.79
StackedRNN^［30］	GRU	0.82	0.78	0.67	0.67	0.68	0.60	0.60	0.66	0.73	0.61	0.79	0.79	0.58	0.46	0.79
HierarchicalRNN^［31］	GRU	0.82	0.77	0.67	0.68	0.66	0.53	0.50	0.63	0.64	0.61	0.80	0.79	0.59	0.47	0.79
SFRNN^［15］	GRU	0.82	0.79	0.69	0.67	0.70	0.51	0.45	0.63	0.61	0.64	0.84	0.84	0.65	0.54	0.84
C3D^［32］	3DConv	0.77	0.67	0.52	0.63	0.44	0.61	0.51	0.75	0.63	0.91	0.84	0.81	0.65	0.57	0.75
I3D^［33］	3DConv	0.80	0.73	0.62	0.67	0.58	0.62	0.56	0.73	0.68	0.79	0.81	0.74	0.63	0.66	0.61
I3D^［33］	Optical	0.81	0.83	0.72	0.60	0.90	0.62	0.51	0.75	0.65	0.88	0.84	0.80	0.63	0.55	0.73
PCPA^［34］	3DConv	0.86	0.91	0.78	0.69	0.89	0.50	0.47	0.59	0.61	0.58	0.70	0.85	0.51	0.36	0.87
Ped Graph+^［11］	GCN	0.89	0.90	0.81	0.83	0.79	0.70	0.70	0.76	0.77	0.75	0.86	0.88	0.65	0.58	0.75
Ours	GCN	0.90	0.87	0.82	0.86	0.78	0.69	0.65	0.77	0.73	0.81	0.89	0.84	0.79	0.86	0.73

[1]	崔格格,吕超,李景行,张哲雨,熊光明,龚建伟. 数据驱动的智能车个性化场景风险图构建[J]. 汽车工程, 2023, 45(2): 231-242.
[2]	吕彦直,魏超,何元浩. 基于GCN和CIL的端到端自动驾驶换道方法[J]. 汽车工程, 2023, 45(12): 2310-2317.
[3]	白中浩, 王韫宇, 张林伟. 基于图卷积网络的多信息融合驾驶员分心行为检测^*[J]. 汽车工程, 2020, 42(8): 1027-1033.