基于多尺度骨架图和局部视觉上下文融合的驾驶员行为识别方法

doi:10.19562/j.chinasae.qcgc.2024.01.001

摘要/Abstract

摘要：

识别非驾驶行为是提高驾驶安全性的重要手段之一。目前基于骨架序列和图像的融合识别方法具有计算量大和特征融合困难的问题。针对上述问题，本文提出一种基于多尺度骨架图和局部视觉上下文融合的驾驶员行为识别模型（skeleton-image based behavior recognition network，SIBBR-Net）。SIBBR-Net通过基于多尺度图的图卷积网络和基于局部视觉及注意力机制的卷积神经网络，充分提取运动和外观特征，较好地平衡了模型表征能力和计算量间的关系。基于手部运动的特征双向引导学习策略、自适应特征融合模块和静态特征空间上的辅助损失，使运动和外观特征间互相引导更新并实现自适应融合。最终在 Drive&Act 数据集进行算法测试，SIBBR-Net在动态标签和静态标签条件下的平均正确率分别为 61.78%和 80.42%，每秒浮点运算次数为 25.92G，较最优方法降低了76.96%。

关键词: 驾驶员行为识别, 多尺度骨架图, 局部视觉上下文, 多模态数据自适应融合

Abstract:

Non-driving behavior identification is one of the important ways to improve the safety of driving. The current recognition method based on skeleton sequence and image fusion has the problems of large model calculation and the difficulty of feature fusion. To address the above problems， the skeleton-image based behavior recognition network （SIBBR-Net） is proposed in this paper， which is based on the multi-scale skeleton graph and the local visual context. SIBBR-Net fully extracts motion and appearance features through a graph convolution network based on multi-scale skeleton graphs and a convolutional neural network based on local vision and attention mechanisms， and better balances the relationship between model representation capabilities and model calculation. The feature bidirectional guided learning strategy based on hand motion， an adaptive feature fusion module and an auxiliary loss on the static feature space can guide mutual guidance and updating between motion and appearance features to achieve adaptive fusion. SIBBR-Net is finally tested on the Drive & Act dataset， and the average accuracy is 61.78% for dynamic labels and 80.42% for static labels. The Floating-point Operations per Second （FLOPS） of SIBBR-Net is 25.92G， which is 76.96% lower than that of the optimal method.

Key words: driver behavior recognition, multi-scale skeleton graph, local visual context, multi-model data adaptive fusion

胡宏宇, 黎烨宸, 张争光, 曲优, 何磊, 高镇海. 基于多尺度骨架图和局部视觉上下文融合的驾驶员行为识别方法[J]. 汽车工程, 2024, 46(1): 1-8.

Hongyu Hu, Yechen Li, Zhengguang Zhang, You Qu, Lei He, Zhenhai Gao. Driver Behavior Recognition Based on Multi-scale Skeleton Graph and Local Visual Context Method[J]. Automotive Engineering, 2024, 46(1): 1-8.

图/表 19

图 1

图2

图 3

图 4

图 5

图 6

图 7

图 8

图 9

表1

表2

图 10

图 11

表3

表4

不同数据预处理方式的实验结果"

模型分类	Accuracy/%
DMGNN （ $T' = 8$ ）	50.98
DMGNN （ $T' = 16$ ）	46.83
EfficientNet+CBAM （选取中间帧）	75.12
EfficientNet+CBAM （随机选取）	72.95

表4

表5

选取不同权重系数的实验结果"

权重系数	Accuracy/%
$α = 0.2$	59.62
$α = 0.3$	60.63
$α = 0.4$	60.08
$α = 0.5$	59.53
$α = 0.6$	61.78
$α = 0.7$	61.67
$α = 0.8$	61.54

表5

表6

表7

图 12

参考文献 18

1	MARBERGER C， MIELENZ H， NAUJOKS F， et al. Understanding and applying the concept of "driver availability" in automated driving［C］. International Conference on Applied Human Factors & Ergonomics. Springer， Cham， 2017.
2	YANG L， DONG K， DMITRUK A J， et al. A dual-cameras-based driver gaze mapping system with an application on non-driving activities monitoring［J］. IEEE Transactions on Intelligent Transportation Systems， 2019， 21（10）： 4318-4327.
3	ZHENG W， ZHANG Q Q， NI Z H， et al. Distracted driving behavior detection and identification based on improved cornernet-saccade［C］.2020 IEEE Intl Conf on Parallel & Distributed Processing with Applications， Big Data & Cloud Computing， Sustainable Computing & Communications， Social Computing & Networking （ISPA/BDCloud/SocialCom/SustainCom）. IEEE， 2020： 1150-1155.
4	JIAO S J， LIU L Y， LIU Q. A hybrid deep learning model for recognizing actions of distracted drivers［J］. Sensors， 2021， 21（21）： 7424.
5	HOLZBOCK A， TSAREGORODTSEV A， DAWOUD Y， et al. A spatio-temporal multilayer perceptron for gesture recognition［C］.2022 IEEE Intelligent Vehicles Symposium （IV）. IEEE， 2022： 1099-1106.
6	LI P， LU M， ZHANG Z， et al. A novel spatial-temporal graph for skeleton-based driver action recognition［C］.2019 IEEE Intelligent Transportation Systems Conference （ITSC）. IEEE， 2019： 3243-3248.
7	XU Q， ZHENG W， SONG Y， et al. Scene image and human skeleton-based dual-stream human action recognition［J］. Pattern Recognition Letters， 2021， 148： 136-145.
8	BRUCE X B， LIU Y， ZHANG X， et al. Mmnet： a model-based multimodal network for human action recognition in rgb-d videos［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 45（3）： 3522-3538.
9	WEYERS P， SCHIEBENER D， KUMMERT A. Action and object interaction recognition for driver activity classification［C］.2019 IEEE Intelligent Transportation Systems Conference （ITSC）， 2019： 4336-4341.
10	TAN M， LE Q. Efficientnet： rethinking model scaling for convolutional neural networks［C］.International Conference on Machine Learning. PMLR， 2019： 6105-6114.
11	YAN S， XIONG Y， LIN D. Spatial temporal graph convolutional networks for skeleton-based action recognition［C］.Proceedings of the AAAI Conference on Artificial Intelligence， 2018， 32（1）.
12	CARREIRA J， ZISSERMAN A. Quo vadis， action recognition？ a new model and the kinetics dataset［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 6299-6308.
13	LI M， CHEN S， ZHAO Y， et al. Dynamic multiscale graph neural networks for 3D skeleton based human motion prediction［C］.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 214-223.
14	TAN M， LE Q. Efficientnet： rethinking model scaling for convolutional neural networks［C］.International Conference on Machine Learning. PMLR， 2019： 6105-6114.
15	WOO S， PARK J， LEE J Y， et al. Cbam： convolutional block attention module［C］.Proceedings of the European Conference on Computer Vision （ECCV）， 2018： 3-19.
16	MARTIN M， ROITBERG A， HAURILET M， et al. Drive&act： a multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles［C］.Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 2801-2810.
17	QIU Z， YAO T， MEI T. Learning spatio-temporal representation with pseudo-3D residual networks［C］.Proceedings of the IEEE International Conference on Computer Vision， 2017： 5533-5541.
18	TRAN D， BOURDEV L， FERGUS R， et al. Learning spatiotemporal features with 3D convolutional networks［C］.Proceedings of the IEEE International Conference on Computer Vision， 2015： 4489-4497.

C0	Closing-door-outside	C17	Drinking
C1	Opening-door-outside	C18	Closing-bottle
C2	Entering-car	C19	Looking-or-moving-around
C3	Closing-door-inside	C20	Preparing-food
C4	Fastening-seat-belt	C21	Eating
C5	Using-multimedia-display	C22	Taking-off-sunglasses
C6	Sitting-still	C23	Putting-on-sunglasses
C7	Pressing-automation-button	C24	Reading-newspaper
C8	Fetching-an-object	C25	Writing
C9	Opening-laptop	C26	Talking-on-phone
C10	Working-on-laptop	C27	Reading-magazine
C11	Interacting-with-phone	C28	Taking-off-jacket
C12	Closing-laptop	C29	Opening-door-inside
C13	Placing-an-object	C30	Exiting-car
C14	Unfastening-seat-belt	C31	Opening-backpack
C15	Putting-on-jacket	C32	Putting-laptop-into-backpack
C16	Opening-bottle	C33	Taking-laptop-from-backpack

C0	Default	C9	Interact-with-jacket
C1	Interact-with-seat-belt	C10	Drinking
C2	Interact-with-multimedia-display	C11	Looking-or-moving- around
C3	Sitting-still	C12	Eating
C4	Interact-with-automation-button	C13	Interact-with-sunglasses
C5	Fetching-an-object	C14	Reading
C6	Interact-with-laptop	C15	Writing
C7	Interact-with-phone	C16	Interact-with-backpack
C8	Placing-an-object

参数名称	参数
操作系统	Window11
测试框架	Pytorch1.11.0
显卡	GeForce RTX 3070ti
优化器	SGD
批次大小	32
迭代轮次	150
初始学习率	0.001

模型分类	模型	Accuracy/%
纯骨架模型	Pose	44.36
	Two-stream	45.39
	ST-GCN	45.34
	DMGNN	50.98
纯图像模型	C3D	43.41
	P3D	45.32
	I3D	63.64
融合模型	SIBBR-Net	61.78

编号	CBAM	ISA	SU	AL	Accuracy/%
（a）	√	√	√	√	61.78
（b）		√	√	√	60.15
（c）	√			√	59.02
（d）	√		√	√	61.19
（e）	√	√		√	60.22
（f）	√	√	√		59.48