基于MobileViT模型和光流融合的驾驶人行为识别

doi:10.19562/j.chinasae.qcgc.2025.08.005

Abstract

Abstract:

Based on the MobileViT algorithm， a novel driver behavior recognition model of Mse-MViT model is proposed in this paper， which integrates Convolutional Neural Networks （CNNs） with Transformers. The model uses the optical flow algorithm for recursive image processing， enabling the extraction of key frame sequences from the initial frame to the apex frame of a video clip to effectively capture driver motion information. A self-constructed Driver-vior dataset is introduced. Through multi-scale feature fusion， an SE attention mechanism， and dual-branch architecture， the model achieves comprehensive integration of motion cues with global and local image features. The experimental results show that the Mse-MViT model achieves a driver behavior recognition accuracy of 95.83%， exhibiting superior performance and robustness. Furthermore， comparative experiments conducted on the State Farm dataset show a 2.5% improvement in accuracy， validating the generalization capability and effectiveness of the proposed method.

Key words: driver behavior recognition, optical flow algorithm, MobileViT, multi-scale feature fusion

Huizhi Xu,Jianzhao Zhang,Xiancai Jiang,Chengju Song. Driver Behavior Recognition Method via MobileViT Model and Optical Flow Fusion[J].Automotive Engineering, 2025, 47(8): 1479-1489.

Figures/Tables 24

References 27

[1]	QU Y， HU H， LIU J， et al. Driver state monitoring technology for conditionally automated vehicles： review and future prospects［J］. IEEE Transactions on Instrumentation and Measurement， 2023.
[2]	刘中姐，赵治国，于勤.基于动态风险评估的车辆辅助驾驶行为决策［J］.汽车工程，2024，46（11）：2005-2016.
	LIU Z J，ZHAO Z G，YU Q.Vehicle assisted driving behavior decision-making based on dynamic risk assessment［J］. Automotive Engineering， 2024，46（11）：2005-2016.
[3]	ZHAO X， LI Z， ZHAO C， et al. Distraction-level recognition based on stacking ensemble learning for IVIS secondary tasks［J］. Expert Systems with Applications， 2024， 244： 122849.
[4]	唐小林，甘露，李国法，等.面向自动驾驶的大模型对齐技术：综述［J］.汽车工程，2024，46（11）：1937-1951.
	TANG X L， GAN L， LI G F， et al. Large model alignment technology for autonomous driving： a review［J］. Automotive Engineering，2024，46（11）：1937-1951.
[5]	孙剑，张赫，赵晓聪，等.面向自动驾驶测试的交互场景策略建模与仿真［J］.汽车工程，2024，46（11）：1962-1972.
	SUN J，ZHANG H，ZHAO X C， et al. Interactive scenario strategy modeling and simulation for autonomous driving testing［J］. Automotive Engineering， 2024，46（11）：1962-1972.
[6]	宗长富，代昌华，张东. 智能汽车的人机共驾技术研究现状和发展趋势［J］. 中国公路学报， 2021， 34（6）： 214-237.
	ZONG C F， DAI C H， ZHANG D. Research status and development trend of human-machine co-driving technology for intelligent vehicles［J］. China Journal of Highway and Transport， 2021， 34（6）： 214-237.
[7]	LIN N， ZONG C， TOMIZUKA M， et al. An overview on study of identification of driver behavior characteristics for automotive control［J］. Mathematical Problems in Engineering， 2014， 2014（1）： 569109.
[8]	OSIPOV A， PLESHAKOVA E， GATAULLIN S， et al. Deep learning method for recognition and classification of images from video recorders in difficult weather conditions［J］. Sustainability， 2022， 14（4）： 2420.
[9]	YAN C， COENEN F， ZHANG B. Driving posture recognition by joint application of motion history image and pyramid histogram of oriented gradients［J］. International Journal of Vehicular Technology， 2014， 2014（1）： 719413.
[10]	AL-DOORI S K S， TASPINAR Y S， KOKLU M. Distracted driving detection with machine learning methods by cnn based feature extraction［J］. International Journal of Applied Mathematics Electronics and Computers， 2021， 9（4）： 116-121.
[11]	WANG Y， DING X， YUAN G， et al. Dual-cameras-based driver’s eye gaze tracking system with non-linear gaze point refinement［J］. Sensors， 2022， 22（6）： 2326.
[12]	CHENG W， WANG X， MAO B. A multi-feature fusion algorithm for driver fatigue detection based on a lightweight convolutional neural network［J］. The Visual Computer， 2024， 40（4）： 2419-2441.
[13]	胡宏宇，黎烨宸，张争光，等. 基于多尺度骨架图和局部视觉上下文融合的驾驶员行为识别方法［J］. 汽车工程， 2024， 46（1）： 1-8.
	HU H G， LI Y C， ZHANG Z G， et al. Driver behavior recognition method based on multi-scale skeleton graph and local visual context fusion［J］. Automotive Engineering， 2024， 46（1）： 1-8.
[14]	HU Y， LU M， LU X. Feature refinement for image-based driver action recognition via multi-scale attention convolutional neural network［J］. Signal Processing： Image Communication， 2020， 81： 115697.
[15]	XIAO W， LIU H， MA Z， et al. Attention-based deep neural network for driver behavior recognition［J］. Future Generation Computer Systems， 2022， 132： 152-161.
[16]	VASWANI A. Attention is all you need［J］. Advances in Neural Information Processing Systems， 2017.
[17]	赵霞，李朝，付锐，等. 基于深度卷积-Tokens 降维优化视觉 Transformer 的分心驾驶行为实时检测［J］. 汽车工程， 2023， 45（6）： 974-988.
	ZHAO X， LI C， FU R， et al. Real-time detection of distracted driving behavior based on deep convolution-tokens dimension reduction and optimization of visual transformer［J］. Automotive Engineering， 2023， 45（6）： 974-988.
[18]	MEHTA S， RASTEGARI M. Mobilevit： light-weight， general-purpose， and mobile-friendly vision transformer［J］. arXiv preprint arXiv：， 2021.
[19]	MOHAMMED Q， GENG X， WANG J， et al. Driver distraction detection using semi-supervised lightweight vision transformer［J］.Engineering Applications of Artificial Intelligence， 2024， 129： 107618.
[20]	ABOUELNAGA Y， ERAQI H M， MOUSTAFA M N. Real-time distracted driver posture classification［J］. arXiv 2017. arXiv preprint arXiv：.
[21]	StateFarm distracted driver detection ［DB/OL］.https：//www.kaggle.com/c/state-farm-distracted-driver-detection 2015.
[22]	LI P， YANG Y， GROSU R， et al. Driver distraction detection using octave-like convolutional neural network［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 23（7）： 8823-8833.
[23]	刘英杰，杨风暴，胡鹏.基于Cascade R-CNN的并行特征金字塔网络无人机航拍图像目标检测算法［J］.激光与光电子学进展，2020，57（20）：302-309.
	LIU Y J， YANG F B， HU P. Parallel feature pyramid network target detection algorithm for UAV aerial images based on Cascade R-CNN［J］. Laser & Optoelectronics Progress， 2020， 57（20）： 302-309.
[24]	MEHTA S， RASTEGARI M. Mobilevit： light-weight， general-purpose， and mobile-friendly vision transformer［J］. arXiv 2021. arXiv preprint arXiv：.
[25]	HAN K， WANG Y， CHEN H， et al. A survey on vision transformer［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 45（1）： 87-110.
[26]	SZEGEDY C， VANHOUCKE V， IOFFE S， et al. Rethinking the inception architecture for computer vision［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2016： 2818-2826.
[27]	HUANG G， LIU Z， VAN DER MAATEN L， et al. Densely connected convolutional networks［C］. Proceedings of the IEEE Con-

序号	驾驶类型	训练集	测试集	图像总数
C0	直行	3 376	372	3 748
C1	左转	4 536	500	5 036
C2	右转	3 364	372	3 736
C3	换挡	4 104	456	4 560
C4	调节雨刷器	1 836	200	2 036
C5	喝水	1 804	200	2 004
C6	吸烟	1 520	168	1 688
C7	打电话	1 016	112	1 128
C8	向后看	3 468	384	3 852
C9	操作导航	4 040	448	4 488
C10	加速	3 204	356	3 560
C11	减速	2 898	322	3 220
C12	降挡	2 819	313	3 132
C13	起步	2 646	294	2 940
总数		40 631	4 497	45 128

环境项	版本
CPU	Intel（R）i9-10900K CPU@370 GHz
GPU	NVIDIA GeForce RTX 3090
内存	64 GB
操作系统	64位Windows10
编程语言	Python 3.8
CUDA	11.3
CUDNN	8.1
深度学习框架	Pytorch 1.11.0

Folds	Accuracy/%	Recall/%	Precision/%	F1-Score/%
Fold-1	91.32	91.45	90.58	90.01
Fold-2	94.65	94.51	94.48	94.49
Fold-3	96.23	96.32	96.25	96.28
Fold-4	93.15	93.14	93.50	93.32
Fold-5	92.23	92.52	92.79	92.65
Avg.	93.51	93.18	93.12	93.15

Class	Avg.P/%	Avg.R/%	Avg.F1/%	Acc/%
accelerator	93.52	96.24	96.38	96.32
brake	92.58	95.48	95.53	95.40
central control	95.12	95.36	95.24	95.48
accelerator	94.58	95.88	95.73	95.15
in gear	92.5	95.21	95.35	95.24
clutch and brake	95.48	95.89	95.68	94.18
look back	95.3	95.24	95.27	96.11
phone	96.24	96.12	96.18	96.35
smoke	97.84	97.78	97.80	97.48
straight	97.46	97.42	97.43	97.12
turn left	92.18	92.21	92.19	92.02
turn right	92.07	92.01	92.04	92.12
water	91.35	91.25	91.30	91.50
wiper	96.58	96.69	96.63	96.45

Model	RAFT	Avg.P/%	Avg.R/%	Avg.F1/%	Acc/%
MobileViT	YES	87.43	87.45	87.60	87.76
MobileViT	NO	85.45	85.18	85.24	85.31
MobileViT-SE	YES	91.12	90.12	90.19	90.27
MobileViT-SE	NO	88.26	88.76	88.54	88.35
MobileViT-MECS	YES	92.34	93.11	93.21	93.32
MobileViT-MECS	NO	91.54	91.86	91.52	91.18
MobileViT- SE-MECS	YES	95.27	95.01	95.42	95.83
MobileViT- SE-MECS	NO	93.48	93.32	93.23	93.15

Driver Behavior Recognition Method via MobileViT Model and Optical Flow Fusion

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 24

References 27

Related Articles 1

Metrics

Comments

Recommended 0

Method	Acc/%	EER	Method	Acc/%	EER
SoftMax+ReLU	89.24	5.67	SoftMax+Lcross	92.46	6.58
SiLU+ReLU	91.65	3.15	SiLU+Lcross	94.75	0.39

NO.	Layer2	Layer3	Layer4	Accuracy/%
Backbone	×	×	×	92.24
1	√	×	×	90.68
2	×	√	×	92.25
3	×	×	√	91.45
4	√	√	×	94.58
5	×	√	√	94.01
6	√	√	√	93.89

Model	Avg.Acc/%
Model	State Farm	AUC
InceptionV3^［26］	92.31	93.19
DenseNet^［［27］	94.26	95.15
ResNet-50^［28］	92.59	93.26
ResNet-101^［28］	93.10	95.23
MobileNetV2^［29］	91.65	94.74
EfficientNet^［30］	92.58	94.37
Vision-Transformer^［31］	90.90	92.54
Swin-Transformer^［32］	94.11	95.48
ConvNextv2^［33］	94.89	95.74
MSDAN^［34］	96.82	94.03
Ours	95.75	96.87

Method	Epoch	Batch Size	Accuracy/%
RNN	100	64	92.68
LSTM	100	64	90.24
GRU	100	64	95.25