基于主观先验强化学习的汽车环岛驾驶决策

doi:10.19562/j.chinasae.qcgc.2025.08.008

Abstract

Abstract:

For the safety problems faced by intelligent vehicles in complex and highly interactive roundabout scenarios， a driving decision strategy based on Subjective Prior Deep Reinforcement Learning （SPDRL） is proposed. Firstly， a roundabout scenario model that includes the vehicle's longitudinal and lateral coupled action space， multi-scale information state space， and multi-objective reward function is constructed. Next， the Soft Actor-Critic （SAC） algorithm optimized with human preference reinforcement learning theory is used to design a driving decision strategy that considers the prior cognition of agent behavior risks. A self-learning subjective risk classifier， based on a multilayer perceptron， is applied to evaluate the prior cognition of agent behavioral risks and guide the driving decisions towards safer outcome. Finally， tests and verification are carried out using the CARLA simulation environment. The results show that the proposed strategy improves the safety performance of driving decisions by approximately 8.73% in roundabout scenarios compared to the standard SAC algorithm.

Key words: intelligent vehicles, roundabout scenarios, driving decision, subjective prior, reinforcement learning

Jian Wu,Yukang Shi,Bing Zhu,Jian Zhao,Zhicheng Chen. Intelligent Vehicle Decision for Roundabouts Based on Subjective Prior Reinforcement Learning[J].Automotive Engineering, 2025, 47(8): 1513-1521.

Figures/Tables 12

符号	参数含义	数值
$D c, D f, D l, D R E, D p$	距离阈值/m	0.1，8.0，1.0，8.0，0.5
$Θ θ 1, Θ θ 2, Θ P$	角度差阈值/rad	0.4，1.0，0.3
$Ω ω$	角速度阈值/（（°）·s^-1）	20
$V l i m$	道路限速/（m·s^-1）	8.0
$β$	风险先验目标系数	1.0
$γ$	分类器超参数	2

风险分类	行为数量	类别占比/%	归一化 $α y$
安全行为	15 987	84.65	0.032 0
低危行为	2 203	11.67	0.232 1
高危行为	694	3.68	0.735 9
数据总量	18 885	100	1.0

References 19

[1]	付新科，蔡英凤，陈龙，等.不确定性环境下的自动驾驶汽车行为决策方法［J］. 汽车工程， 2024， 46（2）： 211-221.
	FU X K， CAI Y F， CHEN L， et al. Decision-making for autonomous driving in uncertain environment［J］. Automotive Engineering， 2024， 46（2）： 211-221.
[2]	CAO Z， XU S B， PENG H， et al. Confidence-aware reinforcement learning for self-driving cars［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 23（7）： 7419-7430.
[3]	DUAN J， KONG Y， JIAO C， et al. Distributional soft actor-critic for decision-making in on-ramp merge scenarios［J］. Automotive Innovation， 2024： 1-15.
[4]	MARUYAMA N， MOURI H. A proposal for adaptive cruise control balancing followability and comfortability through reinforcement learning［J］. ROBOMECH Journal， 2022， 9（1）： 22.
[5]	HUANG Z， WU J， LV C. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 23（8）： 10239-10251.
[6]	朱冰，蒋渊德，赵健，等.基于深度强化学习的车辆跟驰控制［J］.中国公路学报，2019，32（6）：53-60.
	ZHU B， JIANG Y D， ZHAO J， et al. A car-following control algorithm based on deed reinforcement learning［J］. China Journal of Highway and Transport， 2019， 32（6）： 53-60.
[7]	魏宁.深度强化学习落地指南［M］.北京：电子工业出版社， 2021.
	WEI N. Implementation guide for deep reinforcement learning［M］. Beijing： Publishing House of Electronics Industry， 2021.
[8]	HO J， ERMON S. Generative adversarial imitation learning［J］. Advances in Neural Information Processing Systems， 2016， 29.
[9]	HUANG Z， WU J， LV C. Efficient deep reinforcement learning with imitative expert priors for autonomous driving［J］. IEEE Transactions on Neural Networks and Learning Systems， 2022， 34（10）： 7391-7403.
[10]	FINN C， LEVINE S， ABBEEL P. Guided cost learning： deep inverse optimal control via policy optimization［C］. International Conference on Machine Learning. PMLR， 2016： 49-58.
[11]	SUN R， HU S， ZHAO H， et al. Human-like highway trajectory modeling based on inverse reinforcement learning［C］. 2019 IEEE Intelligent Transportation Systems Conference （ITSC）. IEEE， 2019： 1482-1489.
[12]	CHRISTIANO P F， LEIKE J， BROWN T， et al. Deep reinforcement learning from human preferences［J］. Advances in Neural Information Processing Systems， 2017， 30.
[13]	RAFAILOV R， SHARMA A， MITCHELL E， et al. Direct preference optimization： your language model is secretly a reward model［J］. Advances in Neural Information Processing Systems， 2024， 36.
[14]	肖礼明，张发旺，陈良发，等. 依托多风格强化学习的车辆轨迹跟踪避撞控制［J］. 汽车工程， 2024， 46（6）： 945-955.
	XIAO L M， ZHANG F W， CHEN Y F， et al. Vehicle trajectory tracking and collision avoidance control based on multi-style reinforcement learning［J］. Automotive Engineering， 2024， 46（6）： 945-955.
[15]	孙冬鸣，周培栋，宋雪松，等. 引入驾驶风格系数的跟车控制优化策略［J］. 汽车与新动力，2023，6（3）：7-11.
	SUN D M， ZHOU P D， SONG X S， et al. Optimization strategy of following vehicle control with driving style coefficient［J］. Automobile and New Powertrain， 2023，6（3）：7-11.
[16]	LIN T Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［J］. IEEE Transactions on Pattern Analysis & Machine Intelligence， 2017， PP（99）：2999-3007.
[17]	CAO K， WEI C， GAIDON A， et al. Learning imbalanced datasets with label-distribution-aware margin loss［J］. Advances in Neural Information Processing Systems， 2019， 32.
[18]	DOSOVITSKIY A， ROS G， CODEVILLA F， et al. CARLA： an open urban driving simulator［C］. Conference on Robot Learning. PMLR， 2017： 1-16.
[19]	HAARNOJA T， ZHOU A， HARTIKAINEN K， et al. Soft actor-critic algorithms and applications［J］. arXiv preprint arXiv：， 2018.

评价指标	预训练/%	自学习/%
准确率	89.51	96.97
精确率	73.55	90.72
召回率	96.69	99.04
F1 score	75.79	91.91

Intelligent Vehicle Decision for Roundabouts Based on Subjective Prior Reinforcement Learning

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 19

Related Articles 15

Metrics

Comments

Recommended 0

[1]	Junjie Zhao,Yinuo Wang,Jiang Wu,Sichao Wu,Changdi Zou,Hongda Wang,ShengboEben Li,Fei Ma,Jingliang Duan. Research on Diffusion Reinforcement Learning Method for Vehicle Trajectory Tracking and Collision Avoidance of Autonomous Vehicles [J]. Automotive Engineering, 2025, 47(8): 1490-1500.
[2]	Jiayi Guan,Bin Li,Ao Zhou,Zhiguo Zhao,Qiao Lin,Guang Chen. Study on Safe Parking Path Planning Algorithm for Narrow Environment [J]. Automotive Engineering, 2025, 47(5): 797-808.
[3]	Yingjiu Pan,Yi Xi,Yansen Liu,Wenpeng Fang,Wenshan Zhang. An Energy Consumption Prediction-Based Optimization Strategy for Eco-driving of Connected Electric Buses [J]. Automotive Engineering, 2025, 47(5): 839-850.
[4]	Jiahui Liu,Yuan Zou,Wei Sun,Yihao Meng,Xiaoran Lu,Yuanyuan Li. ICV Task Offloading and Resource Allocation Based on Hybrid Deep Reinforcement Learning [J]. Automotive Engineering, 2025, 47(1): 35-43.
[5]	Xianghao Meng,Ling Niu,Junqiang Xi,Danni Chen,Chao Lü. Risk Prediction of Heterogeneous Traffic Participants Based on Spatio-Temporal Graph Neural Networks [J]. Automotive Engineering, 2024, 46(9): 1537-1545.
[6]	Daofei Li,Hao Pan. Application of Scenario Complexity Evaluation in Trajectory Prediction and Automated Driving Decision-Making [J]. Automotive Engineering, 2024, 46(9): 1556-1563.
[7]	Jialiang Zhu,Qiaobin Liu,Fan Yang,Lu Yang,Weihua Li. Two-Dimensional Collision Risk Prediction for Intelligent Vehicles Considering the Influence of Heterogeneous Vehicle Types [J]. Automotive Engineering, 2024, 46(8): 1414-1421.
[8]	Fuxing Yao,Chao Sun,Yungang Lan,Bing Lu,Bo Wang,Haiyang Yu. A Lane Change Decision Method for Intelligent Connected Vehicles Based on Mixture of Expert Model [J]. Automotive Engineering, 2024, 46(5): 882-892.
[9]	Guodong Du,Yuan Zou,Xudong Zhang,Wenjing Sun,Wei Sun. Research on Automatic Driving Motion Control Based on Double Estimator Reinforcement Learning Combined with Forward Predictive Control [J]. Automotive Engineering, 2024, 46(4): 564-576.
[10]	Wenxuan Shen,Rui Dai,Puyuan Tan,Qing Zhou. A Review of Reclined Occupant Crash Injuries and Impact Protection [J]. Automotive Engineering, 2024, 46(12): 2241-2256.
[11]	Xiaohu Geng,Yao Fu,Jie Wang,Yulong Lei,Weidong Liu,Yuhai Wang,Ke Liu. Predictive Cruise Control for Commercial Vehicles Considering Different Time Domains [J]. Automotive Engineering, 2024, 46(11): 2046-2058.
[12]	Jianping Hao,Yanzhao Su,Zhihua Zhong,Jin Huang. Service-Oriented Architecture and Service Scheduling Mechanism for Intelligent Vehicles [J]. Automotive Engineering, 2023, 45(9): 1563-1572.
[13]	Gaoshi Zhao,Long Chen,Yingfeng Cai,Yubo Lian,Hai Wang,Qingchao Liu,Chenglong Teng. Trajectory Prediction Technology Integrating Complex Network and Memory-Augmented Network [J]. Automotive Engineering, 2023, 45(9): 1608-1616.
[14]	Weiguo Liu,Zhiyu Xiang,Weiping Liu,Daoxin Qi,Zixu Wang. Research on Vehicle Control Algorithm Based on Distributed Reinforcement Learning [J]. Automotive Engineering, 2023, 45(9): 1637-1645.
[15]	Qihui Hu,Yingfeng Cai,Hai Wang,Long Chen,Zhaozhi Dong,Qingchao Liu. Heterogeneous Multi-object Trajectory Prediction Method Based on Hierarchical Graph Attention [J]. Automotive Engineering, 2023, 45(8): 1448-1456.