基于场景动力学和强化学习的自动驾驶边缘测试场景生成方法

doi:10.19562/j.chinasae.qcgc.2022.07.004

Abstract

Abstract:

For solving the problem of low-probability and high-risk edge test scenes， an edge scene reinforcement generation method based on scene dynamics and reinforcement learning is proposed to fulfill the automatic generation of edge scenes and simulate the features of confrontation and game behavior between vehicles in the real world. Firstly， the scene models dynamically changing with time is described by a set of differential equations as a scene dynamic system. Then， neural network is used as a general function approximator， to construct the scene black-box controller for fulfilling the optimization solving of edge scene controller based on reinforcement learning. Finally， with the cut-in scene for overtaking as an example， a verification simulation is performed with Matlab/Simulink software. The results show that the edge scene models generated by reinforcement learning exhibit an excellent performance in terms of scene interactive gaming， scene coverage and repeatable test.

Key words: intelligent driving test, edge scenes, scene dynamics, scene controller, reinforcement learning

Jiangkun Li,Weiwen Deng,Bingtao Ren,Wenqi Wang,Juan Ding. Automatic Driving Edge Scene Generation Method Based on Scene Dynamics and Reinforcement Learning[J].Automotive Engineering, 2022, 44(7): 976-986.

Figures/Tables 17

跟驰场景	场景介绍：主车跟驰交通车行驶，交通车能够通过突然加减速干扰主车。交通车编号1，主车编号2。

	状态量： $x 1 = v 1, x 2 = a 1, x 3 = d r e f, x 4 = v 2$ 输入量： $u 1 = f a$
不规则切入场景	场景介绍：主车沿路行驶，交通车按照规定路径从相邻车道切入实施干扰。交通车编号1，主车编号2。

	状态量： $x 1 = v 1, x 2 = a 1, x 3 = d 1, d 2, d 3, x 4 = Δ t 1, Δ t 2, Δ t 3$ 输入量： $u 1 = f a$
多车换道场景	场景介绍：主车换道过程中，前面交通车会加速，后面交通车会加速实施干扰。交通车编号1、2，主车编号3

	状态量： $x 1 = v 1, x 2 = a 1, x 3 = d r e f 1, x 4 = v 2, x 5 = a 2, x 6 = d r e f 2, x 7 = d r e s t, x 8 ? = v 3, x 9 = w 3$ 输入量： $u 1 = f a 1, u 2 = f a 2$

References 27

1	邓伟文，李江坤，任秉韬，等. 面向自动驾驶的仿真场景自动生成方法综述［J］. 中国公路学报， 2021： 1-21.
	DENG W W, LI J K, REN B T, et al. A survey on automatic generation methods on simulation scenario for autonomous driving [J]. China Journal of Highway and Transport, 2021: 1-21.
2	朱冰, 张培兴, 赵健, 等. 基于场景的自动驾驶汽车虚拟测试研究进展[J]. 中国公路学报, 2019,32(6): 1-19.
	ZHU B, ZHANG P X, ZHAO J, et al. Review of scenario-based virtual validation methods for automated vehicles[J]. China Journal of Highway and Transport, 2019, 32(6): 1-19.
3	田鹏, 左大义, 高艳春, 等. 面向实际场景的人工智能脆弱性分析[J]. 计算机技术与发展, 2021,31(11): 129-135.
	TIAN P,ZUO D Y,GAO Y C, et al. Vulnerability analysis of artificial intelligence in real world[J]. Computer Technology and Development,2021,31(11):129-135.
4	徐向阳, 胡文浩, 董红磊, 等. 自动驾驶汽车测试场景构建关键技术综述[J]. 汽车工程, 2021,43(4): 610-619.
	XU X Y, HU W H, DONG H L, et al. Review of key technologies for autonomous vehicle test scenario construction[J]. Automotive Engineering, 2021, 43(4): 610-619.
5	吴旭阳,蒙昊蓝,邢星宇,等. 自动驾驶系统边缘测试用例生成[J].同济大学学报（自然科学版）,2018,46(12):111-115.
	WU X Y, MENG H L, XING X Y, et al. Autonomous driving system edge test case generation[J]. Journal of Tongji University (Natural Science) ,2018,46(12):111-115.
6	白雪松,邓伟文,任秉韬,等.一种自动驾驶仿真场景要素的提取方法[J].汽车工程, 2021,43(7):1030-1036.
	BAI X S, DENG W W, REN B T, et al. An extraction method of scenario elements for autonomous driving simulation[J]. Automotive Engineering, 2021, 43(7): 1030-1036.
7	ZHAO D, LAM H, PENG H, et al. Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(3): 595-607.
8	SCHWALL M, DANIEL T, VICTOR T, et al. Waymo public road safety performance data[J]. ArXiv, 2020.
9	陈君毅, 冯天悦, 刘力豪, 等. 面向决策规划系统测试的具体场景自动化生成方法[J]. 汽车技术, 2020(10): 45-50.
	CHEN J Y, FENG T Y, LIU L H, et al. An automatic generation method of concrete scenario for decision making and planning system test[J]. Automobile Technology,2020(10): 45-50.
10	任秉韬, 邓伟文, 白雪松, 等. 面向智能驾驶测试的仿真场景构建技术综述[J]. 中国图象图形学报, 2021,26(1): 1-12.
	REN B T, DENG W W, BAI X S, et al. Technologies of virtual scenario construction for intelligent driving testing[J]. Journal of Image and Graphics,2021, 26(1): 1-12.
11	DUAN J, GAO F, HE Y. Test scenario generation and optimization technology for intelligent driving systems[J]. IEEE Intelligent Transportation Systems Magazine, 2020.
12	FENG S, FENG Y, YU C, et al. Testing scenario library generation for connected and automated vehicles, part I: methodology[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3): 1573-1582.
13	ZHU B, ZHANG P, ZHAO J, et al. Hazardous scenario enhanced generation for automated vehicle testing based on optimization searching method[J]. IEEE Transactions on Intelligent Transportation Systems, 2021: 1-11.
14	周博林, 张宗实, 陈蔯. ASAM OpenX与基于场景的自动驾驶仿真测试[J]. 标准科学, 2021(S1): 110-122.
	ZHOU B L, ZHANG Z S, CHEN C. ASAM OpenX and scenario based simulation test for autonomous driving[J]. Standard Science, 2021(S1): 110-122.
15	余唯之, 陈桢, 苏奕敏. OpenSCENARIO研究综述[J]. 中国汽车, 2021(6): 17-23.
	YU W Z, CHEN Z, SUN Y M. OpenSCENARIO research review[J]. Chinese Automobile, 2021(6): 17-23.
16	FENG S, FENG Y, SUN H, et al. Testing scenario library generation for connected and automated vehicles: an adaptive framework[J]. IEEE Transactions on Intelligent Transportation Systems, 2020: 1-10.
17	KLÜCK F, ZIMMERMANN M, WOTAWA F, et al. Genetic algorithm-based test parameter optimization for ADAS system testing[C].2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS). IEEE, 2019: 418-425.
18	BEGLEROVIC H, STOLZ M, HORN M. Testing of autonomous vehicles using surrogate models and stochastic optimization[C].2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2017.
19	9］ FENG S, FENG Y, SUN H, et al. Testing scenario library generation for connected and automated vehicles, part Ⅱ: case studies[J]. IEEE Transactions on Intelligent Transportation Systems, 2020: 1-13.
20	FENG S, YAN X, SUN H, et al. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment[J]. Nature Communications, 2021, 12(1).
21	WENG B, CAPITO RUIZ L J, OZGUNER U, et al. Towards guaranteed safety assurance of automated driving systems with scenario sampling: an invariant set perspective[J]. IEEE Transactions on Intelligent Vehicles, 2021.
22	邝先验. 城市混合交通流微观仿真建模研究[D].广州：华南理工大学，2014.
	KUANG X Y. Research on microscopic simulation modeling for urban mixed traffic flow[D]. Guangzhou:South China University of Technology，2014.
23	LILLICRAP T, HUNT J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. ICLR, 2015.
24	杨宁康, 韩立金, 刘辉, 等. 基于效率优化的混合动力车辆强化学习能量管理策略研究[J]. 汽车工程, 2021,43(7): 1046-1056.
	YANG N K, HAN L J, LIU H, et al. Research on efficiency optimization based energy management strategy for a hybrid electric vehicle with reinforcement learning[J]. Automotive Engineering, 2021, 43(7): 1046-1056.
25	宋晓琳, 盛鑫, 曹昊天, 等. 基于模仿学习和强化学习的智能车辆换道行为决策[J]. 汽车工程, 2021,43(1): 59-67.
	SONG X L, SHENG X, CAO H T, et al. Lane‑change behavior decision‑making of intelligent vehicle based on imitation learning and reinforcement learning[J]. Automotive Engineering, 2021, 43(1): 59-67.
26	郭景华, 李文昌, 罗禹贡, 等. 基于深度强化学习的驾驶员跟车模型研究[J]. 汽车工程, 2021,43(4): 571-579.
	GUO J H, LI W C, LUO Y G, et al. Driver car⁃ following model based on deep reinforcement learning[J]. Automotive Engineering, 2021, 43(4): 571-579.
27	TREIBER M, HENNECKE A, HELBING D. Congested traffic states in empirical observations and microscopic simulations[J]. Phys. Rev. E, 2000,62(2):1805-1824.

[1]	ZHOU Wei, ZHANG Cheng-Ning, LI Jun-Qiu. [J]. , 2016, 38(12): 1407 -1414 .
[2]	ZONG Yi-Qi, GU Zheng-Qi, LUO Ze-Min, JIANG Cai-Mao, ZHANG Qi-Dong, YANG Zhen-Dong. [J]. , 2016, 38(12): 1427 -1433 .
[3]	ZHANG Ying-Chao, ZHAN Da-Peng, ZHAO Jing, ZHANG Zhe, LI Jie. [J]. , 2016, 38(12): 1434 -1439 .
[4]	TANG You-Ming, YAN Ling-Bo, LUO Qian, CAO Li-Bo. [J]. , 2016, 38(12): 1452 -1458 .
[5]	ZHANG Lei, LU Jian-Wei, JIANG Jun-Zhao, YAN Pei-Lei, LI Lei. [J]. , 2016, 38(12): 1477 -1482 .
[6]	ZHAO Liang, ZHANG Qiang, GUO Kong-Hui. [J]. , 2017, 39(1): 86 -91 .
[7]	SHEN Deng-Feng, WANG Chen, YU Hai-Sheng, ZHANG Tong, YI Xian-Ke. [J]. , 2017, 39(1): 15 -22 .
[8]	ZOU Tie-Fang, HU Lin, LI Ping-Fan, YI Liang. [J]. , 2017, 39(1): 41 -46 .
[9]	ZHOU Kai, ZANG Jing-Lun. [J]. , 2017, 39(2): 127 -132 .
[10]	ZUO Wen-Jie, CHEN Ji-Shun, LI Yi-Wen, LIU Nian. [J]. , 2017, 39(2): 145 -149 .

网络	层	维度	激活函数
Actor网络	全连接层1	（5，100）	RuLU
	全连接层2	（100，100）	RuLU
	全连接层3	（100，100）	RuLU
	全连接层4	（100，1）	Tanh
Critic网络	全连接层1	（12，100）	RuLU
	全连接层2	（100，100）	RuLU
	全连接层3	（2，100）	RuLU
	全连接层4	（100，100）	RuLU
	全连接层5	（100，1）	RuLU

变量	描述	数值
maxepisodes	最大训练次数	5 000
MiniBatchSize	采样样本大小	64
Ts	采样步长	0.1 s
ActorLearnRate	Actor网络学习率	0.000 1
CriticLearnRate	Critic网络学习率	0.001
DiscountFactor	奖励衰减系数	0.99
Smoothing factor	平滑因子	0.001

[1]	Weiguo Liu,Zhiyu Xiang,Weiping Liu,Daoxin Qi,Zixu Wang. Research on Vehicle Control Algorithm Based on Distributed Reinforcement Learning [J]. Automotive Engineering, 2023, 45(9): 1637-1645.
[2]	Lisheng Jin,Guangde Han,Xianyi Xie,Baicang Guo,Guofeng Liu,Wentao Zhu. Review of Autonomous Driving Decision-Making Research Based on Reinforcement Learning [J]. Automotive Engineering, 2023, 45(4): 527-540.
[3]	Jie Li,Xiaodong Wu,Min Xu,Yonggang Liu. Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios [J]. Automotive Engineering, 2023, 45(10): 1791-1802.
[4]	Chunyang Qi,Chuanxue Song,Shixin Song,Liqiang Jin,Da Wang,Feng Xiao. Research on Energy Management Strategy for Hybrid Electric Vehicles Based on Inverse Reinforcement Learning [J]. Automotive Engineering, 2023, 45(10): 1954-1964.
[5]	Zhenhai Gao,Xiangtong Yan,Fei Gao. A Decision-making Method for Longitudinal Autonomous Driving Based on Inverse Reinforcement Learning [J]. Automotive Engineering, 2022, 44(7): 969-975.
[6]	Dongjian Song,Bing Zhu,Jian Zhao,Jiayi Han,Yanchen Liu. Human-Like Behavior Decision-Making of Intelligent Vehicles Based on Driving Behavior Generation Mechanism [J]. Automotive Engineering, 2022, 44(12): 1797-1808.
[7]	Ningkang Yang,Lijin Han,Hui Liu,Xin Zhang. Research on Efficiency Optimization Based Energy Management Strategy for a Hybrid Electric Vehicle with Reinforcement Learning [J]. Automotive Engineering, 2021, 43(7): 1046-1056.
[8]	Hao Zhang,Qinhao Fan,Wei Wang,Jin Huang,Zhi Wang. Reinforcement Learning Based Energy Management Strategy for Hybrid Electric Vehicles Using Multi⁃mode Combustion [J]. Automotive Engineering, 2021, 43(5): 683-691.
[9]	Jinghua Guo,Wenchang Li,Yugong Luo,Tao Chen,Keqiang Li. Driver Car⁃Following Model Based on Deep Reinforcement Learning [J]. Automotive Engineering, 2021, 43(4): 571-579.
[10]	Zhenhai Gao,Xiangtong Yan,Fei Gao,Tianjun Sun. A Driver-like Decision-making Method for Longitudinal Autonomous Driving Based on DDPG [J]. Automotive Engineering, 2021, 43(12): 1737-1744.
[11]	Xiaolin Song,Xin Sheng,Haotian Cao,Mingjun Li,Binlin Huang Zhi Yi. Lane‑change Behavior Decision‑making of Intelligent Vehicle Based on Imitation Learning and Reinforcement Learning [J]. Automotive Engineering, 2021, 43(1): 59-67.
[12]	Zhou Nan, Chen Gang. Gearshifting Strategy for Robot-driven Vehicles Based on Deep Reinforcement Learning [J]. Automotive Engineering, 2020, 42(11): 1473-1481.
[13]	Xu Guoyan, Zong Xiaopeng, Yu Guizhen, Su Hongjie. A Research on Intelligent Obstacle Avoidance of Unmanned Vehicle Based on DDPG Algorithm [J]. , 2019, 41(2): 206-212.

Automatic Driving Edge Scene Generation Method Based on Scene Dynamics and Reinforcement Learning

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 27

Related Articles 13

Metrics

Comments

Recommended 10