基于场景动力学和强化学习的自动驾驶边缘测试场景生成方法

doi:10.19562/j.chinasae.qcgc.2022.07.004

摘要/Abstract

摘要：

为解决小概率高风险边缘测试场景的问题，本文提出一种基于场景动力学和强化学习的边缘场景生成方法，实现边缘场景的自动生成，能模拟真实世界中车辆间的对抗与博弈行为的特征。首先将随时间动态变化的场景模型由一组微分方程描述为场景动力学系统；然后利用神经网络作为通用函数逼近器来构造场景黑盒控制器，并基于强化学习实现边缘场景控制器的优化求解；最后以超车切入场景为例，在Matlab/Simulink软件进行仿真验证，结果表明，边缘场景强化生成模型在场景交互博弈、覆盖率和可重复测试等方面具有良好的性能。

关键词: 智能驾驶测试, 边缘场景, 场景动力学, 场景控制器, 强化学习

Abstract:

For solving the problem of low-probability and high-risk edge test scenes， an edge scene reinforcement generation method based on scene dynamics and reinforcement learning is proposed to fulfill the automatic generation of edge scenes and simulate the features of confrontation and game behavior between vehicles in the real world. Firstly， the scene models dynamically changing with time is described by a set of differential equations as a scene dynamic system. Then， neural network is used as a general function approximator， to construct the scene black-box controller for fulfilling the optimization solving of edge scene controller based on reinforcement learning. Finally， with the cut-in scene for overtaking as an example， a verification simulation is performed with Matlab/Simulink software. The results show that the edge scene models generated by reinforcement learning exhibit an excellent performance in terms of scene interactive gaming， scene coverage and repeatable test.

Key words: intelligent driving test, edge scenes, scene dynamics, scene controller, reinforcement learning

李江坤,邓伟文,任秉韬,王文奇,丁娟. 基于场景动力学和强化学习的自动驾驶边缘测试场景生成方法[J]. 汽车工程, 2022, 44(7): 976-986.

Jiangkun Li,Weiwen Deng,Bingtao Ren,Wenqi Wang,Juan Ding. Automatic Driving Edge Scene Generation Method Based on Scene Dynamics and Reinforcement Learning[J]. Automotive Engineering, 2022, 44(7): 976-986.

图/表 17

图 1

图 2

表 1

典型工况场景变量选取过程"

跟驰场景	场景介绍：主车跟驰交通车行驶，交通车能够通过突然加减速干扰主车。交通车编号1，主车编号2。

	状态量： $x 1 = v 1, x 2 = a 1, x 3 = d r e f, x 4 = v 2$ 输入量： $u 1 = f a$
不规则切入场景	场景介绍：主车沿路行驶，交通车按照规定路径从相邻车道切入实施干扰。交通车编号1，主车编号2。

	状态量： $x 1 = v 1, x 2 = a 1, x 3 = d 1, d 2, d 3, x 4 = Δ t 1, Δ t 2, Δ t 3$ 输入量： $u 1 = f a$
多车换道场景	场景介绍：主车换道过程中，前面交通车会加速，后面交通车会加速实施干扰。交通车编号1、2，主车编号3

	状态量： $x 1 = v 1, x 2 = a 1, x 3 = d r e f 1, x 4 = v 2, x 5 = a 2, x 6 = d r e f 2, x 7 = d r e s t, x 8 ? = v 3, x 9 = w 3$ 输入量： $u 1 = f a 1, u 2 = f a 2$

表 1

图 3

图 4

图 5

图 6

表 2

表 3

图 7

图 8

图 9

图 10

图 11

图 12

图 13

图 14

参考文献 27

1	邓伟文，李江坤，任秉韬，等. 面向自动驾驶的仿真场景自动生成方法综述［J］. 中国公路学报， 2021： 1-21.
	DENG W W, LI J K, REN B T, et al. A survey on automatic generation methods on simulation scenario for autonomous driving [J]. China Journal of Highway and Transport, 2021: 1-21.
2	朱冰, 张培兴, 赵健, 等. 基于场景的自动驾驶汽车虚拟测试研究进展[J]. 中国公路学报, 2019,32(6): 1-19.
	ZHU B, ZHANG P X, ZHAO J, et al. Review of scenario-based virtual validation methods for automated vehicles[J]. China Journal of Highway and Transport, 2019, 32(6): 1-19.
3	田鹏, 左大义, 高艳春, 等. 面向实际场景的人工智能脆弱性分析[J]. 计算机技术与发展, 2021,31(11): 129-135.
	TIAN P,ZUO D Y,GAO Y C, et al. Vulnerability analysis of artificial intelligence in real world[J]. Computer Technology and Development,2021,31(11):129-135.
4	徐向阳, 胡文浩, 董红磊, 等. 自动驾驶汽车测试场景构建关键技术综述[J]. 汽车工程, 2021,43(4): 610-619.
	XU X Y, HU W H, DONG H L, et al. Review of key technologies for autonomous vehicle test scenario construction[J]. Automotive Engineering, 2021, 43(4): 610-619.
5	吴旭阳,蒙昊蓝,邢星宇,等. 自动驾驶系统边缘测试用例生成[J].同济大学学报（自然科学版）,2018,46(12):111-115.
	WU X Y, MENG H L, XING X Y, et al. Autonomous driving system edge test case generation[J]. Journal of Tongji University (Natural Science) ,2018,46(12):111-115.
6	白雪松,邓伟文,任秉韬,等.一种自动驾驶仿真场景要素的提取方法[J].汽车工程, 2021,43(7):1030-1036.
	BAI X S, DENG W W, REN B T, et al. An extraction method of scenario elements for autonomous driving simulation[J]. Automotive Engineering, 2021, 43(7): 1030-1036.
7	ZHAO D, LAM H, PENG H, et al. Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(3): 595-607.
8	SCHWALL M, DANIEL T, VICTOR T, et al. Waymo public road safety performance data[J]. ArXiv, 2020.
9	陈君毅, 冯天悦, 刘力豪, 等. 面向决策规划系统测试的具体场景自动化生成方法[J]. 汽车技术, 2020(10): 45-50.
	CHEN J Y, FENG T Y, LIU L H, et al. An automatic generation method of concrete scenario for decision making and planning system test[J]. Automobile Technology,2020(10): 45-50.
10	任秉韬, 邓伟文, 白雪松, 等. 面向智能驾驶测试的仿真场景构建技术综述[J]. 中国图象图形学报, 2021,26(1): 1-12.
	REN B T, DENG W W, BAI X S, et al. Technologies of virtual scenario construction for intelligent driving testing[J]. Journal of Image and Graphics,2021, 26(1): 1-12.
11	DUAN J, GAO F, HE Y. Test scenario generation and optimization technology for intelligent driving systems[J]. IEEE Intelligent Transportation Systems Magazine, 2020.
12	FENG S, FENG Y, YU C, et al. Testing scenario library generation for connected and automated vehicles, part I: methodology[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3): 1573-1582.
13	ZHU B, ZHANG P, ZHAO J, et al. Hazardous scenario enhanced generation for automated vehicle testing based on optimization searching method[J]. IEEE Transactions on Intelligent Transportation Systems, 2021: 1-11.
14	周博林, 张宗实, 陈蔯. ASAM OpenX与基于场景的自动驾驶仿真测试[J]. 标准科学, 2021(S1): 110-122.
	ZHOU B L, ZHANG Z S, CHEN C. ASAM OpenX and scenario based simulation test for autonomous driving[J]. Standard Science, 2021(S1): 110-122.
15	余唯之, 陈桢, 苏奕敏. OpenSCENARIO研究综述[J]. 中国汽车, 2021(6): 17-23.
	YU W Z, CHEN Z, SUN Y M. OpenSCENARIO research review[J]. Chinese Automobile, 2021(6): 17-23.
16	FENG S, FENG Y, SUN H, et al. Testing scenario library generation for connected and automated vehicles: an adaptive framework[J]. IEEE Transactions on Intelligent Transportation Systems, 2020: 1-10.
17	KLÜCK F, ZIMMERMANN M, WOTAWA F, et al. Genetic algorithm-based test parameter optimization for ADAS system testing[C].2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS). IEEE, 2019: 418-425.
18	BEGLEROVIC H, STOLZ M, HORN M. Testing of autonomous vehicles using surrogate models and stochastic optimization[C].2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2017.
19	9］ FENG S, FENG Y, SUN H, et al. Testing scenario library generation for connected and automated vehicles, part Ⅱ: case studies[J]. IEEE Transactions on Intelligent Transportation Systems, 2020: 1-13.
20	FENG S, YAN X, SUN H, et al. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment[J]. Nature Communications, 2021, 12(1).
21	WENG B, CAPITO RUIZ L J, OZGUNER U, et al. Towards guaranteed safety assurance of automated driving systems with scenario sampling: an invariant set perspective[J]. IEEE Transactions on Intelligent Vehicles, 2021.
22	邝先验. 城市混合交通流微观仿真建模研究[D].广州：华南理工大学，2014.
	KUANG X Y. Research on microscopic simulation modeling for urban mixed traffic flow[D]. Guangzhou:South China University of Technology，2014.
23	LILLICRAP T, HUNT J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. ICLR, 2015.
24	杨宁康, 韩立金, 刘辉, 等. 基于效率优化的混合动力车辆强化学习能量管理策略研究[J]. 汽车工程, 2021,43(7): 1046-1056.
	YANG N K, HAN L J, LIU H, et al. Research on efficiency optimization based energy management strategy for a hybrid electric vehicle with reinforcement learning[J]. Automotive Engineering, 2021, 43(7): 1046-1056.
25	宋晓琳, 盛鑫, 曹昊天, 等. 基于模仿学习和强化学习的智能车辆换道行为决策[J]. 汽车工程, 2021,43(1): 59-67.
	SONG X L, SHENG X, CAO H T, et al. Lane‑change behavior decision‑making of intelligent vehicle based on imitation learning and reinforcement learning[J]. Automotive Engineering, 2021, 43(1): 59-67.
26	郭景华, 李文昌, 罗禹贡, 等. 基于深度强化学习的驾驶员跟车模型研究[J]. 汽车工程, 2021,43(4): 571-579.
	GUO J H, LI W C, LUO Y G, et al. Driver car⁃ following model based on deep reinforcement learning[J]. Automotive Engineering, 2021, 43(4): 571-579.
27	TREIBER M, HENNECKE A, HELBING D. Congested traffic states in empirical observations and microscopic simulations[J]. Phys. Rev. E, 2000,62(2):1805-1824.

网络	层	维度	激活函数
Actor网络	全连接层1	（5，100）	RuLU
	全连接层2	（100，100）	RuLU
	全连接层3	（100，100）	RuLU
	全连接层4	（100，1）	Tanh
Critic网络	全连接层1	（12，100）	RuLU
	全连接层2	（100，100）	RuLU
	全连接层3	（2，100）	RuLU
	全连接层4	（100，100）	RuLU
	全连接层5	（100，1）	RuLU

变量	描述	数值
maxepisodes	最大训练次数	5 000
MiniBatchSize	采样样本大小	64
Ts	采样步长	0.1 s
ActorLearnRate	Actor网络学习率	0.000 1
CriticLearnRate	Critic网络学习率	0.001
DiscountFactor	奖励衰减系数	0.99
Smoothing factor	平滑因子	0.001

[1]	刘卫国,项志宇,刘伟平,齐道新,王子旭. 基于分布式强化学习的车辆控制算法研究[J]. 汽车工程, 2023, 45(9): 1637-1645.
[2]	金立生,韩广德,谢宪毅,郭柏苍,刘国峰,朱文涛. 基于强化学习的自动驾驶决策研究综述[J]. 汽车工程, 2023, 45(4): 527-540.
[3]	李捷,吴晓东,许敏,刘永刚. 基于强化学习的城市场景多目标生态驾驶策略[J]. 汽车工程, 2023, 45(10): 1791-1802.
[4]	齐春阳,宋传学,宋世欣,靳立强,王达,肖峰. 基于逆强化学习的混合动力汽车能量管理策略研究[J]. 汽车工程, 2023, 45(10): 1954-1964.
[5]	高振海,闫相同,高菲. 基于逆向强化学习的纵向自动驾驶决策方法[J]. 汽车工程, 2022, 44(7): 969-975.
[6]	宋东鉴,朱冰,赵健,韩嘉懿,刘彦辰. 基于驾驶行为生成机制的智能汽车类人行为决策[J]. 汽车工程, 2022, 44(12): 1797-1808.
[7]	杨宁康,韩立金,刘辉,张欣. 基于效率优化的混合动力车辆强化学习能量管理策略研究[J]. 汽车工程, 2021, 43(7): 1046-1056.
[8]	张昊,范钦灏,王巍,黄晋,王志. 基于强化学习的多燃烧模式混合动力能量管理策略[J]. 汽车工程, 2021, 43(5): 683-691.
[9]	郭景华,李文昌,罗禹贡,陈涛,李克强. 基于深度强化学习的驾驶员跟车模型研究[J]. 汽车工程, 2021, 43(4): 571-579.
[10]	高振海,闫相同,高菲,孙天骏. 仿驾驶员DDPG汽车纵向自动驾驶决策方法[J]. 汽车工程, 2021, 43(12): 1737-1744.
[11]	宋晓琳,盛鑫,曹昊天,李明俊,易滨林,黄智. 基于模仿学习和强化学习的智能车辆换道行为决策[J]. 汽车工程, 2021, 43(1): 59-67.
[12]	周楠, 陈刚. 机器人驾驶车辆深度强化学习换挡策略^*[J]. 汽车工程, 2020, 42(11): 1473-1481.
[13]	徐国艳, 宗孝鹏, 余贵珍, 苏鸿杰. 基于DDPG的无人车智能避障方法研究^*[J]. 汽车工程, 2019, 41(2): 206-212.