面向自动驾驶数据生成的风格迁移网络研究

doi:10.19562/j.chinasae.qcgc.2022.05.005

Abstract

Abstract:

The data abundance of the autonomous driving dataset is the key to ensuring the robustness and reliability of autonomous driving algorithm based on deep learning， but the amount of data with night scenes and various climates and weather conditions in current autonomous driving datasets are still very limited. In order to meet the application needs in the field of unmanned driving， a style transfer network is built， which can convert the current autonomous driving data into various forms such as night and snow， etc. The network adopts a structure of single encoder-dual decoder， combined with various means such as semantic segmentation networks， skip connections， and multi-scale discriminators to improve the quality of generated images with good vision effects. Deeplabv3+ semantic segmentation network trained by real data is used to evaluate the images generated and the results show that the mean intersection over union of the images generated by the network adopted is 2.50 and 4.41 percentage points higher than that generated by AugGAN and UNIT networks with double encoder-double decoder structure respectively.

Key words: GANs, style transfer, deep learning, autonomous driving

Dafang Wang,Jingdong Du,Jiang Cao,Mei Zhang,Gang Zhao. Research on Style Transfer Network for Autonomous Driving Data Generation[J].Automotive Engineering, 2022, 44(5): 684-690.

Figures/Tables 14

References 25

1	GEIGER A， LENZ P， STILLER C， et al. Vision meets robotics： the KITTI dataset［J］. International Journal of Robotics Research， 2013， 32（11）：1231-1237.
2	YU F， CHEN H， WANG X， et al. Bdd100k： a diverse driving dataset for heterogeneous multitask learning［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020： 2636-2645.
3	CORDTS M， OMRAN M， RAMOS S， et al. The cityscapes dataset for semantic urban scene understanding［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016： 3213-3223.
4	TIAN Y， PEI K， JANA S， et al. Deeptest： automated testing of deep-neural-network-driven autonomous cars［C］.Proceedings of the 40th International Conference on Software Engineering， 2018： 303-314.
5	GATYS L A， ECKER A S， BETHGE M. A neural algorithm of artistic style［J］. Journal of Vision， 2015.
6	JOHNSON J， ALAHI A， FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution［C］. European Conference on Computer Vision. Springer， Cham， 2016： 694-711.
7	IOFFE S， SZEGEDY C. Batch normalization： accelerating deep network training by reducing internal covariate shift［C］. International Conference on Machine Learning. PMLR， 2015： 448-456
8	ULYANOV D， VEDALDI A， LEMPITSKY V. Instance normalization： the missing ingredient for fast stylization［J］. arXiv. org， 2016.
9	LI Y， WANG N， LIU J， et al. Demystifying neural style transfer［C］. Twenty-sixth International Joint Conference on Artificial Intelligence， 2017.
10	ULYANOV D， VEDALDI A， LEMPITSKY V. Improved texture networks： maximizing quality and diversity in feed-forward stylization and texture synthesis［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 6924-6932.
11	HUANG X， BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization［C］. Proceedings of the IEEE International Conference on Computer Vision， 2017： 1501-1510.
12	KINGMA D P， WELLING M. Auto-encoding variational bayes［J］. arXiv.org， 2014.
13	GOODFELLOW I J， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial nets［M］. MIT Press， 2014.
14	ISOLA P， ZHU J Y， ZHOU T， et al. Image-to-image translation with conditional adversarial networks［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 1125-1134.
15	MIRZA M， OSINDERO S. Conditional generative adversarial nets［J］. Computer Science， 2014：2672-2680.
16	HE D， XIA Y， QIN T， et al. Dual learning for machine translation［J］. Advances in Neural Information Processing Systems， 2016， 29： 820-828.
17	ZHU J Y， PARK T， ISOLA P， et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［C］. Proceedings of the IEEE International Conference on Computer Vision， 2017： 2223-2232.
18	KIM T， CHA M， KIM H， et al. Learning to discover cross-domain relations with generative adversarial networks［C］. International Conference on Machine Learning. PMLR， 2017： 1857-1865.
19	YI Z， ZHANG H， TAN P， et al. DualGAN： unsupervised dual learning for image-to-image translation［J］. IEEE Computer Society， 2017.
20	LIU M Y， BREUEL T， KAUTZ J. Unsupervised image-to-image translation networks［C］. Advances in Neural Information Processing Systems， 2017： 700-708.
21	LIN C T， HUANG S W， WU Y Y， et al. GAN-based day-to-night image style transfer for nighttime vehicle detection［J］. IEEE Transactions on Intelligent Transportation Systems， 2020.
22	ROS G， SELLART L， MATERZYNSKA J， et al. The synthia dataset： a large collection of synthetic images for semantic segmentation of urban scenes［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2016： 3234-3243.
23	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］. International Conference on Medical Image Computing and Computer-assisted Intervention. Springer， Cham， 2015： 234-241.
24	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2015： 3431-3440.
25	CHEN L C， ZHU Y， PAPANDREOU G， et al. Encoder-decoder with atrous separable convolution for semantic image segmentation［J］. Springer， Cham， 2018.

[1]	ZHOU Wei, ZHANG Cheng-Ning, LI Jun-Qiu. [J]. , 2016, 38(12): 1407 -1414 .
[2]	ZONG Yi-Qi, GU Zheng-Qi, LUO Ze-Min, JIANG Cai-Mao, ZHANG Qi-Dong, YANG Zhen-Dong. [J]. , 2016, 38(12): 1427 -1433 .
[3]	ZHANG Ying-Chao, ZHAN Da-Peng, ZHAO Jing, ZHANG Zhe, LI Jie. [J]. , 2016, 38(12): 1434 -1439 .
[4]	TANG You-Ming, YAN Ling-Bo, LUO Qian, CAO Li-Bo. [J]. , 2016, 38(12): 1452 -1458 .
[5]	ZHANG Lei, LU Jian-Wei, JIANG Jun-Zhao, YAN Pei-Lei, LI Lei. [J]. , 2016, 38(12): 1477 -1482 .
[6]	ZHAO Liang, ZHANG Qiang, GUO Kong-Hui. [J]. , 2017, 39(1): 86 -91 .
[7]	SHEN Deng-Feng, WANG Chen, YU Hai-Sheng, ZHANG Tong, YI Xian-Ke. [J]. , 2017, 39(1): 15 -22 .
[8]	ZOU Tie-Fang, HU Lin, LI Ping-Fan, YI Liang. [J]. , 2017, 39(1): 41 -46 .
[9]	ZHOU Kai, ZANG Jing-Lun. [J]. , 2017, 39(2): 127 -132 .
[10]	ZUO Wen-Jie, CHEN Ji-Shun, LI Yi-Wen, LIU Nian. [J]. , 2017, 39(2): 145 -149 .

名称	型号
CPU	AMD Ryzen9 5950X
GPU	RTX3090 24GB
内存	芝奇DDR4 3200MHz 64GB
操作系统	Ubuntu18.04
深度学习框架	Pytorch1.7

类型	通道	尺寸/步幅	特征图尺寸
Conv+IN+Relu	64	7×7/1	513×304
Conv+IN+Relu	128	3×3/2	256×152
Conv+IN+Relu	256	3×3/2	128×76
Conv+IN+Relu	512	3×3/2	64×38
10*Resblock	512	3×3/1	64×38

类型	通道	尺寸/步幅	特征图尺寸
Resbock	512	3×3/1	64×38
插值+Conv+IN+Relu	256	3×3/1	128×76
拼接+Conv+IN+Relu	256	3×3/1	128×76
插值+Conv+IN+Relu	128	3×3/1	256×152
拼接+Conv+IN+Relu	128	3×3/1	256×152
插值+Conv+IN+Relu	64	3×3/1	513×304
拼接+Conv+IN+Relu	64	3×3/1	513×304
Conv+IN+Relu	32	7×7	513×304
Conv+Tanh/Conv	3/16	1×1	513×304

类型	通道	尺寸/步幅	多尺度特征图尺寸
Conv+BN+LeaklyRelu	64	4×4/2	256×152/128×76/64×38
Conv+BN+LeaklyRelu	128	4×4/2	128×76/64×38/32×19
Conv+BN+LeaklyRelu	256	4×4/2	64×38/32×19/16×9
Conv+BN+LeaklyRelu	512	4×4/2	32×19/16×9/8×4
Conv	1	1×1/1	32×19/16×9/8×4

网络	像素准确率/%	类别像素准确率/%	平均交并比/%	频权交并比/%
Ground Truth	96.81	79.31	68.80	93.97
AugGAN	93.23	75.15	58.36	88.25
UNIT	93.05	71.61	56.45	87.94
本文	94.98	75.26	60.86	90.95

Research on Style Transfer Network for Autonomous Driving Data Generation

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 25

Related Articles 15

Metrics

Comments

Recommended 10

项目	像素准确率/%	类别像素准确率/%	平均交并比/%	频权交并比/%
加入前	94.33	75.04	61.19	89.79
加入后	94.98	75.26	60.86	90.95

[1]	Xiaocong Zhao,Shiyu Fang,Zirui Li,Jian Sun. Extraction and Application of Key Utility Term for Social Driving Interaction [J]. Automotive Engineering, 2024, 46(2): 230-240.
[2]	Ze Gao, Zunkang Chu, Jiasheng Shi, Fu Lin, Weixiong Rao, Haiyan Yu. Research on Fast Prediction Method of Stress Field of Automotive Parts Based on Graph Network [J]. Automotive Engineering, 2024, 46(1): 170-178.
[3]	Yanli Ma, Qin Qin, Fangqi Dong, Yining Lou. Takeover Risk Assessment Model Based on Risk Field Theory Under Different Cognitive Secondary Tasks [J]. Automotive Engineering, 2024, 46(1): 9-17.
[4]	Weiguo Liu,Zhiyu Xiang,Weiping Liu,Daoxin Qi,Zixu Wang. Research on Vehicle Control Algorithm Based on Distributed Reinforcement Learning [J]. Automotive Engineering, 2023, 45(9): 1637-1645.
[5]	Weiguo Liu,Zhiyu Xiang,Rui Liu,Guodong Li,Zixu Wang. Research on End-to-End Vehicle Motion Planning Method Based on Deep Learning [J]. Automotive Engineering, 2023, 45(8): 1343-1352.
[6]	Ming Wang,Xiaolin Tang,Kai Yang,Guofa Li,Xiaosong Hu. A Motion Planning Method for Autonomous Vehicles Considering Prediction Risk [J]. Automotive Engineering, 2023, 45(8): 1362-1372.
[7]	Dongyu Zhao, Shuen Zhao. Autonomous Driving 3D Object Detection Based on Cascade YOLOv7 [J]. Automotive Engineering, 2023, 45(7): 1112-1122.
[8]	Jiahao Zhao,Zhiquan Qi,Zhifeng Qi,Hao Wang,Lei He. Calculation of Heading Angle of Parallel Large Vehicle Based on Tire Feature Points [J]. Automotive Engineering, 2023, 45(6): 1031-1039.
[9]	Lisheng Jin,Guangde Han,Xianyi Xie,Baicang Guo,Guofeng Liu,Wentao Zhu. Review of Autonomous Driving Decision-Making Research Based on Reinforcement Learning [J]. Automotive Engineering, 2023, 45(4): 527-540.
[10]	Yanyan Chen,Hai Wang,Yingfeng Cai,Long Chen,Yicheng Li. Efficient Automatic Driving Instance Segmentation Method Based on Detection [J]. Automotive Engineering, 2023, 45(4): 541-550.
[11]	Qingyang Huang,Xiaoping Jin,Yikang Zhang. Analysis of Drivers' Driving Posture Change Rule Under the Condition of Automatic Driving Level Improvement [J]. Automotive Engineering, 2023, 45(3): 382-392.
[12]	Fengchong Lan,Jikai Chen,Jiqing Chen,Xinping Jiang,Zihan Li,Wei Pan. Research on Lithium Battery Remaining Useful Life Prediction Method Driven by Real Vehicle Data [J]. Automotive Engineering, 2023, 45(2): 175-182.
[13]	Linhui Li,Xinliang Zhang,Yifan Fu,Jing Lian,Jiaxu Ma. Research on Visible Light and Infrared Post-Fusion Detection Based on TC-YOLOv7 Algorithm [J]. Automotive Engineering, 2023, 45(12): 2280-2290.
[14]	Zhengfa Liu,Ya Wu,Peigen Liu,Rongqi Gu,Guang Chen. Cross-Domain Object Detection for Intelligent Driving Based on Joint Distribution Matching of Features and Labels [J]. Automotive Engineering, 2023, 45(11): 2082-2091.
[15]	Xiaojun Zhang,Jingzhe Xi,Yanlei Shi,Anlu Yuan. Lightweight YOLOv7-R Algorithm for Road-Side View Target Detection [J]. Automotive Engineering, 2023, 45(10): 1833-1844.