面向自动驾驶数据生成的风格迁移网络研究

doi:10.19562/j.chinasae.qcgc.2022.05.005

摘要/Abstract

摘要：

自动驾驶数据集的丰富性是保证基于深度学习的自动驾驶算法鲁棒性和可靠性的关键。当前的自动驾驶数据集在夜晚场景和各类气候、天气条件下的数据量仍十分有限，为满足无人驾驶领域的应用需求，本文中构建了风格迁移网络，可将当前自动驾驶数据集转换为夜晚、雪天等多种形式。该网络采用单编码器-双解码器结构，综合语义分割网络、跳跃连接和多尺度鉴别器等多种手段用于提高图像的生成质量，生成的图像具有良好的视觉效果。用真实数据训练deeplabv3+语义分割网络来评价生成图像的结果表明，本文采用的网络生成图像的平均交并比比基于双编码-双解码结构的两种网络（AugGAN和UNIT）分别提升了2.50%和4.41%。

关键词: 生成式对抗网络, 风格迁移, 深度学习, 自动驾驶

Abstract:

The data abundance of the autonomous driving dataset is the key to ensuring the robustness and reliability of autonomous driving algorithm based on deep learning， but the amount of data with night scenes and various climates and weather conditions in current autonomous driving datasets are still very limited. In order to meet the application needs in the field of unmanned driving， a style transfer network is built， which can convert the current autonomous driving data into various forms such as night and snow， etc. The network adopts a structure of single encoder-dual decoder， combined with various means such as semantic segmentation networks， skip connections， and multi-scale discriminators to improve the quality of generated images with good vision effects. Deeplabv3+ semantic segmentation network trained by real data is used to evaluate the images generated and the results show that the mean intersection over union of the images generated by the network adopted is 2.50 and 4.41 percentage points higher than that generated by AugGAN and UNIT networks with double encoder-double decoder structure respectively.

Key words: GANs, style transfer, deep learning, autonomous driving

王大方,杜京东,曹江,张梅,赵刚. 面向自动驾驶数据生成的风格迁移网络研究[J]. 汽车工程, 2022, 44(5): 684-690.

Dafang Wang,Jingdong Du,Jiang Cao,Mei Zhang,Gang Zhao. Research on Style Transfer Network for Autonomous Driving Data Generation[J]. Automotive Engineering, 2022, 44(5): 684-690.

图/表 14

图1

图2

图3

图4

图5

表1

表2

表3

表4

图6

图7

表5

图8

表6

参考文献 25

1	GEIGER A， LENZ P， STILLER C， et al. Vision meets robotics： the KITTI dataset［J］. International Journal of Robotics Research， 2013， 32（11）：1231-1237.
2	YU F， CHEN H， WANG X， et al. Bdd100k： a diverse driving dataset for heterogeneous multitask learning［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020： 2636-2645.
3	CORDTS M， OMRAN M， RAMOS S， et al. The cityscapes dataset for semantic urban scene understanding［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016： 3213-3223.
4	TIAN Y， PEI K， JANA S， et al. Deeptest： automated testing of deep-neural-network-driven autonomous cars［C］.Proceedings of the 40th International Conference on Software Engineering， 2018： 303-314.
5	GATYS L A， ECKER A S， BETHGE M. A neural algorithm of artistic style［J］. Journal of Vision， 2015.
6	JOHNSON J， ALAHI A， FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution［C］. European Conference on Computer Vision. Springer， Cham， 2016： 694-711.
7	IOFFE S， SZEGEDY C. Batch normalization： accelerating deep network training by reducing internal covariate shift［C］. International Conference on Machine Learning. PMLR， 2015： 448-456
8	ULYANOV D， VEDALDI A， LEMPITSKY V. Instance normalization： the missing ingredient for fast stylization［J］. arXiv. org， 2016.
9	LI Y， WANG N， LIU J， et al. Demystifying neural style transfer［C］. Twenty-sixth International Joint Conference on Artificial Intelligence， 2017.
10	ULYANOV D， VEDALDI A， LEMPITSKY V. Improved texture networks： maximizing quality and diversity in feed-forward stylization and texture synthesis［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 6924-6932.
11	HUANG X， BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization［C］. Proceedings of the IEEE International Conference on Computer Vision， 2017： 1501-1510.
12	KINGMA D P， WELLING M. Auto-encoding variational bayes［J］. arXiv.org， 2014.
13	GOODFELLOW I J， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial nets［M］. MIT Press， 2014.
14	ISOLA P， ZHU J Y， ZHOU T， et al. Image-to-image translation with conditional adversarial networks［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2017： 1125-1134.
15	MIRZA M， OSINDERO S. Conditional generative adversarial nets［J］. Computer Science， 2014：2672-2680.
16	HE D， XIA Y， QIN T， et al. Dual learning for machine translation［J］. Advances in Neural Information Processing Systems， 2016， 29： 820-828.
17	ZHU J Y， PARK T， ISOLA P， et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［C］. Proceedings of the IEEE International Conference on Computer Vision， 2017： 2223-2232.
18	KIM T， CHA M， KIM H， et al. Learning to discover cross-domain relations with generative adversarial networks［C］. International Conference on Machine Learning. PMLR， 2017： 1857-1865.
19	YI Z， ZHANG H， TAN P， et al. DualGAN： unsupervised dual learning for image-to-image translation［J］. IEEE Computer Society， 2017.
20	LIU M Y， BREUEL T， KAUTZ J. Unsupervised image-to-image translation networks［C］. Advances in Neural Information Processing Systems， 2017： 700-708.
21	LIN C T， HUANG S W， WU Y Y， et al. GAN-based day-to-night image style transfer for nighttime vehicle detection［J］. IEEE Transactions on Intelligent Transportation Systems， 2020.
22	ROS G， SELLART L， MATERZYNSKA J， et al. The synthia dataset： a large collection of synthetic images for semantic segmentation of urban scenes［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2016： 3234-3243.
23	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］. International Conference on Medical Image Computing and Computer-assisted Intervention. Springer， Cham， 2015： 234-241.
24	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation［C］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition， 2015： 3431-3440.
25	CHEN L C， ZHU Y， PAPANDREOU G， et al. Encoder-decoder with atrous separable convolution for semantic image segmentation［J］. Springer， Cham， 2018.

名称	型号
CPU	AMD Ryzen9 5950X
GPU	RTX3090 24GB
内存	芝奇DDR4 3200MHz 64GB
操作系统	Ubuntu18.04
深度学习框架	Pytorch1.7

类型	通道	尺寸/步幅	特征图尺寸
Conv+IN+Relu	64	7×7/1	513×304
Conv+IN+Relu	128	3×3/2	256×152
Conv+IN+Relu	256	3×3/2	128×76
Conv+IN+Relu	512	3×3/2	64×38
10*Resblock	512	3×3/1	64×38

类型	通道	尺寸/步幅	特征图尺寸
Resbock	512	3×3/1	64×38
插值+Conv+IN+Relu	256	3×3/1	128×76
拼接+Conv+IN+Relu	256	3×3/1	128×76
插值+Conv+IN+Relu	128	3×3/1	256×152
拼接+Conv+IN+Relu	128	3×3/1	256×152
插值+Conv+IN+Relu	64	3×3/1	513×304
拼接+Conv+IN+Relu	64	3×3/1	513×304
Conv+IN+Relu	32	7×7	513×304
Conv+Tanh/Conv	3/16	1×1	513×304

类型	通道	尺寸/步幅	多尺度特征图尺寸
Conv+BN+LeaklyRelu	64	4×4/2	256×152/128×76/64×38
Conv+BN+LeaklyRelu	128	4×4/2	128×76/64×38/32×19
Conv+BN+LeaklyRelu	256	4×4/2	64×38/32×19/16×9
Conv+BN+LeaklyRelu	512	4×4/2	32×19/16×9/8×4
Conv	1	1×1/1	32×19/16×9/8×4

网络	像素准确率/%	类别像素准确率/%	平均交并比/%	频权交并比/%
Ground Truth	96.81	79.31	68.80	93.97
AugGAN	93.23	75.15	58.36	88.25
UNIT	93.05	71.61	56.45	87.94
本文	94.98	75.26	60.86	90.95