基于局部窗口交叉注意力的轻量型语义分割

doi:10.19562/j.chinasae.qcgc.2023.09.010

Abstract

Abstract:

For the environmental perception of autonomous vehicle， the application of circumnavigation cameras in the Bird's Eye View （BEV） coordinate for semantic segmentation of lanes， vehicles and other targets has attracted wide attention. For the problems of linear increase of task inference delay due to the increasing number of cameras as well as difficulty in completing semantic segmentation tasks in real-time in autonomous driving perception， this paper proposes a lightweight semantic segmentation method based on local window cross-attention. The model adopts the improved EdgeNeXt backbone network to extract features. By constructing the local window cross attention between BEV query and image features， the feature query between the cross-camera perspectives is constructed. Then， the fused BEV feature map is decoded by up sampling residual block to obtain the BEV semantic segmentation results. The experimental results on the nuScenes dataset show that the proposed method achieves 35.1% mIoU in the lane static segmentation task of BEV map， which is 2.2% higher than that of HDMapNet. In particular， the inference speed increases by 58.2% compared with that of GKT， with the frame detection rate reaching 106 FPS.

Key words: BEV, semantic segmentation, local window, cross-attention

Zuliang Jin,Hanbing Wei,Liu Zheng,Lu Lou,Guofeng Zheng. Lightweight Semantic Segmentation Method Based on Local Window Cross Attention[J].Automotive Engineering, 2023, 45(9): 1617-1625.

Figures/Tables 10

References 22

1	王海，蔡柏湘，蔡英凤，等. 基于语义分割网络的路面积水与湿滑区域检测［J］.汽车工程，2021，43（4）：485-491.
	WANG H， CAI B X， CAI Y F， et al. Detection of water⁃covered and wet areas on road pavement based on semantic segmentation network［J］. Automotive Engineering， 2021， 43（4）： 485-491.
2	高涛，邢可，刘占文，等. 基于金字塔多尺度融合的交通标志检测算法［J］. 交通运输工程学报， 2022，22（3）： 210-224.
	GAO T， XING K， LIU Z W， et al. Traffic sign detection algorithm based on pyramid multi-scale fusion［J］.Journal of Traffic and Transportation Engineering， 2022，22（3）： 210-224.
3	CHEN L C， PAPANDREOU G， KOKKINOS I， et al. DeepLab： semantic image segmentation with deep convolutional nets， atrous convolution， and fully connected CRFs［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017，40（4）： 834-848.
4	RONNEBERGER O， FISCHER P， BROX T. U⁃Net： convolutional networks for biomedical image segmentation［C］.2015 International Conference on Medical Image Computing and Computer-Assisted Intervention （MICCAI 2015）， 2015：234-241.
5	MALLOT， HANSPETER A， LITTLE J， et al. Inverse perspective mapping simplifies optical flow computation and obstacle detection［J］. Biological Cybernetics， 1991， 64（3）： 177-185.
6	PHILION J， FIDLER S. Lift， splat， shoot： encoding images from arbitrary camera rigs by implicitly unprojecting to 3D ［C］.2020 European Conference on Computer Vision （ECCV 2020）， 2020：194-210.
7	LANG A H， VORA S， CAESAR H， et al. PointPillars： fast encoders for object detection from point clouds［C］.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. IEEE， 2019： 12689-12697.
8	HU A， MUREZ Z， MOHAN N， et al. FIERY： future instance prediction in bird's-eye view from surround monocular cameras［C］.2021 IEEE International Conference on Computer Vision （ICCV2021）， 2021：15253-15262.
9	HUANG J， HUANG G， ZHU Z， et al. Bevdet： highperformance multi-camera 3D object detection in bird-eye-view［J］. arXiv preprint arXiv：， 2021.
10	PAN B， SUN J， LEUNG H Y T， et al. Cross view semantic segmentation for sensing surroundings［J］. IEEE Robotics and Automation Letters， 2020，5（3）：4867-4873.
11	LI Q， WANG Y， WANG Y， et al. HDMapNet： an online HD map construction and evaluation framework［J］. arXiv preprint arXiv：，2021.
12	ZHOU B， KRHENBÜHL P. Cross-view transformers for real-time map-view semantic segmentation［C］.2022 IEEE Conference on Computer Vision and Pattern Recognition （CVPR2022）， 2022： 13750-13759.
13	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］.2017 In Advances in Neural Information Processing Systems （NIPS）， 2017.
14	LI Z， WANG W， LI H， et al. BEVFormer： learning bird’s-eye-view representation from multicamera images via spatiotemporal transformers［J］. arXiv preprint arXiv：， 2022.
15	CHEN S Y， CHENG T H， WANG X G， et al. Efficient and robust 2D-to-bev representation learning via geometry-guided kernel transformer［J］. arXiv preprint arXiv：， 2022.
16	LIN T Y， DOLLAR P， GIRSHICK R， et al. Feature pyramid networks for object detection［C］.2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR2017）. IEEE Computer Society， 2017：936-944.
17	MUHAMMAD M， ABDELRAHMAN S， HISHAM C， et al. EdgeNeXt： efficiently amalgamated CNN-transformer architecture for mobile vision applications［J］. arXiv preprint arXiv：2206.10589
18	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］.2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR2016）， 2016：770-778.
19	LONG J， SHELHAMER E， DARRELL T， et al. Fully convolutional networks for semantic segmentation［C］.2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR 2015）， 2015：3431-3440.
20	HOWARD A G， ZHU M， CHEN B， et al. Mobilenets： efficient convolutional neural networks for mobile vision applications［J］. CoRR abs/1704.04861 （2017）.
21	LIU Z， MAO H， WU C Y， et al. A convnet for the 2020s［C］.2022 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR2022）， 2022： 11966-11976.
22	CAESAR H， BANKITI V， LANG A H， et al. nuScenes： a multimodal dataset for autonomous driving［J］. arXiv preprint arXiv：， 2019.

EdgeNeXt	特征金字塔	局部窗口交叉注意力	IoU/%				推理延迟/ms
EdgeNeXt	特征金字塔	局部窗口交叉注意力	车道线	人行横道	道路边缘	平均	推理延迟/ms
			38.8	13.9	37.6	30.1	8.6
√			39.0	14.1	38.2	30.4	7.9
√	√		39.8	14.7	38.9	31.1	8.7
√		√	41.2	15.2	40.4	32.3	8.4
√	√	√	42.2	20.8	42.4	35.1	9.4

方法	图片尺寸	骨干网络	IoU/%				计算量/GFLOPs	推理延迟/ms	FPS
方法	图片尺寸	骨干网络	车道线	人行横道	道路边缘	平均	计算量/GFLOPs	推理延迟/ms	FPS
LLS	128×352	EfficientNet-B0	38.3	14.9	39.3	30.8	62.79	25.1	40
HDMapNet	128×352	EfficientNet-B0	40.6	18.7	39.5	32.9	62.82	19.3	52
CVT	224×480	EfficientNet-B4	39.6	13.8	39.5	30.9	40.01	20.3	49
GKT	224×480	EfficientNet-B4	41.1	15.7	41.0	32.6	42.12	14.9	67
本文方法	128×352	EdgeNeXt-S	42.2	20.8	42.4	35.1	21.57	9.1	106

[1]	Wenguang Wu,Shuangyue Tian,Zhiyong Zhang,Bin Zhang. Research on Semantic Segmentation of Uneven Features of Unpaved Road [J]. Automotive Engineering, 2023, 45(8): 1468-1478.
[2]	Lei Zhang, Keren Guan, Xiaolin Ding, Pengyu Guo, Zhenpo Wang, Fengchun Sun. Tire-Road Friction Estimation Method Based on Image Recognition and Dynamics Fusion [J]. Automotive Engineering, 2023, 45(7): 1222-1234.
[3]	Runhui Huang,Likun Hu,Mingfang Su,Daye Xu,Aoran Chen. Semantic Segmentation Method of LiDAR Point Cloud Based on 3D Conical Grid [J]. Automotive Engineering, 2022, 44(8): 1173-1182.
[4]	Dafang Wang,Lei Liu,Jiang Cao,Gang Zhao,Wenshuo Zhao,Wei Tang. Semantic Segmentation Method of Autonomous Driving Images Based on Atrous Spatial Pyramid Pooling [J]. Automotive Engineering, 2022, 44(12): 1818-1824.
[5]	Dafang Wang,Hai Shang,Jiang Cao,Tao Wang,Xiangteng Xia,Yulin Han. Semantic Segmentation Method of Point Cloud in Automatic Driving Scene Based on Self-attention Mechanism [J]. Automotive Engineering, 2022, 44(11): 1656-1664.
[6]	Shengjie Huang,Manjiang Hu,Yunshui Zhou,Zhouping Yin,Xiaohui Qin,Yougang Bian,Qianqian Jia. Vehicle Visual SLAM in Dynamic Scenes Based on Semantic Segmentation and Motion Consistency Constraints [J]. Automotive Engineering, 2022, 44(10): 1503-1510.
[7]	Xiangteng Xia,Dafang Wang,Jiang Cao,Gang Zhao,Jingming Zhang. Semantic Segmentation Method of On-board Lidar Point Cloud Based on Sparse Convolutional Neural Network [J]. Automotive Engineering, 2022, 44(1): 26-35.
[8]	Hai Wang,Baixiang Cai,Yingfeng Cai,Ze Liu,Kai Sun,Long Chen. Detection of Water⁃covered and Wet Areas on Road Pavement Based on Semantic Segmentation Network [J]. Automotive Engineering, 2021, 43(4): 485-491.

Lightweight Semantic Segmentation Method Based on Local Window Cross Attention

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 22

Related Articles 8

Metrics

Comments

Recommended 10