Research on Key Technology of Ship Re-Identification Based on the USV-UAV Collaboration
Abstract
:1. Introduction
2. Materials and Methods
2.1. Datasets
- Contain multiple image samples under the same ID label.
- Include the same ID label images captured from multiple views.
- Each image sample should feature a complete ship target.
- Image samples of the ship target should maintain similar main features.
- Query images should involve as many angles of the ship target as possible.
2.1.1. Dataset Collection
2.1.2. Dataset Processing
2.2. Fine-Grained Feature Network Design
2.2.1. Non-Local Module
2.2.2. GeM Pooling
2.2.3. Multi-Task Loss Function
- Classification loss
- 2.
- Metric Loss
2.2.4. Evaluation Metric for ReID
2.3. Multi-View Ranking Optimization Based on the USV-UAV Collaboration
3. Results
3.1. Implementation Details
3.2. Comparison with the State-of-the-Art and Ablation Experiment
3.3. Generalization Performance
3.4. Background Noise
3.5. Homologous and Heterologous Multi-View Fusion Retrieval Ranking Performance
3.6. Fusion Time Consumption
4. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Zhang, Z.; Ni, G.; Xu, Y. Trajectory prediction based on AIS and BP neural network. In Proceedings of the 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China, 11–13 December 2020; pp. 601–605. [Google Scholar]
- Zhao, L.; Yang, J.; Shi, G. A Correction Method for Time of Ship Trajectories Based on AIS. In Proceedings of the 1st International Conference on Big Data Research, Osaka, Japan, 22–24 October 2017; pp. 83–88. [Google Scholar]
- Zheng, Z.; Zheng, L.; Yang, Y. A discriminatively learned CNN embedding for person reidentification. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 2017, 14, 1–20. [Google Scholar] [CrossRef]
- Li, J.; Wang, J.; Tian, Q.; Gao, W.; Zhang, S. Global-local temporal representations for video person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 3958–3967. [Google Scholar]
- Wang, Y.; Peng, J.; Wang, H.; Wang, M. Progressive learning with multi-scale attention network for cross-domain vehicle re-identification. Sci. China Inf. Sci. 2022, 65, 160103. [Google Scholar] [CrossRef]
- Lian, J.; Wang, D.; Zhu, S.; Wu, Y.; Li, C. Transformer-based attention network for vehicle re-identification. Electronics 2022, 11, 1016. [Google Scholar] [CrossRef]
- Zheng, L.; Zhang, H.; Sun, S.; Chandraker, M.; Yang, Y.; Tian, Q. Person re-identification in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1367–1376. [Google Scholar]
- Suh, Y.; Wang, J.; Tang, S.; Mei, T.; Lee, K.M. Part-aligned bilinear representations for person re-identification. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 402–419. [Google Scholar]
- Sun, Y.; Zheng, L.; Yang, Y.; Tian, Q.; Wang, S. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 480–496. [Google Scholar]
- Farenzena, M.; Bazzani, L.; Perina, A.; Murino, V.; Cristani, M. Person re-identification by symmetry-driven accumulation of local features. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 2360–2367. [Google Scholar]
- Lin, Y.; Zheng, L.; Zheng, Z.; Wu, Y.; Hu, Z.; Yan, C.; Yang, Y. Improving person re-identification by attribute and identity learning. Pattern Recognit. 2019, 95, 151–161. [Google Scholar] [CrossRef]
- Matsukawa, T.; Suzuki, E. Person re-identification using CNN features learned from combination of attributes. In Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico, 4–8 December 2016; pp. 2428–2433. [Google Scholar]
- Zheng, Z.; Zheng, L.; Yang, Y. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 3754–3762. [Google Scholar]
- Sun, Y.; Zheng, L.; Deng, W.; Wang, S. Svdnet for pedestrian retrieval. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 3800–3808. [Google Scholar]
- Chen, D.; Xu, D.; Li, H.; Sebe, N.; Wang, X. Group consistent similarity learning via deep crf for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 8649–8658. [Google Scholar]
- Wang, Y.; Wang, L.; You, Y.; Zou, X.; Chen, V.; Li, S.; Huang, G.; Hariharan, B.; Weinberger, K.Q. Resource aware person re-identification across multiple resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 8042–8051. [Google Scholar]
- Song, C.; Huang, Y.; Ouyang, W.; Wang, L. Mask-guided contrastive attention model for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1179–1188. [Google Scholar]
- Hermans, A.; Beyer, L.; Leibe, B. In defense of the triplet loss for person re-identification. ar**v 2017, ar**v:1703.07737. [Google Scholar]
- Varior, R.R.; Shuai, B.; Lu, J.; Xu, D.; Wang, G. A siamese long short-term memory architecture for human re-identification. In Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016; pp. 135–153. [Google Scholar]
- Ye, M.; Liang, C.; Wang, Z.; Leng, Q.; Chen, J. Ranking optimization for person re-identification via similarity and dissimilarity. In Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia, 26–30 October 2015; pp. 1239–1242. [Google Scholar]
- Zhong, Z.; Zheng, L.; Cao, D.; Li, S. Re-ranking person re-identification with k-reciprocal encoding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1318–1327. [Google Scholar]
- Zeng, G.; Wang, R.; Yu, W.; Lin, A.; Li, H.; Shang, Y. A transfer learning-based approach to maritime warships re-identification. Eng. Appl. Artif. Intell. 2023, 125, 106696. [Google Scholar] [CrossRef]
- Qiao, D.; Liu, G.; Dong, F.; Jiang, S.-X.; Dai, L. Marine vessel re-identification: A large-scale dataset and global-and-local fusion-based discriminative feature learning. IEEE Access 2020, 8, 27744–27756. [Google Scholar] [CrossRef]
- Ghahremani, A.; Alkanat, T.; Bondarev, E.; de With, P.H. Maritime vessel re-identification: Novel VR-VCA dataset and a multi-branch architecture MVR-net. Mach. Vis. Appl. 2021, 32, 1–14. [Google Scholar] [CrossRef]
- Groot, H.G.; Zwemer, M.H.; Wijnhoven, R.G.; Bondarau, E. Vessel-speed enforcement system by multi-camera detection and re-identification. In Proceedings of the 15th International Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP2020), Valetta, Malta, 27–29 February 2020; pp. 268–277. [Google Scholar]
- Ribeiro, R.; Cruz, G.; Matos, J.; Bernardino, A. A data set for airborne maritime surveillance environments. IEEE Trans. Circuits Syst. Video Technol. 2017, 29, 2720–2732. [Google Scholar] [CrossRef]
- Ghahremani, A.; Kong, Y.; Bondarev, E.; de With, P.H. Towards parameter-optimized vessel re-identification based on IORnet. In Proceedings of the Computational Science–ICCS 2019: 19th International Conference, Faro, Portugal, 12–14 June 2019; pp. 125–136. [Google Scholar]
- Li, W.; Ge, Y.; Guan, Z.; Ye, G. Synchronized Motion-Based UAV–USV Cooperative Autonomous Landing. J. Mar. Sci. Eng. 2022, 10, 1214. [Google Scholar] [CrossRef]
- Shao, G.; Ma, Y.; Malekian, R.; Yan, X.; Li, Z. A novel cooperative platform design for coupled USV–UAV systems. IEEE Trans. Ind. Inform. 2019, 15, 4913–4922. [Google Scholar] [CrossRef]
- Huang, T.; Chen, Z.; Gao, W.; Xue, Z.; Liu, Y. A USV-UAV Cooperative Trajectory Planning Algorithm with Hull Dynamic Constraints. Sensors 2023, 23, 1845. [Google Scholar] [CrossRef]
- Li, W.; Ge, Y.; Guan, Z.; Gao, H.; Feng, H. NMPC-based UAV-USV cooperative tracking and landing. J. Frankl. Inst. 2023, 360, 7481–7500. [Google Scholar] [CrossRef]
- Yao, P.; Gao, Z. UAV/USV Cooperative Trajectory Optimization Based on Reinforcement Learning. In Proceedings of the 2022 China Automation Congress (CAC), **amen, China, 25–27 November 2022; pp. 4711–4715. [Google Scholar]
- Wei, W.; Wang, J.; Fang, Z.; Chen, J.; Ren, Y.; Dong, Y. 3U: Joint design of UAV-USV-UUV networks for cooperative target hunting. IEEE Trans. Veh. Technol. 2022, 72, 4085–4090. [Google Scholar] [CrossRef]
- Lewicka, O.; Specht, M.; Stateczny, A.; Specht, C.; Dardanelli, G.; Brčić, D.; Szostak, B.; Halicki, A.; Stateczny, M.; Widźgowski, S. Integration data model of the bathymetric monitoring system for shallow waterbodies using UAV and USV platforms. Remote Sens. 2022, 14, 4075. [Google Scholar] [CrossRef]
- Han, Y.; Ma, W. Automatic Monitoring of Water Pollution based on the Combination of UAV and USV. In Proceedings of the 2021 IEEE 4th International Conference on Electronic Information and Communication Technology (ICEICT), **’an, China, 18–20 August 2021; pp. 420–424. [Google Scholar]
- Wang, Y.; Liu, W.; Liu, J.; Sun, C. Cooperative USV–UAV marine search and rescue with visual navigation and reinforcement learning-based control. ISA Trans. 2023, 137, 222–235. [Google Scholar] [CrossRef]
- Wu, J.; Li, R.; Li, J.; Zou, M.; Huang, Z. Cooperative unmanned surface vehicles and unmanned aerial vehicles platform as a tool for coastal monitoring activities. Ocean. Coast. Manag. 2023, 232, 106421. [Google Scholar] [CrossRef]
- Li, Y.; Li, S.; Zhang, Y.; Zhang, W.; Lu, H. Dynamic Route Planning for a USV-UAV Multi-Robot System in the Rendezvous Task with Obstacles. J. Intell. Robot. Syst. 2023, 107, 52. [Google Scholar] [CrossRef]
- Yu, Y.; Rodríguez-Piñeiro, J.; Shunqin, X.; Tong, Y.; Zhang, J.; Yin, X. Measurement-based propagation channel modeling for communication between a UAV and a USV. In Proceedings of the 2022 16th European Conference on Antennas and Propagation (EuCAP), Madrid, Spain, 27 March–1 April 2022; pp. 01–05. [Google Scholar]
- Shao, Z.; Wu, W.; Wang, Z.; Du, W.; Li, C. Seaships: A large-scale precisely annotated dataset for ship detection. IEEE Trans. Multimed. 2018, 20, 2593–2604. [Google Scholar] [CrossRef]
- Prasad, D.K.; Rajan, D.; Rachmawati, L.; Rajabally, E.; Quek, C. Video processing from electro-optical sensors for object detection and tracking in a maritime environment: A survey. IEEE Trans. Intell. Transp. Syst. 2017, 18, 1993–2016. [Google Scholar] [CrossRef]
- ShipSpotting. Available online: www.shipspotting.com (accessed on 19 July 2023).
- Naderializadeh, N.; Orhan, O.; Nikopour, H.; Talwar, S. Ultra-dense networks in 5G: Interference management via non-orthogonal multiple access and treating interference as noise. In Proceedings of the 2017 IEEE 86th Vehicular Technology Conference (VTC-Fall), Toronto, ON, Canada, 24–27 September 2017; pp. 1–6. [Google Scholar]
- Hartigan, J.A.; Wong, M.A. Algorithm AS 136: A k-means clustering algorithm. J. R. Stat. Society. Ser. C (Appl. Stat.) 1979, 28, 100–108. [Google Scholar] [CrossRef]
- Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. ar**v 2018, ar**v:1804.02767. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Wang, X.; Girshick, R.; Gupta, A.; He, K. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7794–7803. [Google Scholar]
- Radenović, F.; Tolias, G.; Chum, O. Fine-tuning CNN image retrieval with no human annotation. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 41, 1655–1668. [Google Scholar] [CrossRef]
- Schroff, F.; Kalenichenko, D.; Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 815–823. [Google Scholar]
- Woo, S.; Park, J.; Lee, J.-Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar]
- Zhu, X.; Cheng, D.; Zhang, Z.; Lin, S.; Dai, J. An empirical study of spatial attention mechanisms in deep networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6688–6697. [Google Scholar]
- Gu, Y.; Li, C.; **e, J. Attention-aware generalized mean pooling for image retrieval. ar**v 2018, ar**v:1811.00202. [Google Scholar]
- Zheng, L.; Shen, L.; Tian, L.; Wang, S.; Wang, J.; Tian, Q. Scalable person re-identification: A benchmark. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1116–1124. [Google Scholar]
- Liu, X.; Liu, W.; Ma, H.; Fu, H. Large-scale vehicle re-identification in urban surveillance videos. In Proceedings of the 2016 IEEE International Conference on Multimedia and Expo (ICME), Seattle, WA, USA, 11–15 July 2016; pp. 1–6. [Google Scholar]
- Teng, S.; Zhang, S.; Huang, Q.; Sebe, N. Viewpoint and scale consistency reinforcement for UAV vehicle re-identification. Int. J. Comput. Vis. 2021, 129, 719–735. [Google Scholar] [CrossRef]
- Sun, Y.; Cheng, C.; Zhang, Y.; Zhang, C.; Zheng, L.; Wang, Z.; Wei, Y. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 6398–6407. [Google Scholar]
- Chen, H.; Lagadec, B.; Bremond, F. Partition and reunion: A two-branch neural network for vehicle re-identification. In Proceedings of the CVPR Workshops, Long Beach, CA, USA, 16–20 June 2019; pp. 184–192. [Google Scholar]
- Zhang, X.; Zhang, R.; Cao, J.; Gong, D.; You, M.; Shen, C. Part-guided attention learning for vehicle instance retrieval. IEEE Trans. Intell. Transp. Syst. 2020, 23, 3048–3060. [Google Scholar] [CrossRef]
Class 1 | Class 2 | Class 3 | Class 4 | Class 5 |
---|---|---|---|---|
Dataset | Target | ID Volume | Dataset Scale | Angle of View |
---|---|---|---|---|
VesselID-700 | Vessel | 700 | 56,069 | Five angle types with random multi-angle |
VesselReID | Vessel | 733 | 4616 | Random multi-angle |
Market-1501 | Person | 1501 | 32,643 | Six fixed angles |
VeRI-776 | Vehicle | 776 | 51,035 | Sixteen fixed angles |
UAV-VeID | Vehicle | 4601 | 58,767 | Random multi-angle |
Method | Loss Type | Rank-1 (%) | mAP (%) |
---|---|---|---|
Baseline: ResNet50 | CE | 83.10 | 42.33 |
IORNet [27] | CE + Triplet | 85.76 | 56.63 |
Base-GLF-MVFL [23] | CE + TriHard | 84.14 | 48.78 |
GLF-MVFL [23] | CE + O-Quin | 88.72 | 62.19 |
ResNet50 | CE + Triplet | 86.57 | 58.60 |
ResNet50 | CE + TriHard | 87.09 | 60.35 |
ResNet50 + Non-local | CE + TriHard | 88.99 | 64.36 |
ResNet50 + GeM Pooling | CE + TriHard | 89.05 | 64.09 |
FGFN (ResNet50 + Non-local + GeM Pooling) | CE + TriHard | 89.78 | 65.72 |
Target | Dataset | Model | Rank-1 (%) | mAP (%) |
---|---|---|---|---|
Pedestrian | Market1501 | FGFN | 95.3 | 87.9 |
Circle Loss [57] | 96.1 | 87.4 | ||
Vehicle | VeRI-776 | FGFN | 96.0 | 78.3 |
PRN [58] | 94.3 | 74.3 | ||
PGAN [59] | 96.5 | 79.3 | ||
UAV-VeID | FGFN | 80.0 | 85.6 | |
VSCR [56] | 70.6 | -- |
Bounding Box Labeling | Rank-1 (%) | mAP (%) |
---|---|---|
False | 79.01 | 33.80 |
True | 83.10 | 42.33 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Dou, W.; Zhu, L.; Wang, Y.; Wang, S. Research on Key Technology of Ship Re-Identification Based on the USV-UAV Collaboration. Drones 2023, 7, 590. https://doi.org/10.3390/drones7090590
Dou W, Zhu L, Wang Y, Wang S. Research on Key Technology of Ship Re-Identification Based on the USV-UAV Collaboration. Drones. 2023; 7(9):590. https://doi.org/10.3390/drones7090590
Chicago/Turabian StyleDou, Wenhao, Leiming Zhu, Yang Wang, and Shubo Wang. 2023. "Research on Key Technology of Ship Re-Identification Based on the USV-UAV Collaboration" Drones 7, no. 9: 590. https://doi.org/10.3390/drones7090590