Vision-Based Traffic Sign Detection and Recognition Systems: Current Trends and Challenges
Abstract
:1. Introduction
2. Traffic Signs and Research Trends
3. Traffic Sign Database
4. Traffic Sign Detection, Tracking and Classification Methods
4.1. Detection Phase
4.1.1. Color-Based Methods
4.1.2. Shape-Based Methods
4.1.3. Hybrid Methods
4.2. Tracking Phase
4.3. Classification Phase
5. Current Issues and Challenges
6. Conclusions and Suggestion
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Alturki, A.S. Traffic Sign Detection and Recognition Using Adaptive Threshold Segmentation with Fuzzy Neural Network Classification. In Proceedings of the 2018 International Symposium on Networks, Computers and Communications (ISNCC), Rome, Italy, 19–21 June 2018; pp. 1–7. [Google Scholar]
- Satılmış, Y.; Tufan, F.; Şara, M.; Karslı, M.; Eken, S.; Sayar, A. CNN Based Traffic Sign Recognition for Mini Autonomous Vehicles. In Proceedings of the International Conference on Information Systems Architecture and Technology, Nysa, Poland, 16–18 September 2018; pp. 85–94. [Google Scholar]
- Saadna, Y.; Behloul, A.; Mezzoudj, S. Speed limit sign detection and recognition system using SVM and MNIST datasets. Neural Comput. Appl. 2019, 1–11. [Google Scholar] [CrossRef]
- Guo, J.; Lu, J.; Qu, Y.; Li, C. Traffic-Sign Spotting in the Wild via Deep Features. In Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China, 26–30 June 2018; pp. 120–125. [Google Scholar]
- Vokhidov, H.; Hong, H.; Kang, J.; Hoang, T.; Park, K. Recognition of damaged arrow-road markings by visible light camera sensor based on convolutional neural network. Sensors 2016, 16, 2160. [Google Scholar] [CrossRef] [PubMed]
- De la Escalera, A.; Armingol, J.M.; Mata, M. Traffic sign recognition and analysis for intelligent vehicles. Image Vis. Comput. 2003, 21, 247–258. [Google Scholar] [CrossRef] [Green Version]
- Hoang, T.M.; Baek, N.R.; Cho, S.W.; Kim, K.W.; Park, K.R. Road lane detection robust to shadows based on a fuzzy system using a visible light camera sensor. Sensors 2017, 17, 2475. [Google Scholar] [CrossRef] [PubMed]
- Economic Commission for Europe. Convention on Traffic Signs and Signals; Vienna Convention: Vienna, Austria, 1968. [Google Scholar]
- Luo, H.; Yang, Y.; Tong, B.; Wu, F.; Fan, B. Traffic sign recognition using a multi-task convolutional neural network. IEEE Trans. Intell. Transp. Syst. 2018, 19, 1100–1111. [Google Scholar] [CrossRef]
- Houben, S.; Stallkamp, J.; Salmen, J.; Schlipsing, M.; Igel, C. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN), Dallas, TX, USA, 4–9 August 2013; pp. 1–8. [Google Scholar]
- Stallkamp, J.; Schlipsing, M.; Salmen, J.; Igel, C. The German traffic sign recognition benchmark: A multi-class classification competition. In Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN), San Jose, CA, USA, 31 July–5 August 2011; pp. 1453–1460. [Google Scholar]
- Li, J.; Wang, Z. Real-time traffic sign recognition based on efficient CNNs in the wild. IEEE Trans. Intell. Transp. Syst. 2018, 20, 975–984. [Google Scholar] [CrossRef]
- Madani, A.; Yusof, R. Malaysian Traffic Sign Dataset for Traffic Sign Detection and Recognition Systems. J. Telecommun. Electron. Comput. Eng. (JTEC) 2016, 8, 137–143. [Google Scholar]
- Liu, H.; Ran, B. Vision-based stop sign detection and recognition system for intelligent vehicles. Transp. Res. Rec. J. Transp. Res. Board 2001, 1748, 161–166. [Google Scholar] [CrossRef]
- Nandi, D.; Saif, A.S.; Prottoy, P.; Zubair, K.M.; Shubho, S.A. Traffic Sign Detection based on Color Segmentation of Obscure Image Candidates: A Comprehensive Study. Int. J. Mod. Educ. Comput. Sci. 2018, 10, 35. [Google Scholar] [CrossRef]
- Hannan, M.; Hussain, A.; Mohamed, A.; Samad, S.A.; Wahab, D.A. Decision fusion of a multi-sensing embedded system for occupant safety measures. Int. J. Automot. Technol. 2010, 11, 57–65. [Google Scholar] [CrossRef]
- Møgelmose, A.; Trivedi, M.M.; Moeslund, T.B. Vision-Based Traffic Sign Detection and Analysis for Intelligent Driver Assistance Systems: Perspectives and Survey. IEEE Trans. Intell. Transp. Syst. 2012, 13, 1484–1497. [Google Scholar] [CrossRef] [Green Version]
- Shao, F.; Wang, X.; Meng, F.; Rui, T.; Wang, D.; Tang, J. Real-time traffic sign detection and recognition method based on simplified Gabor wavelets and CNNs. Sensors 2018, 18, 3192. [Google Scholar] [CrossRef] [PubMed]
- Zabihi, S.J.; Zabihi, S.M.; Beauchemin, S.S.; Bauer, M.A. Detection and recognition of traffic signs inside the attentional visual field of drivers. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017. [Google Scholar]
- Chang, X.; Yang, Y. Semisupervised feature analysis by mining correlations among multiple tasks. IEEE Trans. Neural Netw. Learn. Syst. 2017, 28, 2294–2305. [Google Scholar] [CrossRef] [PubMed]
- Yuan, Y.; ** for deep learning based traffic sign recognition. Sensors 2018, 18, 3776. [Google Scholar] [CrossRef] [PubMed]
- Laguna, R.; Barrientos, R.; Blázquez, L.F.; Miguel, L.J. Traffic sign recognition application based on image processing techniques. IFAC Proc. Vol. 2014, 47, 104–109. [Google Scholar] [CrossRef]
Country | US | Japan | Pakistan | Ethiopia | Libya | New Guinea |
---|---|---|---|---|---|---|
Sign | | | | | | |
Dataset | Country | Classes | TS Scenes | TS Images | Image Size (px) | Sign Size (px) | Include Videos |
---|---|---|---|---|---|---|---|
GTSDRB (2012 and 2013) | Germany | 43 | 9000 | 39,209 (training), 12,630 (testing) | 15 × 15 to 250 × 250 | 15 × 15 to 250 × 250 | No |
KULD (2009) | Belgium | 100+ | 9006 | 13,444 | 1628 × 1236 | 100 × 100 to 1628 × 1236 | Yes, 4 tracks |
STSD (2011) | Sweden | 7 | 20,000 | 3488 | 1280 × 960 | 3 × 5 to 263 × 248 | No |
RUGD (2003) | The Netherlands | 3 | 48 | 48 | 360 × 270 | N/A | No |
Stereopolis (2010) | France | 10 | 847 | 251 | 1920 × 1080 | 25 × 25 to 204 × 159 | No |
LISAD (2012) | US | 49 | 6610 | 7855 | 640 × 480 to 1024 × 52 | 6 × 6 to 167 × 168 | All annotations |
UKOD (2012) | UK | 100+ | 43,509 | 1200 (synthetic) | 648 × 480 | 24 × 24 | No |
RTSD (2013) | Russia | 140 | N/A | 80,000+ (synthetic) | 1280 × 720 | 30 × 30 | No |
Techniques | Paper | Segmentation Methods | Advantages | Sign Type | No. of Test Images | Test Image Type |
---|---|---|---|---|---|---|
Color Thresholding Segmentation | [37] | RGB color segmentation | Simple | Any color | 2000 | N/A |
[38] | RGB color segmentation with enhancement of color | Fast and high detection rate | Red, blue, yellow | 135 | Video data | |
HSI/HSV Transform | [40] | HSI thresholding with addition for white signs | Segments adversely illuminated signs | Any color | N/A | High-res |
[33] | HSI color-based segmentation | Simple and fast | Any color | N/A | N/A | |
[41] | RGB to HSI transformation | Segments adversely illuminated signs | Any color | N/A | Low-res | |
[42] | RGB to HSI transformation | N/A | Red | N/A | Low-res | |
[43] | RGB to HSI transformation | N/A | Any color | 3028 | Low-res | |
[44] | HSI color-based segmentation | Simple and high accuracy rate | Red, blue | N/A | Video data | |
[45] | HSI color-based segmentation | Simple and real time application | Any color | 632 | High-res | |
Region Growing | [48] | Started with seed and expand to group pixels with similar affinity | N/A | N/A | N/A | N/A |
[47] | N/A | N/A | High-res | |||
Color Indexing | [50] | Comparison of two any-color images is done by comparing their color histogram | Straightforward, fast method | Any color | N/A | Low-res |
[49] | Any color | N/A | N/A | |||
Dynamic Pixel Aggregation | [52] | Dynamic threshold in pixel aggregation on HSV color space | Hue instability reduced | Any color | 620 | Low-res |
CIECAM97 Model | [54] | RGB to CIE XYZ transformation, then to LCH space using CIECAM97 model | Invariant in different lighting conditions | Red, blue | N/A | N/A |
YCbCr Color Space | [55] | RGB to YCbCr transformation then dynamic thresholding is performed in Cr component to extract red object | Simple and high accuracy | Red | 193 | N/A |
[56] | High accuracy less processing time | Any color | N/A | Low-res |
Technique | Paper | Overall Process | Recognition Feature | Advantages | Sign Type | No. of Test Image | Test Image Type |
---|---|---|---|---|---|---|---|
Hough Transform | [77] | Each pixel of edge image votes for the object center at object boundary | N/A | Invariant to in-plane rotation and viewing angle | Octagon, square, triangle | 45 | Low-res |
[78] | AdaBoost | High accuracy | Any sign | N/A | Low-res | ||
[79] | N/A | Robustness to illumination, scale, pose, viewpoint change and even partial occlusion | Red (circular), blue (square) | 500+ | Low-res | ||
[80] | N/A | Reducing memory consumption and increasing utilization Hough-based SVM | Any sign | 3000 | High-res | ||
[81] | N/A | Robustness | Red (circular) | N/A | 768 × 580 | ||
[59] | Random Forest | Improve efficiency of K-d tree, random forest and SVM | Triangular and circular | 14,763 | 752 × 480 px | ||
[82] | SIFT and SURF based MLP | Applying another state refinement | Red circular | N/A | Video data | ||
Similarity Detection | [52] | Computes a region and sets binary samples for representing each traffic sign shape. | NN | Straight forward method | Any color | 620 | Low-res |
DTM | [61] | Capturing object shape by template hierarchy. | RBF Network | Detects objects of arbitrary shape | Circular and triangular | 1000 | 360 × 288 px |
Edge Detection Feature | [63] | A set of connected curves is found which indicates the boundaries of objects within the image. | Geometric matching | Invariant in translation, rotation and scaling | Any color | 1000 | 640 × 480 |
[64] | Normalized cross correlation | Reliability and high accuracy in real time | Speed limit sign | N/A | 320 × 240 px video data | ||
[65] | N/A | Improved accuracy by training negative sample | Red (circular) | 3907 | Low-res | ||
[66] | N/A | Invariant in noise and lighting | Triangle, circular | 847 | High-res | ||
[67] | CDT | Invariant in noise and illumination | Red, blue, yellow | ||||
Edges with Haar-like Features | [69] | Sums three pixel intensities and calculates the difference of sums by Haar-like features | CDT | Smoother and noise invariant | Rectangular, any color | Video data | |
[70] | SVM | Fast method | Circular, triangular upside-down, rectangle and diamond | 640 × 480 px video data |
Technique | Paper | Advantages | Performance |
---|---|---|---|
Kalman Filter | [82] | For avoiding incorrect assignment, rule-based approach utilizing combined distance direction difference is used. | N/A |
[89] | Takes less time in tracking and verifying | Using 320 × 240 pixel images, takes 0.1 s to 0.2 s. | |
[88] | Used stereo parameters to reduce the error of stereo measurement | N/A | |
Advanced Kalman Filter | [85] | Fast and advanced method, high detection and tracking rate | Using 400 × 300 pixel images, can process 3.26 frames per second. |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time |
---|---|---|---|---|---|---|---|
[90] | RGB to HSV then contrast stretching | Fast and straight forward method | N/A | N/A | N/A | <95% | N/A |
[91] | N/A | N/A | N/A | 100 | 90.9% | N/A |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time | Dataset |
---|---|---|---|---|---|---|---|---|
[94] | HOG based SVM | Used GTSRB and ETH 80 dataset and compared | 90.9% | N/A | 12,569 | 90.46% | 17.9 ms | GTSRB and ETH 80 |
[95,96] | Used Gaussian weighting in HOG to improve performance by 15% | 90% | N/A | 12,569 | 97.2% | 17.9 ms | Own created | |
[92] | MSER based HOG | Eliminating hand labeled database, robust to various lighting and illumination | 83.3% | 0.85 | 640 × 480 px video data | 87.72% | N/A | Own created |
[97] | HOG | Remove false alarm up to 94% | N/A | N/A | 12,569 | 92.7% | 17.9 ms | Own created |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time |
---|---|---|---|---|---|---|---|
[98,99] | Genetic Algorithm | Unaffected by illumination problem | N/A | N/A | Video data | N/A | N/A |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time | Dataset |
---|---|---|---|---|---|---|---|---|
[56] | YCbCr and normalized cross correlation | Robustness and adaptability | 0.96 | 0.08 | 640 × 480 px video data | 97.6% | 0.2 s | Own created |
[101] | N/A | Flexibility and high accuracy | N/A | N/A | N/A | 98.52–99.46% | N/A | Own created |
[106] | Adaptive shape analysis | Invariant in illumination | N/A | N/A | 220 | 95.4% | 0.6 s | Own created |
[107] | NN | Robustness | N/A | N/A | 467 | N/A | N/A | Own created |
[108] | Bimodal binarization and thresholding | Compared TM and NN elaborately | 0.96 | 0.08 | 640 × 480 px video data | 97.6% | 0.2 s | Own created |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time | Dataset |
---|---|---|---|---|---|---|---|---|
[115] | Object bounding box prediction | Predicting position and precise boundary simultaneously | >0.88 mPA | <3 pixels | 3,719 | 91.95% | N/A | GTSDB |
[120] | YCbCr model | High accuracy and speed | N/A | N/A | Video data | 98.6% | N/A | Own created |
[111] | Color space thresholding | Implementing detection and classification | 90.2% | 2.4% | 20,000 | 95% | N/A | GTSRB |
[121] | SVM | Robust against illumination changes | N/A | N/A | Video data | 97.9% | N/A | Own created |
[117] | Scanning window with a Haar cascade detector | Enhanced detection capability with good time performance | N/A | N/A | 16,630 | 99.36% | N/A | GTSRB |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time | Dataset |
---|---|---|---|---|---|---|---|---|
[123] | Sobel edge detection | Comparison of SVM and AdaBoost | N/A | 0.25 | N/A | 92% | N/A | Own created |
[124] | AdaBoost | Fast | N/A | N/A | 200 | >90% | 50 ms | Own created |
[125] | AdaBoost | Invariant in speed, illumination and viewing angle | 92.47% | 0% | 350 | 94% | 51.86 ms | Own created |
[126] | AdaBoost and CHT | Real-time and robust system with efficient SLS detection and recognition | 0.97 | 0.26 | 1850 | 94.5% | 30–40 ms | Own created |
[127] | Haar-like method | Reliability and accuracy | 0.9 | 0.4 | 200 | 92.7% | 50 ms | Own created |
Ref | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time | Dataset |
---|---|---|---|---|---|---|---|---|
[44] | DtBs and SVM | Fast, high accuracy | N/A | N/A | Video data | 92.3% | N/A | GRAM |
[55] | Gabor Filter | Simple and high accuracy | N/A | N/A | 58 | 93.1% | N/A | Own created |
[130] | CIELab and Ramer–Douglas–Peucker algorithm | Illumination proof and high accuracy | N/A | N/A | 405 | 97% | N/A | Own created |
[131] | RGB to HSI then shape analysis | Less processing time | N/A | N/A | 92.6% | Avg. 5.67 s | Own created | |
[88] | Hough transform | Reliability and accuracy | N/A | N/A | Video data | Avg. 92.3% | 35 ms | Own created |
[132] | RGB to HIS then shape localization | Reduce the memory space and time for testing new sample | N/A | N/A | N/A | 95% | N/A | Own created |
[133] | MSER | Invariant in illumination and lighting condition | 0.97 | 0.85 | 43,509 | 89.2% | N/A | Own created |
[134] | HSI and edge detection | Less processing time | N/A | N/A | Video data | N/A | N/A | Own created |
[135] | RGB to HSI | Identify the optimal image attributes | 0.867 | 0.12 | 650 | 86.7% | 0.125 s | Own created |
[136] | Edge Adaptive Gabor Filtering | Reliability and Robustness | 85.93% | 11.62% | 387 | 95.8%. | 3.5–5 ms | Own created |
Ref | Method | Detection Feature | Advantages | True Positive Rate | False Positive Rate | No. of Test Images | Overall Accuracy | Time | Dataset |
---|---|---|---|---|---|---|---|---|---|
[138] | SIFT matching | N/A | Effective in recognizing low light and damaged signs | N/A | N/A | 60 | N/A | N/A | Own created |
[34] | Fringe-adjusted joint Transform Correlation | Color Feature Extraction using Gabor Filter | Excellent discrimination ability between object and non-object | 783 | 217 | 587 | N/A | N/A | Own created |
[139] | Principal Component Analysis | HSV, CIECAM97 and PCA | High accuracy rate | N/A | N/A | N/A | 99.2% | 2.5 s | Own created |
[140] | Improved Fast Radial Symmetry and Pictogram Distribution Histogram based SVM | RGB to LaB color space then IFRS detection | High accuracy rate | N/A | N/A | 300 | 96.93% | N/A | Own created |
[144] | Infrastructures of vehicles | N/A | Eliminating possibility of false positive rate because of ID coding | N/A | N/A | Video data | N/A. | N/A | Own created |
[145] | FCM and Content Based Image Recorder | Fuzzy c means (FCM) | Effective in real time application | N/A | N/A | Video data | <80% | N/A | Own created |
[141] | Template matching and 3D reconstruction algorithm | N/A | Very effective in recognizing damaged or occulted road signs | In 3D, 54 out of 63 | In 3D, 6 out of 63 and 3 signs were missing | 4800 | N/A | N/A | Own created |
[142] | Low Rank Matrix Recovery (LRMR) | N/A | Fast computation and parallel execution | N/A | N/A | 40,000 | 97.51% | >0.2 | GTSRB |
[143] | Karhunen–Loeve Transform and MLP | Oriented gradient maps | Invariant in illumination an different lighting condition | N/A | N/A | 12,600 | 95.9% | 0.0054 s/image | GTSRB |
[35] | Self-Organizing Map | N/A | Fast and accurate | N/A | N/A | N/A | <99% | N/A | Own created |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wali, S.B.; Abdullah, M.A.; Hannan, M.A.; Hussain, A.; Samad, S.A.; Ker, P.J.; Mansor, M.B. Vision-Based Traffic Sign Detection and Recognition Systems: Current Trends and Challenges. Sensors 2019, 19, 2093. https://doi.org/10.3390/s19092093
Wali SB, Abdullah MA, Hannan MA, Hussain A, Samad SA, Ker PJ, Mansor MB. Vision-Based Traffic Sign Detection and Recognition Systems: Current Trends and Challenges. Sensors. 2019; 19(9):2093. https://doi.org/10.3390/s19092093
Chicago/Turabian StyleWali, Safat B., Majid A. Abdullah, Mahammad A. Hannan, Aini Hussain, Salina A. Samad, Pin J. Ker, and Muhamad Bin Mansor. 2019. "Vision-Based Traffic Sign Detection and Recognition Systems: Current Trends and Challenges" Sensors 19, no. 9: 2093. https://doi.org/10.3390/s19092093