JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2019, Vol. 54 ›› Issue (3): 28-37.doi: 10.6040/j.issn.1671-9352.0.2018.601

•   • Previous Articles     Next Articles

A survey of visual saliency and salient object detection methods

Jia XU1(),Peng JIANG2,*()   

  1. 1. Xinjiang Irtysh River Basin Development & Construction Administrative Bureau, Urumqi 830002, Xinjiang, China
    2. School of Qilu Transportation, Shandong University, Jinan 250002, Shandong, China
  • Received:2018-10-18 Online:2019-03-20 Published:2019-03-19
  • Contact: Peng JIANG E-mail:xj8476@qq.com;sdujump@gmail.com

RichHTML

24

PDF (PC)

419

Abstract:

Saliency detection aims to locate the most attractive areas in image or video data, as the basic task of computer vision field, has receive intensive attentions. Many methods have been proposed recently, these methods usually can be classified into two branches: visual saliency detection and salient object detection. Tough the methods of two branches usually share the similar features and even the frameworks, their performances on datasets of different branch have large gap, seldom works have compared and analyzed them. In this work, we will provide a detailed review and analysis of main works in two branches, including their mechanism, metrics and datasets. Besides, in this work, we summarized approaches to boost visual saliency detection methods for the task of salient object detection. With these approaches, visual saliency detection methods can be applied to detect the salient object and show superior performance that even outperform some specialized state-of-the-art salient object detection methods, thus reduce the performance inconsistence in different specialized datasets.

Key words: saliency detection, visual saliency detection, salient object detection

CLC Number: 

  • TP309

Fig.1

Results of saliency detection methods in different branches"

Fig.2

The main components of Itti saliency detection method[7]"

Table 1

Common feature contrast metrics D and feature contrast scope R"

参数 可选项
D KL 散度  L1 范数  L2 范数  χ2(卡方距离)  (内积)
R 局部 部分 全局

Fig.3

Saliency computation direction[49]"

Table 2

Saliency features and their taxonomy"

组 类 形式 特征 方法(见文献)
节点 像素、超像素、区域 位置、大小、形状 [15, 19, 23, 24, 38, 45]
颜色 RGB、LAB、HSV等 原值、均值、方差、直方图、分布 [3-5, 7-9, 12-24, 26, 27, 29-46]
边缘 方向、梯度、HOG、聚焦度 [4, 7-10, 12, 22, 35, 45]
低层特征 角点 LBP, SIFT, Convex Hull [11, 21, 35, 39]
滤波器 Gabor, LM, Local steering 原值、直方图 [5, 8, 10, 21, 33]
先验 中心、背景 位置、连接度Connectivity [3, 8, 20, 21, 24, 25, 27, 29, 31, 36-38, 45-48]
其他 ICA, PCA, SVD 系数 [1, 2, 12, 23, 27, 29]
高层特征 先验 位置、颜色、语义 中心、颜色冷暖度、人脸 [8, 15, 19, 22, 24, 25, 31, 33]
其他 深度神经网络 特征响应 [39]等

Table 3

Datasets with their attributes and difficulty ranking"

数据集 年份 任务 图像数量 物体个数 标注形式 分辨率 标注人数 难易程度
Toronto[1] 2005 V 120 多个 眼动数据 [681, 581] 20 1
MIT[8] 2009 V 1 003 多个 眼动数据 [1 024, 768] 15 2
MSRA1000[14] 2007 O 1 000 单个 像素级标注 [400, 300] 1 1
MSRA10K[16] 2013 O 10 000 单个 像素级标注 [400, 300] 1 3
ECSSD[36] 2013 O 1 000 多个 像素级标注 [X, 400] 5 2
DUT-OMRON[37] 2013 V/O 5 168 多个 眼动数据、包围盒、像素级标注 [X, 400] 5 4

Fig.4

Transform between visual saliency detectionresults and salient object detection results"

Fig.5

The diffusion pipeline"

Fig.6

Evaluation results by different metrics on MSRA10K dataset"

Fig.7

Evaluation results by different metrics on DUT-OMRON dataset"

Fig.8

PR curves of six salient objection detectionmethods on MSRA10K and DUT-OMRON dataset"

Fig.9

Comparison of three state-of-the-art salient objection detection and four boosted visual saliency detection methods"

1 BRUCE N, TSOTSOS J. Saliency based on information maximization[C]// Proceedings of 18th Advances in Neural Information Processing Systems (NIPS 2005). Vancouver: MIT Press, 2005: 155-162.
2 BRUCE N , TSOTSOS J . Saliency, attention, and visual search: an information theoretic approach[J]. Journal of Vision, 2009, 3 (9): 5.1- 24.
3 FRINTROP S, WERNER T, GARCIA M. Traditional saliency reloaded: a good old model in new shape[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015). Boston: MA, 2015: 82-90.
4 GAO D , VASCONCELOS N . Decision-theoretic saliency: computational principles, biological plausibility, and implications for neurophysiology and psychophysics[J]. Neural Computation, 2009, 21 (1): 239- 271.
doi: 10.1162/neco.2009.11-06-391
5 HAREL J, KOCH C, PERONA P. Graph-based visual saliency[C]// Proceedings of 19th Advances in Neural Information Processing Systems (NIPS 2006). Canada: MIT Press, 2006: 545-552.
6 HOU X, ZHANG L. Saliency detection: a spectral residual approach[C]// Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007). Minneapolis: MN, 2007: 1-8.
7 ITTI L , KOCH C , NIEBUR E . A model of saliency-based visual attention for rapid scene analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20 (11): 1254- 1259.
doi: 10.1109/34.730558
8 JUDD T, EHINGER K, DURAND F, et al. Learning to predict where humans look[C]// Proceedings of 2009 International Conference on Computer Vision (ICCV 2009). Kyoto: IEEE, 2009: 2106-2113.
9 KLEIN D, FRINTROP S. Center-surround divergence of feature statistics for salient object detection[C]// Proceedings of 2011 International Conference on Computer Vision (ICCV 2011). Barcelona: IEEE, 2011: 2214-2219.
10 SEO H , MILANFAR P . Static and space-time visual saliency detection by self-resemblance[J]. Journal of Vision, 2009, 9 (12): 15.1- 27.
11 YANG J, YANG M. Top-down visual saliency via joint crf and dictionary learning[C]// Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012). Rhode Island: IEEE, 2012: 2296-2303.
12 ZHANG L , TONG M H , MARKS T K , et al. Sun: a Bayesian framework for saliency using natural statistics[J]. Journal of Vision, 2008, 8 (7): 32.1- 20.
13 ACHANTA R, ESTRADA F, WILS P, et al. Salient region detection and segmentation[C]// Proceedings of 2008 International Conference on Computer Vision Systems (ICVS 2008). Berlin: Springer, 2008: 66-75.
14 ACHANTA R, HEMAMI S, ESTRADA F, et al. Frequency-tuned salient region detection[C]// Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009). Miami: IEEE, 2009: 1597-1604.
15 CHANG K, LIU T, CHEN H, et al. Fusing generic objectness and visual saliency for salient object detection[C]// Proceedings of 2011 International Conference on Computer Vision (ICCV 2011). Barcelona: IEEE, 2011: 914-921.
16 CHENG M , MITRA N J , HUANG X , et al. Global contrast based salient region detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37 (3): 569- 582.
doi: 10.1109/TPAMI.2014.2345401
17 CHENG M, WARRELL J, LIN W, et al. Efficient salient region detection with soft image abstraction[C]// Proceedings of 2013 International Conference on Computer Vision (ICCV 2013). Sydney: IEEE, 2013: 1529-1536.
18 FELZENSZWALB P F , HUTTENLOCHER D P . Efficient graph-based image segmentation[J]. International Journal of Computer Vision, 2004, 59 (2): 167- 181.
doi: 10.1023/B:VISI.0000022288.19776.77
19 GOFERMAN S , ZELNIK-MANOR L , TAL A . Context-aware saliency detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34 (10): 1915- 1926.
doi: 10.1109/TPAMI.2011.272
20 JIANG B, ZHANG L, LU H, et al. Saliency detection via absorbing markov chain[C]// Proceedings of 2013 International Conference on Computer Vision (ICCV 2013). Sydney: IEEE, 2013: 1665-1672.
21 JIANG H, WANG J, YUAN Z, et al. Salient object detection: a discriminative regional feature integration approach[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013). Portland: IEEE, 2013: 2083-2090.
22 JIANG P, LING H, YU J, et al. Salient region detection by ufo: uniqueness, focusness and objectness[C]// Proceedings of 2013 International Conference on Computer Vision (ICCV 2013). Syndney: IEEE, 2013: 1976-1983.
23 KIM J, HAN D, TAI Y W, et al. Salient region detection via high-dimensional color transform[C]// Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014). Columbus: IEEE, 2014: 883-890.
24 LI X, LU H, ZHANG L, et al. Saliency detection via dense and sparse reconstruction[C]// Proceedings of 2013 International Conference on Computer Vision (ICCV 2013). Sydney: IEEE, 2013: 2976-2983.
25 LIU R, CAO J, LIN Z, et al. Adaptive partial differential equation learning for visual saliency detection[C]// Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014). Columbus: IEEE, 2014: 3866-3873.
26 LIU T , YUAN Z , SUN J , et al. Learning to detect a salient object[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33 (2): 353- 367.
doi: 10.1109/TPAMI.2010.70
27 LU S, MAHADEVAN V, VASCONCELOS N. Learning optimal seeds for diffusion-based salient object detection[C]// Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014). Columbus: IEEE, 2014: 2790-2797.
28 MAI L, NIU Y, LIU F. Saliency aggregation: a data-driven approach[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013). Portland: IEEE, 2013: 1131-1138.
29 MARGOLIN R, TAL A, ZELNIK-MANOR L. What makes a patch distinct?[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013). Portland: IEEE, 2013: 1139-1146.
30 PERAZZI F, KRAHENBUHL P, PRITCH Y, et al. Saliency filters: contrast based filtering for salient region detection[C]// Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012). Providence: IEEE, 2012: 733-740.
31 QIN Y, LU H, XU Y, et al. Saliency detection via cellular automata[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015). Boston: IEEE, 2015: 110-119.
32 REN Z, HU Y, CHIA L T, et al. Improved saliency detection based on superpixel clustering and saliency propagation[C]// Proceedings of 18th ACM international conference on Multimedia (MM2010). Firenze: ACM, 2010: 1099-1102.
33 SHEN X, WU Y. A unified approach to salient object detection via low rank matrix recovery[C]// Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012). Providence: IEEE, 2012: 853-860.
34 SIVA P, RUSSELL C, XIANG T, et al. Looking beyond the image: unsupervised learning for object saliency and detection[C]// Procceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013). Portland: IEEE, 2013: 3238-3245.
35 XIE Y , LU H , YANG M . Bayesian saliency via low and mid level cues[J]. IEEE Transactions on Image Processing, 2013, 22 (5): 1689- 1698.
doi: 10.1109/TIP.2012.2216276
36 YAN Q, XU L, SHI J, et al. Hierarchical saliency detection[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013). Portland: IEEE, 2013: 1155-1162.
37 YANG C, ZHANG L, LU H, et al. Saliency detection via graph-based manifold ranking[C]// Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013). Portland: IEEE, 2013: 3166-3173.
38 ZHU W, LIANG S, WEI Y, et al. Saliency optimization from robust background detection[C]// Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014). Columbus: IEEE, 2014: 2814-2821.
39 ZOU W, KOMODAKIS N. Harf: hierarchy-associated rich features for salient object detection[C]// Proceedings of 2015 International Conference on Computer Vision (ICCV 2015). Santiago: IEEE, 2015: 406-414.
40 王军, 吴泽民, 杨巍, 等. 基于稀疏恢复与优化的显著性目标检测算法[J]. 计算机科学, 2018, 45 (8): 258- 263.
WANG Jun , WU Zemin , YANG Wei , et al. Salient object detection algorithm based on sparse recovery and optimization[J]. Computer Science, 2018, 45 (8): 258- 263.
41 孙赫赫, 尚晓清, 王冲. 基于区域对比的图像显著性检测方法[J]. 计算机工程与应用, 2018, 54 (10): 192- 195.
doi: 10.3778/j.issn.1002-8331.1612-0439
SUN Hehe , SHANG Xiaoqing , WANG Chong . Method of saliency detection based on region contrast[J]. Computer Eninerring and Applications, 2018, 54 (10): 192- 195.
doi: 10.3778/j.issn.1002-8331.1612-0439
42 刘亚宁, 吴清, 魏雪. 基于流行排序的前景背景显著性检测算法[J]. 科学技术与工程, 2018, 18 (18): 74- 81.
doi: 10.3969/j.issn.1671-1815.2018.18.011
LIU Yaning , WU Qing , WEI Xue . Saliency detection combined foreground with background based on manifold ranking[J]. Science Technology and Engineering, 2018, 18 (18): 74- 81.
doi: 10.3969/j.issn.1671-1815.2018.18.011
43 梁大川, 李静, 刘赛. 基于图和稀疏主成分分析的多目标显著性检测[J]. 计算机研究与发展, 2018, 55 (5): 1078- 1089.
LIANG Dachuan , LI Jing , LIU Sai . Multiple object saliency detection based on graph and sparse principal component analysis[J]. Journal of Computer Research and Development, 2018, 55 (5): 1078- 1089.
44 侯向丹, 刘晓曼, 刘洪普, 等. 基于多尺度多特征的显著性区域检测[J]. 河北工业大学学报, 2017, 1 (6): 26- 31.
HOU Xiangdan , LIU Xiaoman , LIU Hongpu , et al. Salient region detection via multi-scale and multi-feature[J]. Journal of Hebei University of Technology, 2017, 1 (6): 26- 31.
45 ERDEM E , ERDEM A . Visual saliency estimation by non-linearly integrating features using region covariance[J]. Journal of Vision, 2013, 13 (4): 11.
doi: 10.1167/13.4.11
46 JIANG P, VASCONCELOS N, PENG J. Generic promotion of diffusion-based salient object detection[C]// Proceedings of 2015 International Conference on Computer Vision (ICCV 2015). Santiago: IEEE, 2015: 217-225.
47 ZHANG J, SCLAROFF S. Saliency detection: a boolean map approach[C]// Proceedings of 2013 International Conference on Computer Vision (ICCV 2013). Sydney: IEEE, 2013: 153-160.
48 ZHANG J, SCLAROFF S, LIN Z, et al. Minimum barrier salient object detection at 80 fps[C]// Proceedings of 2015 International Conference on Computer Vision (ICCV 2015). Santiago: IEEE, 2015: 1404-1412.
49 蒋鹏.图像内容显著性检测的理论和方法研究[D].济南:山东大学, 2016.
JIANG Peng. Research on theories and methodologies for saliency detection from images[D]. Jinan: Shandong Unversity, 2016.
[1] Fu-sheng WU,Huan-guo ZHANG,Ming-tao NI,Jun WANG. Security analysis model of behavior based on cryptographic protocols implement at source code level [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(3): 18-27.
[2] Xiao-jie XIE,Ying LIANG,Xiang-xiang DONG. Sensitive attribute iterative inference method for social network users [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(3): 10-17, 27.
[3] Tian-tian CHANG,Xing-shu CHEN,Yong-gang LUO,Xiao LAN. Security domain-based data isolation protection framework for Hive [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(3): 1-9.
[4] Ze-nan WU,Li-qin TIAN,Zhi-gang WANG. A user behavior trust evaluation combined with sliding window and recommended trust [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(1): 53-59.
[5] Yao-yao DU,Ping PAN,Jin-hua LINGHU. Evaluation method of information system grade protection based on DIT [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(1): 47-52.
[6] Zhao-xia WU,Jia-qi WANG. Wireless single spectrum secure auction algorithm [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(11): 51-55.
[7] JIAO Hong-ru, QIN Jing. Quantum secret sharing scheme realizing all hyperstar quantum access structure [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(9): 62-68.
[8] YAN Yan, HAO Xiao-hong. Differential privacy partitioning algorithm based on adaptive density grids [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(9): 12-22.
[9] ZHANG Jian-biao, LI Zhi-gang, LIU Guo-jie, WANG Chao, WANG Wei. Process active dynamic measurement method for Windows environment [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(7): 46-50.
[10] LIU Zheng, NIU Fang-lin, QIAN Da-xing, CAI Xi-biao, GUO Ying. Design of anti-eavesdropping code based on fountain codes [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(7): 60-64.
[11] XU Li-dong, WANG Ming-qiang. A meet-in-the-middle attack on 10-round AES-128 [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(7): 39-45.
[12] CUI Zhao-yang, SUN Jia-qi, XU Song-yan, JIANG Xin. A secure clustering algorithm of Ad Hoc network for colony UAVs [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(7): 51-59.
[13] LIU Ming-ming, ZHANG Min-qing, LIU Jia, GAO Pei-xian. Steganalysis method based on shallow convolution neural network [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(3): 63-70.
[14] RUAN Shu-hua, WENG Jun-hao, MAO Hui, CHEN Xue-lian. Metric model for cloud computing security risk assessment [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(3): 71-76.
[15] KANG Hai-yan, HUANG Yu-xuan, CHEN Chu-qiao. Enhancing privacy for geographic information based on video analysis [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(1): 19-29.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] YANG Yong-wei1, 2, HE Peng-fei2, LI Yi-jun2,3. On strict filters of BL-algebras#br#[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 63 -67 .
[2] ZHAO Tong-xin1, LIU Lin-de1*, ZHANG Li1, PAN Cheng-chen2, JIA Xing-jun1. Pollinators and pollen polymorphism of  Wisteria sinensis (Sims) Sweet[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 1 -5 .
[3] GUO Lan-lan1,2, GENG Jie1, SHI Shuo1,3, YUAN Fei1, LEI Li1, DU Guang-sheng1*. Computing research of the water hammer pressure in the process of #br# the variable speed closure of valve based on UDF method[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 27 -30 .
[4] LI Min1,2, LI Qi-qiang1. Observer-based sliding mode control of uncertain singular time-delay systems#br#[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 37 -42 .
[5] HAN Ya-fei, YI Wen-hui, WANG Wen-bo, WANG Yan-ping, WANG Hua-tian*. Soil bacteria diversity in continuous cropping poplar plantation#br# by high throughput sequencing[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(05): 1 -6 .
[6] WANG Kai-rong, GAO Pei-ting. Two mixed conjugate gradient methods based on DY[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2016, 51(6): 16 -23 .
[7] MA Yuan-yuan, MENG Hui-li, XU Jiu-cheng, ZHU Ma. Normal distribution of lattice close-degree based on granular computing[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(08): 107 -110 .
[8] XU Jun-feng. On the growth of the meromorphic solutions of complex algebraic differential equations[J]. J4, 2010, 45(6): 91 -93 .
[9] PEI Sheng-yu，ZHOU Yong-quan. A mult-objective particle swarm optimization algorithm based on  the  chaotic mutation[J]. J4, 2010, 45(7): 18 -23 .
[10] DU Ji-xiang1,2, YU Qing1, ZHAI Chuan-ming1. Age estimation of facial images based on non-negative matrix factorization with sparseness constraints[J]. J4, 2010, 45(7): 65 -69 .
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd