JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2022, Vol. 57 ›› Issue (9): 25-32.doi: 10.6040/j.issn.1671-9352.2.2021.123

Previous Articles     Next Articles

Research and implementation of real-time processing model of high-speed network stream

SU Zi-peng1,2, YUAN Lei3, LIU Peng3, CHEN Xing-shu1,2*, LUO Yong-gang1,2, CHEN Liang-guo1,2   

  1. 1. School of Cyber Science and Engineering, Sichuan University, Chengdu 610207, Sichuan, China;
    2. Cybersecurity Research Institute, Sichuan University, Chengdu 610207, Sichuan, China;
    3. China Mobile(Chengdu)Information Communication Technology Co., Ltd., Chengdu 611730, Sichuan, China
  • Published:2022-09-15

PDF (PC)

290

Abstract: With the increasing scale of network traffic, it becomes more and more difficult to analyze network traffic in real time, efficiently and accurately. Aiming at the problem that the current network flow processing is not real-time enough and it is difficult to adapt to different network traffic scale, this paper studies and implements a real-time network flow restructuring architecture with multiple traffic scenarios,Designed to be able to adapt to different network traffic scenarios during network stream processing, And effectively improve system resource utilization. The architecture is based on DPDK to achieve real-time collection of high-speed traffic, and the process of stream reorganization is divided into stages, to construct operating topologies for different scenarios, and to achieve dynamic allocation of system resources. The system was implemented in a stand-alone environment and tested in a 10Gbps network environment. The results show that the architecture is superior to similar methods in throughput and resource utilization.

Key words: network security, flow record, network flow feature extraction, DPDK

CLC Number: 

  • TP393
[1] GU G, PORRAS P A, YEGNESWARAN V, et al. Bothunter: detecting malware infection through ids-driven dialog correlation[C] //Proceedings of the 16th USENIX Security Symposium. Boston: USENIX, 2007, 7: 1-16.
[2] KARAGIANNIS T, BROIDO A, FALOUTSOS M, et al. Transport layer identification of P2P traffic[C] //Proceedings of the 4th ACM SIGCOMM Conference on Internet Measurement. Taormin: ACM, 2004: 121-134.
[3] PAPADOGIANNAKIS A, POLYCHRONAKIS M, MARKATOS E P. Stream-oriented network traffic capture and analysis for high-speed networks[J]. IEEE Journal on Selected Areas in Communications, 2014, 32(10): 1849-1863.
[4] DERI L, SPA N E T. nProbe: an open source netflow probe for gigabit networks[C] //TERENA Networking Conference. Zagreb: CARNet, 2003: 1-4.
[5] INACIO C M, TRAMMELL B. Yaf: yet another flowmeter[C] //Proceedings of LISA10: 24th Large Installation System Administration Conference. San Diego: SEI, 2010: 107.
[6] CLAISE B. Cisco systems NetFlow services export version 9[EB/OL].[2021-06-06]. RFC 3954, 2004.4. http://www.doc88.com/p-9502720205086.html.
[7] WANG M, LI B, LI Z. sFlow: towards resource-efficient and agile service federation in service overlay networks[C] //24th International Conference on Distributed Computing Systems, 2004. Proceedings. Tokyo:IEEE, 2004: 628-635.
[8] SANTIAGO DEL RIO P M, ROSSI D, GRINGOLI F, et al. Wire-speed statistical classification of network traffic on commodity hardware[C] //Proceedings of the 2012 Internet Measurement Conference. Boston: ACM, 2012: 65-72.
[9] ZHANG T, LINGUAGLOSSA L, GALLO M, et al. FloWatcher-DPDK: lightweight line-rate flow-level monitoring in software[J]. IEEE Transactions on Network and Service Management, 2019, 16(3): 1143-1156.
[10] EMMERICH P, PUDELKO M, GALLENMÜLLER S, et al. Flowscope: efficient packet capture and storage in 100 gbit/s networks[C] //2017 IFIP Networking Conference(IFIP Networking)and Workshops. Stockholm: IEEE, 2017: 1-9.
[11] EMMERICH P, PUDELKO M, SCHEITLE Q, et al. Efficient dynamic flow tracking for packet analyzers[C] //2018 IEEE 7th International Conference on Cloud Networking(CloudNet). Tokyo: IEEE, 2018: 1-6.
[12] TREVISAN M, FINAMORE A, MELLIA M, et al. Traffic analysis with off-the-shelf hardware: challenges and lessons learned[J]. IEEE Communications Magazine, 2017, 55(3): 163-169.
[13] APACHE Metron. Apache metron home page[EB/OL]. [2020-03-20]. http://metron.apache.org/.
[14] 王煜骢,陈兴蜀,罗永刚,等.NTCI-Flow:一种可扩展的高速网络流量处理框架[J].工程科学与技术,2017,49(S1):168-174. WANG Yucong, CHEN Xingshu, LUO Yonggang, et al. NTCI-Flow: an extensible framework for high speed network traffic processing[J]. Advanced Engineering Sciences, 2017, 49(S1):168-174.
[15] WANG M, LIU J, ZHOU W. Design and implementation of a high-performance stream-oriented big data processing system[C] //2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics(IHMSC). Hangzhou: IEEE, 2016, 1: 363-368.
[16] CAI Y, WU B, ZHANG X, et al. Flow identification and characteristics mining from internet traffic with hadoop[C] //2014 International Conference on Computer, Information and Telecommunication Systems(CITS). Jeju: IEEE, 2014: 1-5.
[1] CUI Zhao-yang, SUN Jia-qi, XU Song-yan, JIANG Xin. A secure clustering algorithm of Ad Hoc network for colony UAVs [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(7): 51-59.
[2] ZHU Dan, XIE Xiao-yao, XU Yang, XIA Meng-ting. Evaluation method for network security level based on cloud model and Bayesian feedback [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(1): 53-62.
[3] LI Yang, CHENG Xiong, TONG Yan, CHEN Wei, QIN Tao, ZHANG Jian, XU Ming-di. Method for threaten users mining based on traffic statistic characteristics [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(1): 83-88.
[4] WU Huan, ZHAN Jing, ZHAO Yong, TAO Zheng, YANG Jing. An efficient multilevel interconnection network security mechanism based on virtualization [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2016, 51(3): 98-103.
[5] LV Liang, YANG Bei, CHEN Zhen-Xiang. Research and design of a network security protection system [J]. J4, 2009, 44(9): 47-51.
[6] ZHANG Jia,DUAN Hai-xin,GE Lian-sheng . Analysis algorithm for the worm metwork behavior based on event sequence [J]. J4, 2007, 42(9): 36-40 .
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] YANG Jun. Characterization and structural control of metalbased nanomaterials[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2013, 48(1): 1 -22 .
[2] HE Hai-lun， CHEN Xiu-lan* . Circular dichroism detection of the effects of denaturants and buffers on the conformation of cold-adapted protease MCP-01 and  mesophilic protease BP01[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2013, 48(1): 23 -29 .
[3] ZHAO Jun1, ZHAO Jing2, FAN Ting-jun1*, YUAN Wen-peng1,3, ZHANG Zheng1, CONG Ri-shan1. Purification and anti-tumor activity examination of water-soluble asterosaponin from Asterias rollestoni Bell[J]. J4, 2013, 48(1): 30 -35 .
[4] SUN Xiao-ting1, JIN Lan2*. Application of DOSY in oligosaccharide mixture analysis[J]. J4, 2013, 48(1): 43 -45 .
[5] LUO Si-te, LU Li-qian, CUI Ruo-fei, ZHOU Wei-wei, LI Zeng-yong*. Monte-Carlo simulation of photons transmission at alcohol wavelength in  skin tissue and design of fiber optic probe[J]. J4, 2013, 48(1): 46 -50 .
[6] YANG Lun, XU Zheng-gang, WANG Hui*, CHEN Qi-mei, CHEN Wei, HU Yan-xia, SHI Yuan, ZHU Hong-lei, ZENG Yong-qing*. Silence of PID1 gene expression using RNA interference in C2C12 cell line[J]. J4, 2013, 48(1): 36 -42 .
[7] MAO Ai-qin1,2, YANG Ming-jun2, 3, YU Hai-yun2, ZHANG Pin1, PAN Ren-ming1*. Study on thermal decomposition mechanism of  pentafluoroethane fire extinguishing agent[J]. J4, 2013, 48(1): 51 -55 .
[8] YANG Ying, JIANG Long*, SUO Xin-li. Choquet integral representation of premium functional and related properties on capacity space[J]. J4, 2013, 48(1): 78 -82 .
[9] LI Yong-ming1, DING Li-wang2. The r-th moment consistency of estimators for a semi-parametric regression model for positively associated errors[J]. J4, 2013, 48(1): 83 -88 .
[10] DONG Wei-wei. A new method of DEA efficiency ranking for decision making units with independent subsystems[J]. J4, 2013, 48(1): 89 -92 .
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd