An Online Product Review Information Collection Method Based on Storm

Luo Kui-yong; Hao Zhi-feng; Cai Rui-chu; Wen Wen; Yuan Qin

doi:10.12052/gdutxb.170042

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

An Online Product Review Information Collection Method Based on Storm

更新时间：2026-05-27

- An Online Product Review Information Collection Method Based on Storm
- Vol. 34, Issue 3, Pages: 83-88(2017)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.170042
  CLC： TP391
- Online First：09 May 2017，
  
  Published：2017
- 稿件说明：
移动端阅览
骆魁永, 郝志峰, 蔡瑞初, 温雯, 袁琴. 一种基于Storm的在线产品评论信息采集的方法[J]. 广东工业大学学报, 2017, 34(3): 83-88.

Luo Kui-yong, Hao Zhi-feng, Cai Rui-chu, et al. An Online Product Review Information Collection Method Based on Storm[J]. 2017, 34(3): 83-88.
骆魁永, 郝志峰, 蔡瑞初, 温雯, 袁琴. 一种基于Storm的在线产品评论信息采集的方法[J]. 广东工业大学学报, 2017, 34(3): 83-88. DOI： 10.12052/gdutxb.170042.

Luo Kui-yong, Hao Zhi-feng, Cai Rui-chu, et al. An Online Product Review Information Collection Method Based on Storm[J]. 2017, 34(3): 83-88. DOI： 10.12052/gdutxb.170042.

摘要

针对如何尽早地获取电商网站中产品的评论信息，进而实时地把握产品舆情，提出了一种基于Storm的在线产品评论信息采集方法.该方法将流计算的概念应用于网络爬虫中，并通过SHHD算法对采集周期进行动态调整.实验结果表明：基于Storm平台进行信息采集具有吞吐量大、可扩展性强等优点；SHHD算法可以有效地降低采集系统对网络带宽和系统资源的消耗，实现了适应性的增量的在线产品评论信息采集过程；SHHD在产品的评论信息获取的滞后时间上较Poisson、SART等方法具有明显的优势.

Abstract

With regard to getting comment information of the products in the electricity sales website as soon as possible and grasping product public opinion in real time

a method of online product reviews information collection based on Storm is presented. The concept of flow computation is applied to the web crawler

and the SHHD (Simhash Hamming Distance) algorithm is used to dynamically adjust the acquisition period. Experimental results show that information collection based on Storm has the advantages of large throughput and easy updating. The SHHD algorithm can effectively reduce the acquisition system on the network bandwidth and system resources consumption and achieve an adaptive incremental online product review information collection process. SHHD has certain advantages in the lag of product comment information acquisition than Poisson and SART.

关键词

Keywords

references

中国互联网信息中心(CNNIC). 2015年中国网络购物市场研究报告[R]. 北京:CNNIC, 2016. 6

NIRAJ S, ASHUTOSH D, SHARMA A K. Design of a priority based frequency regulated incremental crawler[J]. International Journal of Computer Applications, 2010, 1(1):42-47.

SHARMA AK, DIXIT A. Self adjusting refresh time based architecture for incremental web crawler[J]. International Journal of Computer Science and Network Security, 2008, 8(12):349-354.

TESSERA D, CALZAROSSA M. Modeling and predicting temporal patterns of web content changes[J]. Journal of Network and Computer Applications, 2015, 2015(56):115-123.

SIA K C, CHO J, CHO H K. Efficient monitoring algorithm for fast news alerts[J]. IEEE Transactions on Knowledge & Data Engineering, 2007, 19(7):950-961.

孟涛, 王继民, 闫宏飞. 网页变化与增量搜集技术[J]. 软件学报, 2006, 17(5):1051-1067. MENG T, WANG J M, YAN H F. Web evolution and incremental crawling[J]. Journal of Software, 2006, 17(5):1051-1067.

CHO J, GARCIA-MOLINA H. Estimating frequency of change[J]. Acm Transactions on Internet Technology, 2003, 3(3):256-290.

DIXIT A, SHARMA A K. A mathematical model for crawler revisit frequency[C]//Advance Computing Conference (IACC), 2010 IEEE 2nd International.[S.l.]:IEEE, 2010:316-319.

崔星灿, 禹晓辉, 刘洋, 等. 分布式流处理技术综述[J]. 计算机研究与发展, 2015, 52(2):318-332. Cui X C, Yu X H, LIU Y, et al. Distributed stream processing:a survey[J]. Journal of Computer Research and Development, 2015, 52(2):318-332.

邓立龙, 徐海水. Storm实现的应用模型研究[J]. 广东工业大学学报, 2014, 31(3):114-118. Deng L L, Xu H S. Research on applied models based on Storm[J]. Journal of Guangdong University of Technology, 2014, 31(3):114-118.

YANG W, LIU X, ZHANG L, et al. Big data real-time pro-cessing based on Storm[C]//201312th IEEE International Conference on Trust, Security and Privacy in Computing and Communications.[S.l.]:IEEE, 2013:1784-1787.

UDAPURE T V. Study of web crawler and its different types[J]. IOSR Journals (IOSR Journal of Computer Engineering), 2014, 1(16):1-5.

董博, 郑庆华, 宋凯磊, 等. 基于多SimHash指纹的近似文本检测[J]. 小型微型计算机系统, 2011, 32(11):2152-2157. DONG B, ZHENG Q H, SONG K L, et al. Efficient near-duplicate detection based on multiple simhash fingerprints[J]. Journal of Chinese Computer Systems, 2011, 32(11):2152-2157.

寇月, 李冬, 申德荣等. D-EEM:一种基于DOM树的Deep Web实体抽取机制[J]. 计算机研究与发展, 2010, 47(5):858-86. KOU Y, LI D, SHEN D R. D-EEM:A DOM-tree based entity extraction mechanism for deep web[J]. Journal of Computer Research and Development, 2010, 47(5):858-86.

MANKU G S, JAIN A, DAS SARMA A. Detecting near-duplicates for web crawling[C]//International Conference on World Wide Web.[S.l.]:ACM, 2007:141-150.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on Applied Models Based on Storm

A Study of Power Quality Regulation Strategies of Energy Storage Converters

Energy Management Strategy of Hybrid Locomotive Based on Road Condition Parameters

Bimodal Iterative Cross-attention Fusion Ensemble Framework

Research and Implementation of a Pulsed Laser Ranging Method Based on Digital Mixing

Related Author

Deng Li-long

Xu Hai-shui

Related Institution

No data

AI问答

⁰