期刊文献+

基于Spark Streaming的在线KMeans聚类模型研究 预览 被引量:1

Research on Online KMeans Clustering Model Based on Spark Streaming
在线阅读 下载PDF
收藏 分享 导出
摘要 针对基于MapReduce框架处理海量数据实时响应能力较差的问题,设计并实现了基于Spark Streaming的在线计算模型进行大规模的KMeans聚类分析。该模型将整个过程分为数据接入、在线训练等模块,各模块通过数据流连通形成任务实体,提交到Spark分布式集群运行完成。通过比对分析实验和性能检测,验证了该在线KMeans聚类模型具有高吞吐、低延迟的优势,且集群运行状况良好。 Aiming at the poor ability to deal with huge amounts of data realtime response based on MapReduce framework,this paper designs and implements online calculation model for large-scale KMeans clustering analysis based on the Spark Streaming.The whole process can be divided into the model data access,online training modules,each module through data linger form task entity,submitted to a Spark distributed cluster.Through the comparative analysis experiment and performance testing,the online KMeans clustering model is validated with the advantages of high throughput and low delay,and cluster is running in good condition.
出处 《计算机与数字工程》 2018年第4期783-787,共5页 Computer & Digital Engineering
关键词 MAPREDUCE SPARK STREAMING 在线计算 低延迟 MapReduce Spark Streaming online calculation low latency
作者简介 侯敬儒,男,硕士研究生,研究方向:数据挖掘与机器学习。;吴晟,男,教授,硕士生导师,研究方向:信息安全与算法。;李英娜,女,副教授,硕士生导师,研究方向:传感网组建与信息集成和智能分析。
  • 相关文献

参考文献11

二级参考文献96

  • 1尹清波,张汝波,李雪耀,王慧强.基于线性预测与马尔可夫模型的入侵检测技术研究[J].计算机学报,2005,28(5):900-907. 被引量:27
  • 2[1]K Ilgun,R Kemmerer,P Porras.State transition analysis:A rule-based intrusion detection approach.IEEE Trans on Software Engineering,1995,21(3):181-199 被引量:1
  • 3[2]Chen Wunhwa,Sheng Hsun Hsu,Hwang-Pin Shen.Application of SVM and ANN for intrusion detection.Computers & Operations Research,2005,32(10):2617-2634 被引量:1
  • 4[5]S A Hofmeyr,S Forrest,A Somayaji.Intrusion detection using sequences of system calls.Journal of Computer Security,1998,6(3):151-180 被引量:1
  • 5[6]W Lee,X Dong.Information-Theoretic measures for anomaly detection.In:Proc of the 2001 IEEE Symposium on Security and Privacy.Los Alamitos,CA:IEEE Computer Society Press,2001.130-143 被引量:1
  • 6[7]Dit-Yan Yeung,Yuxin Ding.Host-based intrusion detection using dynamic and static behavioral models.Pattern Recognition,2003,36(1):229-243 被引量:1
  • 7[8]Zhang Zonghua,Shen Hong.Application of online-training SVMs for real-time intrusion detection with different considerations.Computer Communications,2005,28(12):1428-1442 被引量:1
  • 8[9]C Kruegel,D Mutz,F Valeur,et al.Bayesian event classification for intrusion detection.In:Proc of the 19th Annual Computer Security Applications Conf.Los Alamitos,CA:IEEE Computer Society Press,2000 被引量:1
  • 9[10]W Lee,S Stolfo,P Chan,et al.Real time data mining-based intrusion detection.In:Proc of the Second DARPA Information Survivability Conf and Exposition.Los Alamitos,CA:IEEE Computer Society Press,2001.89-100 被引量:1
  • 10[11]Eleazar Eskin.Anomaly detection over noisy data using learned probability distributions.The 2000 Int'l Conf on Machine Learning (ICML-2000),Palo Alto,CA,2000 被引量:1

共引文献77

同被引文献14

引证文献1

投稿分析

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部 意见反馈