期刊文献+

基于Thrift的HBase数据存储机制优化 预览 被引量:1

Storage Mechanism Optimization of HBase Data Based on Thrift
在线阅读 免费下载
收藏 分享 导出
摘要 针对Thrift接口服务定义的IDL对HBase数据库按行存储,当数据量大时频繁进行数据请求操作,增加服务调用时间,影响数据通信性能的问题,在详细分析Thrift源码架构基础上,提出了一种新的Thrift IDL设计模型。该模型重新定义了数据传输结构,将多行数据绑定在一起,经过一次RPC调用即可完成多行数据按块存储;采用新的IDL模型,修改了HBase Thrift 服务端的接口以及客户端的非阻塞实现。理论分析和实验结果表明,该方法可有效降低IDL向服务端发送数据操作请求频率,使得HBase储存效率提高4~5倍。 The IDL defined for the Thrift interface service stores the HBase database in rows. When the amount of data is large,the data request operation is frequently performed,the service call time is increased,and the data communication performance is affected. Based on the detailed analysis of the Thrift source architecture,a new Thrift IDL design model was proposed. The data transfer structure was redefined,to bound multiple rows of data together,and to complete multi-row data block storage after an RPC call. With the new IDL model,the interface of the HBase Thrift server and the non-blocking implementation of the client were modified. Theoretical analysis and experimental results show that this method can effectively reduce the frequency of IDL sending data operation requests to the server,which makes HBase storage efficiency increase by 4 ~ 5 times.
作者 温振蕙 樊永生 余红英 WEN Zhen-hui;FAN Yong-sheng;YU Hong-ying(School of Data Science and Technology,North University of China,Taiyuan 030051,China;School of Electrical and Control Engineering,North University of China,Taiyuan 030051,China)
出处 《科学技术与工程》 北大核心 2019年第6期185-189,共5页 Science Technology and Engineering
基金 山西省自然科学基金(201601D102029)资助.
关键词 HBASE THRIFT 远程访问 IDL 大数据 HBase Thrift remote access IDL big data
作者简介 第一作者:温振蕙(1992—),女,汉族,山西祁县人,硕士研究生。E-mail:wzhhui025@163.com;通信作者:樊永生(1967—),男,汉族,山西运城人,博士,教授。E-mail:fanys67@163.com。
  • 相关文献

参考文献4

二级参考文献46

  • 1Dean J, Ghemawat S. MapKeduce: simplified data processing onlarge clusters. In: Proceedings of the 6th Symposium on OperatingSystems Design and Implementation ( OSDI. SanFrancisco,CA: ACM,2004:137-150. 被引量:1
  • 2Taylor R C. An overview of the Hadoop/MapReduce/HBase frame-work and its current applications in bioinformatics. Bmc Bioinformat-ics SI, 2010; 11(6) :3395-3407. 被引量:1
  • 3White T. Hadoop : the definitive guide. O ’ reilly Media Inc Graven-stein Highway North, 2010; 215(11) :1-4. 被引量:1
  • 4Storage S, Worldwide R. The Google file system. In: Proceedings ofthe 19th ACM Symposium on Operating Systems Priciples(SOSP. New York, NY : ACM ,1999: 29-43. 被引量:1
  • 5陈时远.基于HDFS的分布式海量遥感影像数据存储技术研究.北京:中国科学院大学,2013. 被引量:1
  • 6王北辰.基于结构化索引的RDF数据存储及查询方法的研究与实现.北京:北京交通大学,2013. 被引量:1
  • 7Chang F, Dean J, Ghemawat S, et al. Bigtable: A distributed storage system for structured data//Proeeedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI). Seattle, USA, 2006 : 205-218. 被引量:1
  • 8Lakshman A, Malik P. Cassandra--A decentralized struc- tured storage system. ACM SIGOPS Operating Systems Review, 2010, 44(2): 35-40. 被引量:1
  • 9DeCandia G, Hastorun D, Jampani M, et al. Dynamo: Amazon's highly available key-value store//Proeeedings of the 21st ACM Symposium on Operating Systems Principles. Stevenson, USA, 2007:205-220. 被引量:1
  • 10中国计算机学会大数据专家委员会.2013年中国大数据技术与产业发展白皮书,2013. 被引量:1

共引文献39

同被引文献9

引证文献1

投稿分析

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部 意见反馈
新型冠状病毒肺炎防控与诊疗专栏