-
摘要: 地质工作已迈入大数据时代，但地学信息被记录成的报告、图件等非结构化数据，仍按照较为简单的方式组织归类到一起并存储在文件系统中，形成很多个内部构成复杂的数据集。这种方式不能很好的表达非结构化数据承载的丰富地学信息，也不便表达信息之间的复杂关系，更不利于发现跨数据集存在的深层知识。为尝试解决这个问题，笔者提出了多粒度级别内容树模型和支持演化的数据建模方式。这些特性使得通过模型可以对数据内容进行不同尺度的拆分，对信息的精确定位，还可以使模型根据数据主体需要，拓展主体特征描述的维度，逐步发现数据包含的信息和建立信息与信息之间的关系。考虑到地质大数据的特点，设计了以HBase为核心的数据模型持久化方式，以达到使用大数据技术体系下技术分析处理数据的目的；最后给出了对成果地质数据进行建模的实例，将文档、图件等非结构化数据以内容实体为最小单元进行拆分和重构，达到了较好的内容组织和信息表达效果。Abstract: Geological work has entered the era of big data, yet the unstructured data, such as reports and maps carrying geosciences information, are still classified in simple ways and stored in the file system, forming a lot of data set with complex internal structures. This method cannot well deliver the abundant geosciences information carried by unstructured data or the complex relationships with information, nor can it discover the knowledge deeply existing across data sets. To solve the problem, this paper proposes a multi-granularity level content tree model and a data modeling method that supports evolution. The model can split the data content at different scales and accurately locate the information and meanwhile expand the dimension of the subject's feature description according to the need of the data subject. The information contained in the data is finally discovered and the relationship with information is thus established. This paper designs a persistence method of data model with HBase as the core to achieve the purpose of processing data under the big data technology system. A modeling example shows preferable effect in content organization and information conveying, with the unstructured data of documents and maps split and reconstructed as the smallest unit of the content entity.
-
Key words:
- geological big data /
- unstructured data /
- data model /
- content storage
-
-
赵鹏大.地质大数据特点及其合理开发利用[J]. 地学前缘, 2019, 26(4):1-5.
ZHAO P D.Characteristics and Rational Utilization of Geological Big Data[J]. Earth Science Frontiers, 2019, 26(4):1-5.
陈建平, 李靖, 谢帅, 等.中国地质大数据研究现状[J]. 地质学刊, 2017, 41(03):353-366.
CHEN J P, LI J, XIE S, et al. China Geological Big Data Research Status[J]. Journal of Geology, 2017, 41(03):353-366.
李超岭, 李健强, 张宏春, 等.智能地质调查大数据应用体系架构与关键技术[J]. 地质通报, 2015, 34(07):1288-1299.
LI C L, LI J Q, ZHANG H C, et al. Big Data Application Architecture and Key Technologies of Intelligent Geological Survey[J]. Geological Bulletin of China, 2015, 34(07):1288-1299.
王珊, 王会举, 覃雄派, 等.架构大数据:挑战、现状与展望[J]. 计算机学报, 2011, 34(10):1741-1752.
WANG S, WANG H J, QIN X P, et al. Architecting Big Data:Challenges, Studies and Forecasts[J]. Chinese Journal of Computers, 2011, 34(10):1741-1752.
覃雄派, 王会举, 李芙蓉, 等.数据管理技术的新格局[J]. 软件学报, 2013, 24(02):175-197.
QIN X P, WANG H J, LI F R, et al. New Landscape of Data Management Technologies[J]. Journal of Software, 2013, 24(2):175-197
王梅, 周娇玲, 乐嘉锦.一种列存储数据仓库中的数据复用策略[J]. 计算机学报, 2013, 36(08):1626-1635.
WANG M, ZHOU J L, LE J J.A Data Reusing Strategy in Column-Store Data Warehouse[J]. Chinese Journal of Computers, 2013, 36(08):1626-1635.
吴冲龙, 刘刚, 张夏林.地质科学大数据及其利用的若干问题探讨[J]. 科学通报, 2016, 61(16):1797-1807.
WU C L, LIU G, ZHANG X L.Discussion on Geological Science Big Data and its Applications[J]. Chinese Science Bulletin, 2016, 61(16):1797-1807.
杨鹏, 林俊晖.一种基于MongoDB和Hadoop的海量非结构化物联网数据处理方案[J]. 微电子学与计算机, 2018, 35(04):68-72+78.
YANG P, LIN J H.A Scheme for Massive Unstructured Iot Data Processing Based on MongoDB and Hadoop[J]. Microelectronics & Computer, 2018, 35(04):68-72+78.
谢华成, 陈向东.面向云存储的非结构化数据存取[J]. 计算机应用, 2012, 32(07):1924-1928+1942.
XIE H C, CHEN X D.Cloud storage-oriented unstructured data storage[J]. Journal of Computer Applications, 2012, 32(07):1924-1928+1942.
李玉坤, 孟小峰, 张相於.数据空间技术研究[J]. 软件学报, 2008(08):2018-2031.
LI Y K, MENG X F, ZHANG X Y.Research on Dataspace[J]. Journal of Software,, 2008(08):2018-2031.
Biham E, Chen R, Joux A, et al. Collisions in SHA-0 and Reduced SHA-1[M]. Springer Berlin Heidelberg, 2005.
Dean J, Ghemawat S.MapReduce:Simplified data process-ing on large clusters[J]. Communications of the ACM, 2004, 51(1):137-150.
Ashley I.Naimi, Daniel J.Westreich.Big Data:A Revolution That Will Transform How We Live, Work, and Think[J]. American Journal of Epidemiology, 2014, 179(9)Pages 1143-1144.
Cuzzocrea A, Song I Y, Davis K C.Analytics over Large-scale Multidimensional Data:the Big Data Revolution[A]//International Workshop on Dolap[C]. ACM, 2011, 101-104.
Franklin M, Halevy A, Maier D.From Databases to Dataspaces:A New Abstraction for Information Management[J]. Sigmod Record:Acm Sigmod (management of data), 2005, 34(4):27-33.
Chang F, Dean J, Ghemawat S, et al. Bigtable:A Distributed Storage System for Structured Data[J]. Acm Transactions on Computer Systems, 2008, 26(2):1-26.
-
计量
- 文章访问数: 1141
- PDF下载数: 64
- 施引文献: 0
下载: