CN102833331B - Cloud storage system and metadata writing-in method and metadata reading method thereof - Google Patents
Cloud storage system and metadata writing-in method and metadata reading method thereof Download PDFInfo
- Publication number
- CN102833331B CN102833331B CN201210299624.9A CN201210299624A CN102833331B CN 102833331 B CN102833331 B CN 102833331B CN 201210299624 A CN201210299624 A CN 201210299624A CN 102833331 B CN102833331 B CN 102833331B
- Authority
- CN
- China
- Prior art keywords
- metadata
- meta data
- view
- data server
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a cloud storage system and a metadata writing-in method and a metadata reading method thereof. The cloud storage system comprises a metadata server cluster, a view management module and a client side. The client side provides an interface of visiting the cloud storage system for a user and analyzes a view position of a metadata server to be accessed by the user; the view management module stores all metadata view information, establishes a logic association relation among all metadata servers according to all metadata view information and dispenses the metadata view information associated to the metadata server per se to each metadata server; each metadata server stores the metadata view information dispensed by the view management module and determines the network access address information of the metadata servers; and the metadata server carries out layered deployment, wherein the metadata server at the bottommost layer also stores metadata contents and the metadata view information and provides the metadata information stored by the metadata server per se for the user according to a user request. According to the cloud storage system and the metadata writing-in method and the metadata reading method thereof, the bandwidth consumption can be reduced.
Description
Technical field
The present invention relates to a kind of cloud storage system, particularly relating to cloud storage system for storing Large Volume Data and content metadata wiring method thereof and content metadata read method.
Background technology
Metadata be " data about data " in geospatial information for describing other features of the content of geographical data set, quality, representation, georeferencing, way to manage and data set, it is one of core standard realizing geospatialinformation sharing.At present, the tissue studied Spatial motion analysis content in the world mainly contains three, is European Committee for Standardization (CEN/TC287), the geodata committee of the United States Federal (FGDC) and International Organization for standardization's geography information/Geo-information technology committee (ISO/TC211) respectively.Spatial motion analysis content divides two levels.Ground floor is directory information, is mainly used in carrying out macroscopic description to data set information, uses when it is adapted at management and search space information in the national geo-spatial data exchange center of digital earth or region and global range.The second layer is details, and being used in detail or comprehensive Spatial motion analysis content describing geospatial information, is the information that the data set producer must provide when providing space data sets.
Along with going deep into of the level of informatization, information data amount is increasing, and the cloud storage system of the metadata server cluster comprised for storing metadata (Metadata), has appearred in extensive use metadata technique.
In the cloud storage system of existing metadata server cluster, in each meta data server of metadata server cluster, store the Metadata View The information of whole cloud storage system.But, along with the data volume of content metadata is increasing.In the safe data cloud storage system of unusual magnanimity, Metadata View The information also can become mass data.Now, each meta data server in metadata server cluster all needs to store all magnanimity Metadata View The information, thus takies a large amount of memory spaces.
In addition, if each meta data server in metadata server cluster stores the identical Metadata View The information of a large amount of contents, then when client retrieves content metadata, it also will be very time-consuming for searching in the Metadata View The information of magnanimity.
Summary of the invention
One of technical problem to be solved by this invention needs to provide a kind of cloud storage system and metadata wiring method, the metadata read method that reduce the bandwidth consumption of network.
In order to solve the problems of the technologies described above, the invention provides a kind of cloud storage system.This system comprises metadata server cluster, view management module and client, wherein:
Client is used for providing the interface of accessing described cloud storage system to user, and parses the view location of the meta data server that user will access according to the access request of user;
View management module is for storing whole Metadata View The information of whole metadata server cluster, according to the logic association relation between each meta data server in described whole Metadata View The information architecture metadata server cluster, and according to the Metadata View The information that this logic association relation self is associated to each meta data server distribution with this meta data server;
In metadata server cluster, each meta data server is for storing the Metadata View The information of being distributed by view management module, and the view location of the meta data server sent according to client or other meta data server determines the access to netwoks address information of the meta data server corresponding with the view location of this meta data server;
Multiple meta data server layerings of described metadata server cluster are disposed, be in the meta data server of the bottom in described metadata server cluster also for the Metadata View The information that storing metadata content and view management module are distributed, and according to user's request, the metadata information that self stores be supplied to user.
Further, each meta data server of described metadata server cluster all only stores the part content metadata of whole described cloud storage system.
According to a further aspect of the invention, described view management module comprises Data Management Unit, information process unit and content view unit further, wherein: Data Management Unit, for storing whole Metadata View The information of described cloud storage system; Information process unit, for the whole Metadata View The information stored according to described Data Management Unit, build the logic association relation of each meta data server in described metadata cluster, and Metadata View The information is processed, obtain Metadata View The information, and distribute described Metadata View The information to each meta data server, make each meta data server store the view information self be associated with this meta data server; Content view unit, when the content metadata for the file stored in a meta data server in described cloud storage system is accessed by another meta data server continually, writes this another meta data server by this file.
According to a further aspect of the invention, described Data Management Unit also correspondingly stores the complete Metadata View The information corresponding respectively with each default view strategy of many covers according to different default view strategies; And, described view management module comprises decision package further, for client based on metadata access request and described default view strategy determine multiple for access target metadata server time, by described multiple meta data server being defined as final access for one of access destination meta data server.
According to a further aspect of the invention, described view management module comprises receiving element further, wherein: described receiving element is used for the load information of the meta data server receiving each bottom from each meta data server; Described information process unit is also predicted based on the load information of meta data server, obtains the load estimation information of the meta data server in next period; Described decision package, for client based on metadata access request and described default view strategy determine multiple for access target metadata server time, according to described load estimation information by described multiple meta data server being defined as final access for one of access destination meta data server.
According to a further aspect of the invention, described decision package carries out following load balancing control further: when described information process unit there occurs fault based on described load estimation INFORMATION DISCOVERY existence or do not have the meta data server responded to send out, described decision package upgrades the Metadata View The information of the meta data server associated with the meta data server broken down based on whole Metadata View The information of the described cloud storage system stored in described Data Management Unit, with no longer to breaking down or not having the meta data server responded to send metadata access request; And/or when there is the excessive meta data server of load by information process unit based on load estimation INFORMATION DISCOVERY, described decision package indicate the excessive meta data server of this load to stop providing Metadata Service or by all or part of metadata replication to the little meta data server of load in described cloud storage system, or described decision package triggers the mirror image of the excessive meta data server of generation described load on the meta data server that load is little.
According to another aspect of the invention, also provide a kind of metadata wiring method of cloud storage system.The method comprises:
Client receive user to write the write request of metadata to be written time, according to default access strategy determine access access access meta data server, said write request comprises content metadata to be written;
Described client carries out preliminary treatment to said write request, to obtain the view location of the target metadata server for depositing described content metadata to be written;
If the meta data server that the view location of determined target metadata server is corresponding is described access meta data server, then the access to netwoks address information of described access meta data server is defined as the access to netwoks address information of target metadata server, otherwise, based on the hierarchical relationship of each meta data server in described metadata cluster, according to the view location of the Metadata View The information stored in described access meta data server and target metadata server, target metadata server is searched step by step from described access meta data server, to determine the access to netwoks address information of target metadata server,
According to the access to netwoks address information of determined target metadata server, this metadata to be written is write described target metadata server.
Further, described default access strategy is physics is geographically defined as described access meta data server from the meta data server that described user is nearest or the Metadata View The information corresponding according to described default view strategy is confirmed as the fastest meta data server of described user access being defined as described access meta data server.
Further, if described client saves the many cover Metadata View The corresponding respectively with multiple described default view strategy, then when each cover Metadata View The information of described client corresponding to various described default view strategy determines the view location of multiple target metadata server, one of view location of the plurality of target metadata server is defined as the view location of final target metadata server; According to the access to netwoks address information of determined final target metadata server, this metadata to be written is write described target metadata server.
According to another aspect of the invention, the method that the metadata additionally providing a kind of cloud storage system reads.The method comprises: the expectation that client receives user is read when the read requests of metadata, determines the access meta data server of accessing access according to default access strategy, described client records according to the metadata write that described read requests and described client are preserved and Metadata View The information is determined to house the view location that user expects the target metadata server of the metadata read, if the meta data server that the view location of determined target metadata server is corresponding is not described access meta data server, then based on the hierarchical relationship of meta data server each in described cloud storage system, according to the view location accessing Metadata View The information and the target metadata server stored in meta data server, target metadata server is searched step by step from access meta data server, to determine the access to netwoks address information of target metadata server, and the content metadata that user will read is distributed to access meta data server according to described read requests by target metadata server described in the access to netwoks address information request of based target meta data server, described client reads from described access meta data server the metadata that described user will read.
Compared with prior art, one or more embodiment of the present invention can have the following advantages by tool:
The large cloud storage system of view information amount can be applied to preferably according to the cloud storage system that the embodiment of the present invention provides, the each meta data server arranged by layering realizes metadata access mutually in phase according to respective stored Metadata View The, compare conventional art, the bandwidth consumption of network can be reduced, meet the requirement of the view equilibrium of network traffics and meta data server.
Other advantages of the present invention, target, to set forth in the following description to a certain extent with feature, and to a certain extent, based on will be apparent to those skilled in the art to investigating hereafter, or can be instructed from the practice of the present invention.Target of the present invention and other advantages can by specifications below, claims, and in accompanying drawing, specifically noted structure realizes and obtains.
Accompanying drawing explanation
Accompanying drawing is used to provide a further understanding of the present invention, and forms a part for specification, with embodiments of the invention jointly for explaining the present invention, is not construed as limiting the invention.In the accompanying drawings:
Fig. 1 is the structural representation of the cloud storage system according to the embodiment of the present invention one;
Fig. 2 is the flow chart of the metadata wiring method of cloud storage system according to the embodiment of the present invention two;
Fig. 3 is the flow chart of the metadata read method of cloud storage system according to the embodiment of the present invention three.
Embodiment
For making object of the present invention, technical characteristic and implementation result clearly, embodiments of the present invention is described in detail below in conjunction with the accompanying drawings and the specific embodiments.It should be noted that, only otherwise form conflict, each embodiment in the present invention and each feature in each embodiment can be combined with each other, and the technical scheme formed is all within protection scope of the present invention.
In addition, can perform in the computer system of such as one group of computer executable instructions in the step shown in the flow chart of accompanying drawing, and, although show logical order in flow charts, but in some cases, can be different from the step shown or described by order execution herein.
The content mentioned in the following embodiment of the present invention is the content metadata that user needs to access, and the file mentioned in the embodiment of the present invention, is the file comprising content metadata.
Embodiment one
Fig. 1 describes the structural representation of a kind of cloud storage system that the embodiment of the present invention provides.This system comprises: metadata server cluster 10, view management module 20 and client 30.
Multiple meta data server layerings of metadata server cluster 10 are disposed.The server of metadata server cluster 10 can realize cooperation by network, i.e. request service and provide service mutually.
In metadata server cluster 10, each meta data server is for storing the view information of being distributed by view management module 20, and the request of sending according to client 30 or other meta data server, inquires about the access to netwoks address information of the meta data server of specifying.This access to netwoks address information can be such as network address, domain name etc. and makes client 30 can the network address information of accesses meta-data server.
Further, be in the meta data server of the bottom in metadata server cluster 10 also for the view information that storing metadata content (being also called metadata information or metadata) and view management module 20 are distributed, and according to user's request, the metadata information that self stores be supplied to user.It should be noted that, (namely each meta data server does not all store whole content metadatas of whole cloud storage system, each meta data server only stores the part content metadata of whole described cloud storage system), the union of the content metadata that each meta data server stores is whole content metadatas.
View management module 20, for storing whole Metadata View The information (also whole Metadata View The information being called view index information) of whole metadata server cluster 10, and according to this view index information, build the logic association relation (or being called logic connecting relation) between each meta data server in metadata server cluster 10, and according to the view information that this logic association relation self is associated to each meta data server distribution with this meta data server.The view information self be associated with certain meta data server can comprise the title of this meta data server self, geospatial information and access to netwoks address information, title, geospatial information and access to netwoks address information etc. with other meta data server of this meta data server logic association.
Preferably, view management module 20 also can the periodically load information of each meta data server in Detection theory data server cluster 10, preferably only periodically can detect the load information of each bottom meta data server.Like this, be convenient to cloud storage system the service of meta data server excessive for load pressure is switched on other little meta data servers of relative load.The described cycle can be set to 10 minutes, one hour, one day etc., can determine according to the change speed of the access frequency of content metadata, as very fast in access frequency change then the cycle can establish shorter, and access frequency change is comparatively slow, and then the cycle can establish longer.
Preferably, view management module 20 also can according to different default view strategies, build the many covers logic association relation between each meta data server in metadata server cluster 10, and according to many covers view information that this many covers logic association relation self is associated to each meta data server distribution with this meta data server.Each cover view information is corresponding with each cover logic association relation.
Like this, each meta data server stores the view information that each cover corresponding with each cover logic association relation self is associated with this meta data server.When the meta data server of cover view information more than client 30 will be store by access determines the access to netwoks address information of the meta data server that will access, the view information first self be associated based on each cover and this meta data server is wanted to determine an access to netwoks address information respectively, and then finally determined the access to netwoks address information that will use by decision package, thus content metadata is distributed to most suitable meta data server in cloud storage system as far as possible.Presetting view strategy can be varied, such as, presetting view strategy can according to actual Regional Distribution, as below China point of North China, East China, south China ... point Beijing, Tianjin, Hebei below North China ..., point Haidian District, Chaoyang District, Dongcheng District, Xicheng District below Beijing.Presetting view strategy can also according to kind of document, and as being divided into work, amusement, study etc., point technology, economy, humanity etc. below work, divide music, physical culture, film etc. below amusement.According to the difference of business, different logical relations can also be splitted into.
As described in Figure 1, n-th layer meta data server, when receiving the metadata access request from client 30, obtains the access to netwoks address information (detailed in Example two) of the meta data server that client 30 will be accessed according to the Metadata View The information self stored.
In the present invention, n-th layer meta data server is as the actual storage locations of content metadata; Each n-th layer meta data server does not store whole metadata information, and the content sum that all n-th layer meta data servers store is complete content metadata.
It should be noted that, view management module 20 can be arranged on certain meta data server, also can be independent logic module.
In addition, described Metadata View The information both can be the regional information (geospatial information) of meta data server, also can be common tree structure information, or user-defined logical relation information, or according to the Metadata View The information that specific method (such as hash algorithm) constructs.
Client 30 accesses the interface of described cloud storage system for providing to user, resolve the read and write access request of user, this access request comprises the information such as view location information (abbreviation view location), file identification, file size of the meta data server that will access.In addition, client 30 also preserves the written information record about write metadata.View location information can, for needing the access to netwoks address of metadata in first storage networking of access, if the view information built by region, can be China-North China-concrete meta data file in Beijing-Haidian.
Preferably, described view management module 20 can comprise: Data Management Unit, information process unit, receiving element, decision package and content view unit.
Data Management Unit, for storing whole Metadata View The information of cloud storage system.Preferably, can according to the different default view strategies Metadata View The information corresponding respectively with each default view strategy that correspondingly storing multiple sets is complete, like this, make each meta data server store and one to one overlap Metadata View The information from different default view strategies more.In this case, each cover Metadata View The information stored in each meta data server is not all a whole set of Metadata View The information, but part Metadata View The information relevant to oneself in each the whole series (all) Metadata View The information.
Information process unit, for the whole Metadata View The information stored according to Data Management Unit, build the logic association relation of each meta data server in metadata cluster, and Metadata View The information is processed, obtain the Metadata View The information will distributed to each meta data server, and distribute, make each meta data server store the view information self be associated with this meta data server.
Preferably, this cloud storage system also can comprise receiving element.Receiving element receives the load information of the meta data server of each bottom from each meta data server.Further, can be predicted by the load information of information process unit based on meta data server, obtain the load estimation information of the meta data server in next period.Such as, according to the load balancing preset, obtain the load estimation information of the meta data server to next period by the status data of the weight integration multiple period of history pre-set.
Described load information can comprise: the access frequency of each metadata in cloud storage system on each meta data server, and the available storage of each meta data server, outlet bandwidth, ingress bandwidth and service view capacity (free memory), specifically loading algorithm and response delay etc.
Decision package, for client 30 based on the metadata access request of user and aforementioned default view strategy determine multiple for access target metadata server time, by multiple meta data server being defined as final access for one of access destination meta data server.Wherein, the metadata access request of user can be read requests or write request etc.Like this, decision package can realize metadata and more rationally distributes by comprehensive several default view strategy.
Such as, the load estimation information that decision package can obtain according to information process unit process, is defined as the meta data server that will access and notifies client 30 by one of above-mentioned multiple target metadata server.That is, when client 30 determines different target metadata servers according to multiple default view strategy, client 30 does not know finally will access which meta data server, therefore carry out with decision package alternately, notify by decision package which that client 30 finally will access in these target metadata servers.
In addition, when information process unit based on load estimation INFORMATION DISCOVERY exist there occurs fault or not have response meta data server send out time, decision package carries out load balancing control.More specifically, dynamically can be upgraded the Metadata View The information of the meta data server that the meta data server that breaks down with this associates based on whole Metadata View The information of the described cloud storage system stored in described Data Management Unit by decision package, no longer there occurs fault to that or not have the meta data server responded to send metadata access request, such as, can by Metadata View The information about the information deletion of the meta data server broken down.
In addition, when there is the excessive meta data server of load by information process unit based on load estimation INFORMATION DISCOVERY, load balancing can be carried out by decision package to control, such as, decision package can automatically indicate the excessive meta data server of this load to stop providing Metadata Service or by its all or part of metadata replication to the less meta data server of load in this system, or on the meta data server that load is less, the mirror image of the excessive meta data server of generation this load is triggered by decision package, thus by the meta data server that this load is less substitute the excessive meta data server of this load supply service.
Content view unit, for the view location information according to described content metadata, is distributed to the meta data server of suitable view in described cloud storage system by content metadata.
More specifically, when the content metadata of the file that content view unit can store in a certain meta data server in cloud storage system is accessed by another meta data server continually, this file is write another meta data server, and the routing information of another meta data server described in the write of this file is fed back to client 30.Like this, system can provide meta data file access services to client faster.
Further, in order to ensure the consistency of data, system can be divided into groups to meta data server, makes to carry out data backup between the meta data server in group.If any 500 meta data servers, 5 is one group, carries out data backup in group, ensures the consistency of data, so not only ensure that the reliability of data, also improve the efficiency of Outside Access, performance simultaneously.
Embodiments provide a kind of cloud storage system, be applied to the cloud storage system that view information amount is large, the each meta data server arranged by layering realizes metadata access mutually in phase according to from each stored Metadata View The, compare conventional art, the bandwidth consumption of network can be reduced, improve the hit rate of user's accesses meta-data content, meet the requirement of the view equilibrium of network traffics and meta data server.
Embodiment two
About the related description of cloud storage system in previous embodiment, be adapted to the present embodiment equally.Fig. 2 is the flow chart of the metadata wiring method of a kind of cloud storage system that the embodiment of the present invention provides.The metadata wiring method of the cloud storage system of the embodiment of the present invention two is described in detail below in conjunction with Fig. 2.
Step S210, client 30 receive user to write the write request of metadata to be written time, determine the meta data server (be called for short access meta data server) of accessing access according to default access strategy.
This default access strategy can for being such as geographically defined as physics access meta data server, maybe the view information corresponding by aforementioned default view strategy being determined the fastest meta data server of this user access is defined as access meta data server etc. from the meta data server that this user is nearest.This access meta data server is preferably in the meta data server of the bottom.
This write request comprises the content metadata (referred to as content metadata to be written) that will write.
Step S220, client 30 can carry out preliminary treatment to write request, for the view location of target metadata server depositing this content metadata to be written.
Alternatively, step S220 also can comprise further: if this client 30 saves the many cover Metadata View The corresponding respectively with multiple default view strategy, then when each cover Metadata View The information of client 30 corresponding to various default view strategy determines the view location of multiple target metadata server, client 30 can utilize decision package that one of view location of the plurality of target metadata server is defined as the view location of final target metadata server, thus makes metadata access amount can more balancedly be distributed to each meta data server.
Step S230, whether the meta data server (when there is the view location of multiple target metadata server, the view location of this target metadata server refers to the view location of final access meta data server) corresponding to view location judging the target metadata server determined in step S220 is the access meta data server determined in step S210.If so, then step S240 is entered, otherwise, enter step S250.
Step S240, is defined as the access to netwoks address information of target metadata server by the access to netwoks address information of access meta data server.
Step S250 is based on the hierarchical relationship of each meta data server, according to the view location accessing view information and the target metadata server stored in meta data server, target metadata server is searched step by step, to determine the access to netwoks address information of target metadata server from access meta data server.
More specifically, the meta data server (being called for short one-level associated server) associated with the view information stored in this access meta data server is determined according to the view information stored in access meta data server, if there is this target metadata server in one-level associated server, then determine the access to netwoks address information of target metadata server according to the view information stored in access meta data server; Otherwise, further according to the view information stored in (multiple) one-level associated server, determine the meta data server (being called for short secondary associated server) be associated with one-level associated server, if there is this target metadata server in secondary associated server, the access to netwoks address information of target metadata server is then determined according to the view information stored in one-level associated server, the rest may be inferred, in view information, find this target metadata server, thus determine the access to netwoks address information of this target metadata server.
Step S260, according to the access to netwoks address information of this target metadata server determined in step S240 or S250, writes this target metadata server by this metadata to be written.
Alternatively, step S270, result can be returned to client 30 by target metadata server, and client 30 can be preserved the metadata relevant to writing metadata and be write record.Like this, improve user and determine the view location of target metadata server and/or the efficiency of access meta data server when reading metadata from cloud storage system.
In sum, the meta data server arranged by layering in data server cluster assists the access to netwoks address information of searching the target metadata server that will finally will write, the process of the access to netwoks address information determining target metadata server can be distributed to meta data server at different levels to carry out, thus make load more balanced.
Embodiment three
About the related description of cloud storage system in previous embodiment, be adapted to the present embodiment equally.Fig. 3 is the flow chart of the method that the metadata of a kind of cloud storage system that the embodiment of the present invention provides reads.The metadata read method of the cloud storage system of the embodiment of the present invention three is described in detail below in conjunction with Fig. 3.
Step S310, the expectation that client 30 receives user is read when the read requests of metadata, determines the meta data server (being called for short access meta data server) of accessing access according to default access strategy.Determine to access in the mode of meta data server and step S210 similar in this step, launch explanation no longer in detail at this.
This read requests can comprise the attribute that user expects the metadata read.User expects that the attribute of the metadata read can expect the value of one or more fields of the metadata read for user.
Step S320, client 30 records according to the metadata write that this read requests and this client 30 are preserved and view information is determined to house the view location that user expects the target metadata server of the metadata read.More specifically, based on preserved metadata write record and view information, preliminary treatment is carried out to content metadata to be read, obtain housing the view location that user expects the target metadata server of the metadata read.
Step S330, judges whether the meta data server corresponding to view location of the target metadata server determined in step s 320 is the access meta data server determined in step S310.If so, then step S360 is entered, otherwise, enter step S340.
Step S340, based on the hierarchical relationship of each meta data server, according to the view location accessing view information and the target metadata server stored in meta data server, target metadata server is searched step by step from access meta data server, to determine the access to netwoks address information of target metadata server, then enter step S350.Process and the step S250 of this step are similar, no longer launch explanation at this.
Step S350, the content metadata that user will read is distributed to access meta data server according to this read requests by the access to netwoks address information request target metadata server of based target meta data server.
Step S360, the metadata that client 30 will read from access meta data server reading user.
In sum, by access meta data server be not target metadata server time, from access meta data server search target metadata server step by step to determine the access to netwoks address information of target metadata server, the data volume of the view information that the single meta data server of so good minimizing stores, thus improve search efficiency.
Those skilled in the art should be understood that, above-mentioned of the present invention each module or each step can realize with general calculation element, they can concentrate on single calculation element, or be distributed on network that multiple calculation element forms, alternatively, they can realize with the executable program code of calculation element, thus, they can be stored and be performed by calculation element in the storage device, or they are made into each integrated circuit modules respectively, or the multiple module in them or step are made into single integrated circuit module to realize.Like this, the present invention is not restricted to any specific hardware and software combination.
Although the execution mode disclosed by the present invention is as above, the execution mode that described content just adopts for the ease of understanding the present invention, and be not used to limit the present invention.Technical staff in any the technical field of the invention; under the prerequisite not departing from the spirit and scope disclosed by the present invention; any amendment and change can be done what implement in form and in details; but scope of patent protection of the present invention, the scope that still must define with appending claims is as the criterion.
Claims (9)
1. a cloud storage system, is characterized in that, comprises metadata server cluster, view management module and client, wherein:
Client is used for providing the interface of accessing described cloud storage system to user, and parses the view location of the meta data server that user will access according to the access request of user;
View management module is for storing whole Metadata View The information of whole metadata server cluster, according to the logic association relation between each meta data server in described whole Metadata View The information architecture metadata server cluster, and according to the Metadata View The information that this logic association relation self is associated to each meta data server distribution with this meta data server;
In metadata server cluster, each meta data server is for storing the Metadata View The information of being distributed by view management module, and the view location of the meta data server sent according to client or other meta data server determines the access to netwoks address information of the meta data server corresponding with the view location of this meta data server;
Multiple meta data server layerings of described metadata server cluster are disposed, be in the meta data server of the bottom in described metadata server cluster also for the Metadata View The information that storing metadata content and view management module are distributed, and according to user's request, the metadata information that self stores be supplied to user.
2. cloud storage system according to claim 1, is characterized in that, each meta data server of described metadata server cluster all only stores the part content metadata of whole described cloud storage system.
3. cloud storage system according to claim 1, is characterized in that, described view management module comprises Data Management Unit, information process unit and content view unit further, wherein:
Data Management Unit, for storing whole Metadata View The information of described cloud storage system;
Information process unit, for the whole Metadata View The information stored according to described Data Management Unit, build the logic association relation of each meta data server in described metadata cluster, and Metadata View The information is processed, obtain Metadata View The information, and distribute described Metadata View The information to each meta data server, make each meta data server store the view information self be associated with this meta data server;
Content view unit, when the content metadata for the file stored in a meta data server in described cloud storage system is accessed by another meta data server continually, writes this another meta data server by this file.
4. cloud storage system according to claim 3, is characterized in that:
Described Data Management Unit also correspondingly stores the complete Metadata View The information corresponding respectively with each default view strategy of many covers according to different default view strategies; And,
Described view management module comprises decision package further, for client based on metadata access request and described default view strategy determine multiple for access target metadata server time, by described multiple meta data server being defined as final access for one of access destination meta data server.
5. cloud storage system according to claim 4, is characterized in that, described view management module comprises receiving element further, wherein:
Described receiving element is used for the load information of the meta data server receiving each bottom from each meta data server;
Described information process unit is also predicted based on the load information of meta data server, obtains the load estimation information of the meta data server in next period;
Described decision package, for client based on metadata access request and described default view strategy determine multiple for access target metadata server time, according to described load estimation information by described multiple meta data server being defined as final access for one of access destination meta data server.
6. cloud storage system according to claim 5, is characterized in that, described decision package carries out following load balancing control further:
When described information process unit there occurs fault based on described load estimation INFORMATION DISCOVERY existence or does not have the meta data server responded to send out, described decision package upgrades the Metadata View The information of the meta data server associated with the meta data server broken down based on whole Metadata View The information of the described cloud storage system stored in described Data Management Unit, with no longer to breaking down or not having the meta data server responded to send metadata access request; And/or
When there is the excessive meta data server of load by information process unit based on load estimation INFORMATION DISCOVERY, described decision package indicate the excessive meta data server of this load to stop providing Metadata Service or by all or part of metadata replication to the little meta data server of load in described cloud storage system, or described decision package triggers the mirror image of the excessive meta data server of generation described load on the meta data server that load is little.
7. a metadata wiring method for cloud storage system, is characterized in that,
Client receive user to write the write request of metadata to be written time, according to default access strategy determine access access access meta data server, said write request comprises content metadata to be written;
Described client carries out preliminary treatment to said write request, to obtain the view location of the target metadata server for depositing described content metadata to be written, wherein, if described client saves the many cover Metadata View The corresponding respectively with multiple default view strategy, then when each cover Metadata View The information of described client corresponding to various described default view strategy determines the view location of multiple target metadata server, one of view location of the plurality of target metadata server is defined as the view location of final target metadata server,
If the meta data server that the view location of determined target metadata server is corresponding is described access meta data server, then the access to netwoks address information of described access meta data server is defined as the access to netwoks address information of target metadata server, otherwise, based on the hierarchical relationship of each meta data server in described metadata cluster, according to the view location of the Metadata View The information stored in described access meta data server and target metadata server, target metadata server is searched step by step from described access meta data server, to determine the access to netwoks address information of target metadata server,
According to the access to netwoks address information of determined target metadata server, this metadata to be written is write described target metadata server.
8. method according to claim 7, it is characterized in that, described default access strategy is physics is geographically defined as described access meta data server from the meta data server that described user is nearest or the Metadata View The information corresponding according to described default view strategy is confirmed as the fastest meta data server of described user access being defined as described access meta data server.
9. a method for the metadata reading of cloud storage system, is characterized in that:
The expectation that client receives user is read when the read requests of metadata, determines the access meta data server of accessing access according to default access strategy;
Described client records according to the metadata write that described read requests and described client are preserved and Metadata View The information is determined to house the view location that user expects the target metadata server of the metadata read, wherein, if described client saves the many cover Metadata View The corresponding respectively with multiple default view strategy, then when each cover Metadata View The information of described client corresponding to various described default view strategy determines the view location of multiple target metadata server, one of view location of the plurality of target metadata server is defined as the view location of final target metadata server,
If the meta data server that the view location of determined target metadata server is corresponding is not described access meta data server, then based on the hierarchical relationship of meta data server each in described cloud storage system, according to the view location accessing Metadata View The information and the target metadata server stored in meta data server, target metadata server is searched step by step from access meta data server, to determine the access to netwoks address information of target metadata server, and the content metadata that user will read is distributed to access meta data server according to described read requests by target metadata server described in the access to netwoks address information request of based target meta data server,
Described client reads from described access meta data server the metadata that described user will read.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210299624.9A CN102833331B (en) | 2012-08-21 | 2012-08-21 | Cloud storage system and metadata writing-in method and metadata reading method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210299624.9A CN102833331B (en) | 2012-08-21 | 2012-08-21 | Cloud storage system and metadata writing-in method and metadata reading method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102833331A CN102833331A (en) | 2012-12-19 |
| CN102833331B true CN102833331B (en) | 2015-06-10 |
Family
ID=47336298
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210299624.9A Active CN102833331B (en) | 2012-08-21 | 2012-08-21 | Cloud storage system and metadata writing-in method and metadata reading method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102833331B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103368966B (en) * | 2013-07-18 | 2016-08-17 | 北京网鼎芯睿科技有限公司 | Server system and the transmission method of data stream |
| CN107305582B (en) * | 2016-04-25 | 2020-05-08 | 华为技术有限公司 | Metadata processing method and device |
| CN105893630A (en) * | 2016-05-26 | 2016-08-24 | 广州迈安信息科技有限公司 | Distributed data service system based on meta data |
| CN106131227A (en) * | 2016-08-31 | 2016-11-16 | 浪潮(北京)电子信息产业有限公司 | Balancing method of loads, meta data server system and load balance system |
| CN106649601A (en) * | 2016-11-24 | 2017-05-10 | 郑州云海信息技术有限公司 | Method and system for processing data of file system as well as client and server |
| CN106777370A (en) * | 2017-01-24 | 2017-05-31 | 桂林电子科技大学 | A kind of oceanographic data collection and metadata management method |
| CN109309694A (en) * | 2017-07-27 | 2019-02-05 | 杭州海康威视数字技术股份有限公司 | A kind of method and system of data storage |
| CN114491111B (en) * | 2022-02-16 | 2022-09-16 | 北京中电兴发科技有限公司 | Distributed metadata system for picture storage |
| CN114610725B (en) * | 2022-05-11 | 2022-08-16 | 成都九洲电子信息系统股份有限公司 | Computer equipment, application method thereof and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158546A (en) * | 2011-02-28 | 2011-08-17 | 中国科学院计算技术研究所 | Cluster file system and file service method thereof |
| CN102243660A (en) * | 2011-07-18 | 2011-11-16 | 中兴通讯股份有限公司 | Data access method and device |
| WO2012024801A1 (en) * | 2010-08-25 | 2012-03-01 | Rayan Zachariassen | Method and system for cache tiering |
-
2012
- 2012-08-21 CN CN201210299624.9A patent/CN102833331B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012024801A1 (en) * | 2010-08-25 | 2012-03-01 | Rayan Zachariassen | Method and system for cache tiering |
| CN102158546A (en) * | 2011-02-28 | 2011-08-17 | 中国科学院计算技术研究所 | Cluster file system and file service method thereof |
| CN102243660A (en) * | 2011-07-18 | 2011-11-16 | 中兴通讯股份有限公司 | Data access method and device |
Non-Patent Citations (1)
| Title |
|---|
| 《空间数据存储对象的元数据可伸缩性管理》;李锐等;《计算机应用研究》;20111231;第28卷(第12期);4567-4571 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102833331A (en) | 2012-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102833331B (en) | Cloud storage system and metadata writing-in method and metadata reading method thereof | |
| CN103067461B (en) | A kind of metadata management system of file and metadata management method | |
| CN110213352B (en) | Method for aggregating dispersed autonomous storage resources with uniform name space | |
| EP2638476B1 (en) | Method and apparatus of accessing data of virtual machine | |
| CN104850358B (en) | A kind of magneto-optic electricity mixing storage system and its data acquisition and storage method | |
| US9996552B2 (en) | Method for generating a dataset structure for location-based services and method and system for providing location-based services to a mobile device | |
| US20110153606A1 (en) | Apparatus and method of managing metadata in asymmetric distributed file system | |
| CN113672175A (en) | Distributed object storage method, device and equipment and computer storage medium | |
| US20070179981A1 (en) | Efficient data management in a cluster file system | |
| CN101741907A (en) | Method and system for balancing server load and main server | |
| CN109933312B (en) | Method for effectively reducing I/O consumption of containerized relational database | |
| CN104615606A (en) | Hadoop distributed file system and management method thereof | |
| CN117008818A (en) | Data processing method, apparatus, computer device, and computer readable storage medium | |
| US9110820B1 (en) | Hybrid data storage system in an HPC exascale environment | |
| CN106570113B (en) | Mass vector slice data cloud storage method and system | |
| CN104270412A (en) | Three-level caching method based on Hadoop distributed file system | |
| CN104750757B (en) | A kind of date storage method and equipment based on HBase | |
| US20150169623A1 (en) | Distributed File System, File Access Method and Client Device | |
| KR101341412B1 (en) | Apparatus and method of controlling metadata in asymmetric distributed file system | |
| US10534765B2 (en) | Assigning segments of a shared database storage to nodes | |
| CN111737052A (en) | Distributed object storage system and method | |
| CN101783814A (en) | Metadata storing method for mass storage system | |
| CN113204520B (en) | Remote sensing data rapid concurrent read-write method based on distributed file system | |
| CN114647388B (en) | Distributed block storage system and management method | |
| CN103389986B (en) | Storage, the lookup method of session Session information, Apparatus and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20170807 Address after: No. 555 Kang Hangzhou Science Park of Zhejiang province Binjiang District Qianmo road 310053 Patentee after: Hangzhou Hikvision Digital Technology Co., Ltd. Address before: 100085, room 101, block B, Zhongguancun development building, No. 12 information road, Beijing, Haidian District Patentee before: Bangnuo Memory Science and Technology Co., Ltd., Beijing |
|
| TR01 | Transfer of patent right |