CN106230998A - A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device - Google Patents
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device Download PDFInfo
- Publication number
- CN106230998A CN106230998A CN201610874206.6A CN201610874206A CN106230998A CN 106230998 A CN106230998 A CN 106230998A CN 201610874206 A CN201610874206 A CN 201610874206A CN 106230998 A CN106230998 A CN 106230998A
- Authority
- CN
- China
- Prior art keywords
- data
- bitmap
- grades
- copy
- virtual block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
nullThe invention discloses a kind of many Back end data synchronization scenario based on TCMU Virtual Block Device，Including front end TCMU Virtual Block Device and back-end storage device，Back-end storage device includes three data trnascriptions，Three data trnascriptions of the Virtual Block Device opposite rear end of front end，The front end applications of the present invention is TCMU Virtual Block Device，Its small letter IO is 4K，And data 4K are alignd，Bitmap is used to record the synchronous situation of data，Memory consumption is smaller，But，If only using one-level bitmap，In bitmap 1 when representing 4K data，Bitmap consumes internal memory the most very much，And the time scanning one time is more long，If in bitmap 1 when representing larger data，Such as 4M，EMS memory occupation can be much smaller，If but front end only have modified 4K data，Again must re-synchronization 4M data，So can reduce speed，Affect network and Disk bandwidth.The present invention both advantage comprehensive, uses two grades of bitmap, i.e. takies little internal memory, the most do not affect synchronizing speed, network and Disk bandwidth.
Description
Technical field
The present invention relates to many Back end data simultaneous techniques field, a kind of many rear ends based on TCMU Virtual Block Device
Data syn-chronization scheme.
Background technology
Along with the arriving of big data age, need storage sea based on the Internet or ISP based on communication network
The data of amount, to support that it is runed, meanwhile, ISP, by the analysis to mass data, can provide a user with more convenient
And have more personalized service, thus reach to improve the purpose of its service level.
At present, generally being stored the data of magnanimity by distributed memory system, such as, server cluster is divided by technical staff
Distribution is put, and the data of magnanimity is stored in different servers according to some regular-scattered；These servers generally can root
According to the request of user, provide the user the service such as data query, data renewal, in order to improve the safety and reliability of data,
Distributed memory system is further typically provided with a number of backup server, for Backup Data.
Distributed memory system, is data dispersion to be stored on the equipment of many platform independent, traditional network store system
The all data of storage server repository concentrated, storage server are used to become the bottleneck of systematic function, be also reliability and peace
The focus of full property, it is impossible to meet the needs of Mass storage application, distributed network storage system uses extendible system knot
Structure, utilizes multiple stage storage server to share storage load, utilizes location server to position storage information, and it not only increases system
Reliability, availability and access efficiency, be also easy to extension.
Existing distributed memory system is all by network service, and the unstability of network easily causes Back end data not
Unanimously, if data syn-chronization can not quickly be carried out, data integrity and the high availability of distributed memory system are just substantially reduced,
Prior art data syn-chronization is the slowest, takies the too high network bandwidth, and is extremely difficult to data consistent in the case of writing IO
Property.
Summary of the invention
The technical problem to be solved in the present invention is to overcome existing defect, it is provided that a kind of many based on TCMU Virtual Block Device
Back end data synchronization scenario, the advantage taking little internal memory in conjunction with TCMU Virtual Block Device and two grades of bitmap so that this base
Many Back end data synchronization scenario in TCMU Virtual Block Device has at high speed, low memory consumption, low network band width consumption, low magnetic
The feature of dish bandwidth consumption, can effectively solve the problem in background technology.
For achieving the above object, the present invention provides following technical scheme: a kind of many rear ends based on TCMU Virtual Block Device
Data syn-chronization scheme, including front end TCMU Virtual Block Device and back-end storage device, described back-end storage device includes three
Data trnascription, three data trnascriptions of the Virtual Block Device opposite rear end of front end.
As a preferred technical solution of the present invention, it is assumed that size of data is 40G, copy 2 loss of data, copy 1 to
Copy 2 carries out data syn-chronization, and the concrete process step of data syn-chronization is as follows:
Copy 1 initializes one-level bitmap, and each represents 4M data, and totally 10240, initialization value is all 1.
Copy 1, according to the order of one-level bitmap, is given out a contract for a project and is asked the copy 2 data cryptographic Hash to position.
As a preferred technical solution of the present invention, copy 2 wraps the side-play amount and data length carried, meter according to request
The cryptographic Hash of the evidence that counts, then returns bag.
As a preferred technical solution of the present invention, after copy 1 receives time bag of copy 2, calculate the Kazakhstan of corresponding data
Uncommon value, the cryptographic Hash then carried with 2 times bags of copy compares, if equal, data need not synchronize, and one-level bitmap is corresponding
Position 0, if unequal, sends synchronization packets, one-level bitmap correspondence position 0.
As a preferred technical solution of the present invention, copy 2 receives synchronization packets, writes the data to, and returns bag and returns
Write successfully or failure.
If copy 1 receives copy 2 and returns failure, one-level bitmap correspondence position is re-set as 1, represents and need weight
New synchronization, if there being two grades of bitmap of correspondence, deletes this two grades of bitmap.
As a preferred technical solution of the present invention, during synchronizing due to Back end data, front end also has data to write
Entering, use two grades of bitmap to record the data being modified, front 16M data have synchronously completed, if the number of the write of front end
According to side-play amount more than 16M, corresponding one-level bitmap is 1, represents and does not synchronizes, thus between write data, need not give birth to
Becoming two grades of corresponding bitmap, if the data offset of the write of front end is less than 16M, corresponding one-level bitmap is 0, represents
Synchronized, now needed to generate two grades of corresponding bitmap, the data that record has been modified.Two grades of bitmap each
Representing 4K data, totally 1024, two grades of bitmap are saved in hash map, and key assignments is the call number of one-level bitmap.
As a preferred technical solution of the present invention, repeat above step, until one-level bitmap is all 0, then travel through
Two grades of bitmap, assemble synchronization packets according to two grades of bitmap, and two grades of bitmap need not first verify the Hash of data in synchronizing
Value, copy 2 returns when writing successfully, and corresponding two grades of bitmap positions 0, if one the two grades all positions of bitmap are all 0, are deleted
These two grades of bitmap.
As a preferred technical solution of the present invention, repeat the above steps, until only existing two grades of bitmap of minority,
Now suspending front-end processing and write IO, repeat the above steps again, until two grades of bitmap are empty, data syn-chronization completes, and again opens
IO is write in the front end that IO and startup copy 2 are write in dynamic process front end.
Compared with prior art, the invention has the beneficial effects as follows: these many Back end data based on TCMU Virtual Block Device are same
The front end applications of step scheme is TCMU Virtual Block Device, and its small letter IO is 4K, and the alignment of data 4K, uses bitmap to remember
The synchronous situation of record data, memory consumption is smaller, but, if only using one-level bitmap, in bitmap, 1 represents 4K number
According to time, bitmap consumes internal memory the most very much, and the time scanning one time is relatively more long, if in bitmap 1 represent plurality
According to time, such as 4M, EMS memory occupation can be much smaller, but if front end only have modified 4K data, again must re-synchronization 4M data, knot
The advantage that conjunction TCMU Virtual Block Device and two grades of bitmap take little internal memory so that should be many based on TCMU Virtual Block Device
Back end data synchronization scenario has at high speed, low memory consumption, low network band width consumption, the feature that low Disk bandwidth consumes.
Accompanying drawing explanation
Fig. 1 is the distributed memory system structural representation of the present invention；
Fig. 2 is the one-level bitmap structural representation of the present invention；
Fig. 3 is two grades of bitmap structural representations of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Describe, it is clear that described embodiment is only a part of embodiment of the present invention rather than whole embodiments wholely.Based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under not making creative work premise
Embodiment, broadly falls into the scope of protection of the invention.
Referring to Fig. 1-3, the present invention provides a kind of technical scheme: a kind of many Back end data based on TCMU Virtual Block Device
Synchronization scenario, including front end TCMU Virtual Block Device and back-end storage device, described back-end storage device includes three data
Copy, three data trnascriptions of the Virtual Block Device opposite rear end of front end.
Assuming that size of data is 40G, copy 2 loss of data, copy 1 carries out data syn-chronization to copy 2, the tool of data syn-chronization
The process step of body is as follows:
Step 1: as in figure 2 it is shown, copy 1 initializes one-level bitmap, each represents 4M data, totally 10240, initialization value
It is all 1.
Step 2: copy 1, according to the order of one-level bitmap, is given out a contract for a project and asked the copy 2 data cryptographic Hash to position.
Step 3: copy 2 wraps the side-play amount and data length carried according to request, calculates the cryptographic Hash of data, then returns
Bag.
Step 4: after copy 1 receives time bag of copy 2, calculate the cryptographic Hash of corresponding data, then carry with 2 times bags of copy
Cryptographic Hash compare, if equal, data need not synchronize, one-level bitmap correspondence position 0, if unequal, send with
Step data bag, one-level bitmap correspondence position 0.
Step 5: copy 2 receives synchronization packets, writes the data to, returns bag return and writes successfully or failure.
Step 6: if copy 1 receives copy 2 and returns failure, is re-set as 1 one-level bitmap correspondence position, represents
Needing re-synchronization, if there being two grades of bitmap of correspondence, deleting this two grades of bitmap.
Step 7: during synchronizing due to Back end data, front end also has data to write, and uses two grades of bitmap to carry out record
The data being modified, as it is shown on figure 3, front 16M data have synchronously completed, if the data offset of the write of front end is more than
16M, corresponding one-level bitmap is 1, represents and does not synchronizes, so between write data, two grades of correspondence need not be generated
Bitmap, if the data offset of the write of front end is less than 16M, corresponding one-level bitmap is 0, represents and had synchronized,
Now need to generate two grades of corresponding bitmap, the data that record has been modified, two grades of bitmap each represent 4K data,
Totally 1024, two grades of bitmap are saved in hash map, and key assignments is the call number of one-level bitmap.
Step 8: repeat above step, until one-level bitmap is all 0, then travels through two grades of bitmap, according to two grades
Bitmap assembles synchronization packets, and two grades of bitmap need not first verify the cryptographic Hash of data in synchronizing, copy 2 returns and writes successfully
Time, corresponding two grades of bitmap positions 0, if one the two grades all positions of bitmap are all 0, delete this two grades of bitmap.
Step 9: repeat step 8, until only existing two grades of bitmap of minority, now suspending front-end processing and writing IO, again
Repeating step 8, until two grades of bitmap be empty, data syn-chronization completes, and is again started up processing before front end writes IO and startup copy 2
End writes IO.
The present invention propose a kind of quickly, memory cost is little, save block device many Back end data synchronization scenario of the network bandwidth,
Front end applications is TCMU Virtual Block Device, and its small letter IO is 4K, and the alignment of data 4K, uses bitmap to record data
Synchronous situation, memory consumption is smaller, but, if only using one-level bitmap, in bitmap 1 when representing 4K data,
Bitmap consumes internal memory the most very much, and the time scanning one time is relatively more long, if in bitmap 1 when representing larger data,
Such as 4M, EMS memory occupation can be much smaller, but if front end only have modified 4K data, necessary re-synchronization 4M data, so can again
Reduce speed, affect network and Disk bandwidth, the present invention comprehensively both advantage, use two grades of bitmap, i.e. take little interior
Deposit, the most do not affect synchronizing speed, network and Disk bandwidth.
Although an embodiment of the present invention has been shown and described, for the ordinary skill in the art, permissible
Understand and these embodiments can be carried out multiple change without departing from the principles and spirit of the present invention, revise, replace
And modification, the scope of the present invention be defined by the appended.
Claims (8)
1. many Back end data synchronization scenario based on TCMU Virtual Block Device, including front end TCMU Virtual Block Device and rear end
Storage device, it is characterised in that: described back-end storage device includes three data trnascriptions, after the Virtual Block Device correspondence of front end
Three data trnascriptions of end.
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device the most according to claim 1, its feature exists
In: assuming that size of data is 40G, copy 2 loss of data, copy 1 carries out data syn-chronization to copy 2, data syn-chronization concrete
Process step is as follows:
Copy 1 initializes one-level bitmap, and each represents 4M data, and totally 10240, initialization value is all 1；
Copy 1, according to the order of one-level bitmap, is given out a contract for a project and is asked the copy 2 data cryptographic Hash to position.
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device the most according to claim 2, its feature exists
In: copy 2 wraps the side-play amount and data length carried according to request, calculates the cryptographic Hash of data, then returns bag.
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device the most according to claim 3, its feature exists
In: after copy 1 receives time bag of copy 2, calculate the cryptographic Hash of corresponding data, then wrap the cryptographic Hash ratio carried 2 times with copy
Relatively, if equal, data need not synchronize, and one-level bitmap correspondence position 0, if unequal, sends synchronization packets,
One-level bitmap correspondence position 0.
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device the most according to claim 4, its feature exists
In: copy 2 receives synchronization packets, writes the data to, and returns bag return and writes successfully or failure；
If copy 1 receives copy 2 and returns failure, one-level bitmap correspondence position is re-set as 1, represents that needs are again same
Step, if there being two grades of bitmap of correspondence, deletes this two grades of bitmap.
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device the most according to claim 5, its feature exists
In: during synchronizing due to Back end data, front end also has data to write, and uses two grades of bitmap to record the number being modified
According to, front 16M data have synchronously completed, if the data offset of the write of front end more than 16M, corresponding one-level bitmap is
1, represent and do not synchronize, so between write data, two grades of bitmap of correspondence need not be generated, if the write of front end
Data offset is less than 16M, and corresponding one-level bitmap is 0, represents and had synchronized, and now needs to generate corresponding two grades
Bitmap, the data that have been modified of record, two grades of bitmap each represent 4K data, totally 1024, two grades of bitmap protect
Existing in hash map, key assignments is the call number of one-level bitmap.
7. according to a kind of based on TCMU Virtual Block Device the many Back end data synchronization scenario described in claim 1-6, its feature
It is: repeat above step, until one-level bitmap is all 0, then travels through two grades of bitmap, assemble according to two grades of bitmap and synchronize
Packet, two grades of bitmap need not first verify the cryptographic Hash of data in synchronizing, copy 2 returns when writing successfully, corresponding two grades
Bitmap position 0, if one the two grades all positions of bitmap are all 0, deletes this two grades of bitmap.
A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device the most according to claim 7, its feature exists
In: repeat the above steps, until only existing two grades of bitmap of minority, now suspending front-end processing and writing IO, again repeating above-mentioned
Step, until two grades of bitmap be empty, data syn-chronization completes, and is again started up processing front end and writes the front end of IO and startup copy 2 and write
IO。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610874206.6A CN106230998A (en) | 2016-10-08 | 2016-10-08 | A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610874206.6A CN106230998A (en) | 2016-10-08 | 2016-10-08 | A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106230998A true CN106230998A (en) | 2016-12-14 |
Family
ID=58076824
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610874206.6A Pending CN106230998A (en) | 2016-10-08 | 2016-10-08 | A kind of many Back end data synchronization scenario based on TCMU Virtual Block Device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106230998A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113726861A (en) * | 2021-08-18 | 2021-11-30 | 华云数据控股集团有限公司 | Cloud hard disk data synchronization method and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101656094A (en) * | 2009-09-25 | 2010-02-24 | 杭州华三通信技术有限公司 | Data storage method and storage device |
| CN201689880U (en) * | 2009-12-31 | 2010-12-29 | 国民技术股份有限公司 | Safe chip |
| US7991748B2 (en) * | 2003-09-23 | 2011-08-02 | Symantec Corporation | Virtual data store creation and use |
| CN104156393A (en) * | 2014-07-09 | 2014-11-19 | 深圳市中博科创信息技术有限公司 | Method for creating virtual block device and distributed file system |
-
2016
- 2016-10-08 CN CN201610874206.6A patent/CN106230998A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7991748B2 (en) * | 2003-09-23 | 2011-08-02 | Symantec Corporation | Virtual data store creation and use |
| CN101656094A (en) * | 2009-09-25 | 2010-02-24 | 杭州华三通信技术有限公司 | Data storage method and storage device |
| CN201689880U (en) * | 2009-12-31 | 2010-12-29 | 国民技术股份有限公司 | Safe chip |
| CN104156393A (en) * | 2014-07-09 | 2014-11-19 | 深圳市中博科创信息技术有限公司 | Method for creating virtual block device and distributed file system |
Non-Patent Citations (1)
| Title |
|---|
| HALEYCOMET: ""tcmu design"", 《HTTPS:https://BLOG.CSDN.NET/HALEYCOMET/ARTICLE/DETAILS/52344086》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113726861A (en) * | 2021-08-18 | 2021-11-30 | 华云数据控股集团有限公司 | Cloud hard disk data synchronization method and electronic equipment |
| CN113726861B (en) * | 2021-08-18 | 2024-02-09 | 华云数据控股集团有限公司 | Cloud hard disk data synchronization method and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105187464B (en) | Method of data synchronization, apparatus and system in a kind of distributed memory system | |
| CN106066896B (en) | Application-aware big data deduplication storage system and method | |
| CN102821138B (en) | Metadata distributed storage method applicable to cloud storage system | |
| CN106294585A (en) | A kind of storage method under cloud computing platform | |
| CN103873505B (en) | Method, system and the device uploaded offline to cloud storage service device | |
| CN105183400B (en) | It is a kind of based on content addressed object storage method and system | |
| US10310904B2 (en) | Distributed technique for allocating long-lived jobs among worker processes | |
| CN103034684A (en) | Optimizing method for storing virtual machine mirror images based on CAS (content addressable storage) | |
| CN103020315A (en) | Method for storing mass of small files on basis of master-slave distributed file system | |
| US20160140201A1 (en) | Facilitating distributed deletes in a replicated storage system | |
| CN106326239A (en) | Distributed file system and file meta-information management method thereof | |
| CN104408111A (en) | Method and device for deleting duplicate data | |
| CN103942259B (en) | A kind of method that data buffer storage is realized in database synchronization | |
| CN102904948A (en) | Super-large-scale low-cost storage system | |
| CN109407977A (en) | A kind of big data distributed storage management method and system | |
| CN106570113B (en) | Mass vector slice data cloud storage method and system | |
| CN106953910A (en) | A kind of Hadoop calculates storage separation method | |
| CN101923571B (en) | Method and device for managing terminal data logging | |
| CN109522283A (en) | A kind of data de-duplication method and system | |
| CN105677251A (en) | Storage system based on Redis cluster | |
| CN103501319A (en) | Low-delay distributed storage system for small files | |
| CN110058964A (en) | Data reconstruction method, data recovery system and computer program product | |
| CN104899161B (en) | A kind of caching method of the continuous data protection based on cloud storage environment | |
| CN110083306A (en) | A kind of distributed objects storage system and storage method | |
| CN108540510A (en) | A kind of cloud host creation method, device and cloud service system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20161214 |