KR101875763B1 - The database management system and method for preventing performance degradation of transaction when table reconfiguring - Google Patents
The database management system and method for preventing performance degradation of transaction when table reconfiguring Download PDFInfo
- Publication number
- KR101875763B1 KR101875763B1 KR1020160095225A KR20160095225A KR101875763B1 KR 101875763 B1 KR101875763 B1 KR 101875763B1 KR 1020160095225 A KR1020160095225 A KR 1020160095225A KR 20160095225 A KR20160095225 A KR 20160095225A KR 101875763 B1 KR101875763 B1 KR 101875763B1
- Authority
- KR
- South Korea
- Prior art keywords
- partition
- journal file
- synchronization
- record
- journal
- Prior art date
Links
Images
Classifications
-
- G06F17/30097—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to an online database management system and method for minimizing the performance degradation of a transaction when a table is reorganized, and in an online database management system for reorganizing a table, the system is divided into a plurality of partitions and a partition hash map A partition hash map configuration module to store; A partition selection module for selecting a first partition to be moved among a plurality of partitions partitioned by the partition hash map configuration module; A journal file generation module that generates a synchronization journal file that records a record insertion / deletion / change history in the first partition selected by the partition selection module; A partition transfer module for transferring the at least one second partition to be migrated to the first partition where the synchronization journal file is generated in the journal file generation module; A partition deletion module for deleting a second partition when transfer of the second partition to the first partition is completed; And a journal file deletion module for deleting the synchronization journal file generated in the first partition. [5] The present invention relates to an online database management system and method for minimizing the performance degradation of a transaction during table reorganization. According to the present invention, it is possible to provide an online database management system and a method for minimizing a performance degradation of a transaction when a table is reconstructed, by omitting a record deletion process by dividing a plurality of partitions in advance at the time of table creation.
Description
The present invention relates to an online database management system and method for minimizing the performance degradation of a transaction when a table is reorganized. More particularly, the present invention relates to an online database management system and method, The present invention relates to an online database management system and method for minimizing the performance degradation of a transaction.
The present invention relates to a database management system.
The process of reorganizing tables in an existing database management system is as described in Fig.
The existing table reorganization consists of a process similar to partitioning of partitioned tables. In a nutshell, the existing table reorganization process consists of adding a new partition to the table, and transferring the records that match the condition from the existing partition to the new partition.
The cluster database management system must do this online. That is, even if the table reorganization process is in progress, the transaction that accesses the table must be allowed.
Existing systems use synchronization triggers to solve these problems. Synchronization triggers are created in the destination table after adding a new partition and deleted when the record transfer to the new partition is complete. If a transaction changes records during the table reorganization process, it is remotely reflected.
The existing technology causes table fragmentation because the records stored in the existing partition are transferred to the new partition. The transfer of the record is made from the existing partition to the new partition, and it is deleted after the point of time when the transfer ends. Even if you delete a record, the entire table space is not shrunk, and you can not reuse records that have been deleted from another table. Therefore, existing technologies using partitioning have a problem of low space efficiency.
The use of synchronization triggers also has the problem of temporarily degrading the performance of service transactions. If a record insert / delete / change operation occurs during the table reorganization, it must be synchronized with the record transferred remotely. The synchronization process is performed by a trigger and involves a remote communication process, so that the performance of a service transaction is significantly degraded.
The present invention has been made to solve the above problems and it is an object of the present invention to provide an on-line data recording method and a data recording method capable of omitting a record deletion process by dividing a plurality of partitions at the time of table creation, Base management system and method.
It is another object of the present invention to provide a method and system for performing a service transaction by collectively processing a table reconfiguration transaction, unlike a trigger performed by a service transaction, using a journaling technique capable of minimizing performance degradation of a service transaction for data synchronization An object of the present invention is to provide an online database management system and method that minimizes the performance degradation of a transaction when a table is reconfigured to prevent remote access for data synchronization.
According to an aspect of the present invention, there is provided an online database management system for restructuring a table, the system comprising: A partition hash map configuration module for dividing the partition into a plurality of partitions and configuring and storing the partition hash map; A partition selection module for selecting a first partition to be moved among a plurality of partitions partitioned by the partition hash map configuration module; A journal file generation module that generates a synchronization journal file that records a record insertion / deletion / change history in the first partition selected by the partition selection module; A partition transfer module for transferring the at least one second partition to be migrated to the first partition where the synchronization journal file is generated in the journal file generation module; A partition deletion module for deleting a second partition when transfer of the second partition to the first partition is completed; And a journal file deletion module for deleting the synchronization journal file generated in the first partition.
In addition, the synchronization journal file is a set of journal records, and journal records are recorded one by one whenever a transaction inserts / deletes / changes a record.
The partition hash map may include a hash value of a key column of each partition and a node number corresponding to a hash value of the key column.
In addition, the pre-partition module may include a journal file synchronization module that synchronizes the synchronization journal file in real time with the first partition.
In addition, when the second partition is deleted, the deleted second partition is returned to the table space.
According to an aspect of the present invention, there is provided an online database management method for restructuring a table, the method comprising: (a) Partitioning the partition into a plurality of partitions and constructing and storing a partition hash map; (b) selecting a first partition to be moved among the plurality of partitions partitioned in step (a); (c) generating a synchronization journal file to record a record insert / delete / change history in the first partition selected in step (b); (d) migrating the at least one second partition to be migrated to the first partition where the synchronization journal file is created in step (c); (e) deleting the second partition when migration of the second partition to the first partition is completed; And (f) deleting the synchronization journal file generated in the first partition.
In addition, the synchronization journal file is a set of journal records, and journal records are recorded one by one whenever a transaction inserts / deletes / changes a record.
The partition hash map may include a hash value of a key column of each partition and a node number corresponding to a hash value of the key column.
The step (d) may further comprise: (d1) synchronizing the synchronization journal file in real time with the first partition.
In the step (e), when the second partition is deleted, the deleted second partition is returned to the table space.
As described above, according to the present invention, it is possible to provide an online database management system and method for minimizing the performance degradation of a transaction when a table is reconstructed, in which a plurality of partitions are divided in advance before a table is created, have.
In addition, according to the present invention, by using a journaling technique that minimizes the performance degradation of service transactions for data synchronization, it is possible to perform batch processing by a table reconfiguration transaction, unlike a trigger performed by a service transaction, The present invention can provide an online database management system and method that minimizes a performance degradation of a transaction when a table is reconfigured to prevent a remote access to the database.
FIG. 1 is a flowchart showing a general table reconstruction process in the existing technology
2 is a block diagram illustrating a configuration of an online database management system that minimizes a performance degradation of a transaction when a table is reorganized according to an embodiment of the present invention.
FIG. 3 illustrates an online database management system and method for minimizing the performance degradation of a transaction when a table is reorganized according to an embodiment of the present invention. FIG. FIG.
FIG. 4 is a flowchart illustrating a procedure for providing an online database management system that minimizes performance degradation of a transaction when a table is reconstructed according to an embodiment of the present invention.
BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings.
The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described with reference to the accompanying drawings, which illustrate an online database management system and method for minimizing the performance degradation of a transaction when a table is reconstructed according to embodiments of the present invention.
The present invention relates to an online database management system for reconfiguring a table.
FIG. 2 is a block diagram illustrating a configuration of an online database management system that minimizes a performance degradation of a transaction when a table is reconfigured according to an embodiment of the present invention. FIG. In an online database management system and method in which a table is divided into 16 groups in two groups.
2 and 3, an on-line database management system that minimizes the performance degradation of a transaction when a table is reconstructed according to the present invention includes a partition hash map configuration module, a partition selection module, a journal file creation module, , And a journal file deletion module.
The partition hash map configuration module divides the table into a number of partitions and stores the partition hash map.
Here, the partition hash map includes a hash value of a key column of each partition and a node number corresponding to a hash value of the key column.
That is, the partition hash map is a data structure for managing the location of the node where each partition is stored, and has a hash value of a key column and a corresponding node number, and selects a partition through the node number It is possible to do.
The partition selection module selects a first partition to be moved among a plurality of partitions partitioned by the partition hash map configuration module.
Here, it is preferable that the size of the first partition is larger than that of the second partition.
The journal file generation module generates a synchronization journal file that records a record insertion / deletion / change history in the first partition selected by the partition selection module.
This is for recording the history of change such as record insertion / deletion / change operation of the corresponding partition by other transactions during the partition transfer process.
The logging of the synchronization journal file is performed by the corresponding transaction and has the advantage that the performance of ongoing transactions is degraded because it is not accompanied by a remote synchronization process.
The synchronization journal file is a set of journal records, and journal records are recorded one by one whenever a transaction inserts / deletes / changes a record.
It is desirable that information such as a transaction ID, a system change number (SCN), a changed record, and the like are recorded in the journal record.
The SCN is preferably used to select journal records to be remotely recorded during the journal file synchronization process of the journal file synchronization module.
For example, if a journal file synchronization transaction has migrated partitions to
Since the present invention uses a delay synchronization method using journaling, it is preferable that data synchronization is batch processed by a journal file synchronization transaction, unlike a trigger performed by a service transaction.
That is, assuming that a service transaction performs insert / delete / update operations on a partition that is being migrated, it writes the changes to the journal file without performing remote synchronization immediately after performing the partition. Partitioning through the previous module. It is preferable to perform the batch processing remotely.
The delayed synchronization scheme can minimize the performance degradation because the service transaction does not access remotely to synchronize the data, and can perform the partition transfer quickly because it is processed in a remote batch.
The pre-partition module transfers the at least one second partition to be migrated to the first partition where the synchronization journal file is generated in the journal file generation module.
In addition, the pre-partition module may include a journal file synchronization module that synchronizes the synchronization journal file in real time with the first partition.
Here, the journal synchronization module remotely synchronizes the journal data because a change of the partition may have occurred during the partition transfer process.
Also, during the journal file synchronization process, a change of the partition may be caused by another transaction.
The journal file synchronization module is repeatedly performed until a predetermined number of journal records to be recorded remain. When the condition is satisfied, a shared lock is acquired at the corresponding partition, and the remaining journal records are synchronized without synchronization It is preferable to record it as it is.
The journal file synchronization module is performed in a batch process and enables fast synchronization in real time by maximizing the network bandwidth.
In the partition deletion module, the journal file deletion module deletes the second partition when migration of the second partition to the first partition is completed.
In addition, when the second partition is deleted, the deleted second partition is returned to the table space.
That is, the deleted partition space can be immediately reused in another table, and unlike the existing technology, there is no need to use a separate space reclaim command.
The journal file deletion module deletes the synchronization journal file created in the first partition.
FIG. 4 is a flowchart illustrating a procedure for providing an online database management system that minimizes performance degradation of a transaction when a table is reconstructed according to an embodiment of the present invention.
Referring to FIG. 4, when creating a table, the partition is divided into a plurality of partitions, and a partition hash map is constructed and stored (S110).
Next, a first partition to be moved is selected among the plurality of partitions partitioned in step S110 (S120).
In step S130, a synchronization journal file is recorded to record a record insertion / deletion / change history in the first partition selected in step S120.
In step S140, the second partition to be migrated is migrated to the first partition where the synchronization journal file is created in step S140.
Preferably, the step (S140) includes synchronizing the synchronization journal file in real time with the first partition, and synchronizing the journal file in real time may include synchronizing It is preferable to acquire a shared lock on the partition and record the remaining journal records when the condition is satisfied.
Then, when the migration of the second partition to the first partition is completed, the second partition is deleted (S150).
Finally, the synchronization journal file created in the first partition is deleted (S160).
It will be understood by those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the foregoing detailed description, and all changes or modifications derived from the meaning and scope of the claims and the equivalents thereof are included in the scope of the present invention Should be interpreted.
100: Database Management System 110: Partition Hash Map Configuration Module
120: partition selection module 130: journal file generation module
140: pre-partition module 145: journal file synchronization module
150: Partition deletion module 160: Journal file deletion module
Claims (10)
A partition hash map configuration module that divides the table into a plurality of partitions and constructs and stores a partition hash map;
A partition selection module for selecting a first partition to be moved among a plurality of partitions partitioned by the partition hash map configuration module;
A journal file generation module that generates a synchronization journal file that records a record insertion / deletion / change history in the first partition selected by the partition selection module;
A partition pre-module for transferring the second partition to the first partition where the synchronization journal file is generated in the journal file generation module;
A partition deletion module for deleting the second partition when the migration of the second partition is completed in the first partition; And
A journal file deletion module for collectively deleting the synchronization journal files generated in the first partition; ≪ / RTI >
The synchronization journal file is a set of journal records that records a change history of insertion / deletion / change of a record of a corresponding partition by other transactions during a partition transfer process, and records a journal record Is recorded,
The journal record includes a transaction ID, a system change number (SCN), information of a changed record, and a record ID,
Wherein when the second partition is deleted, the deleted second partition is returned to the table space. In the table reconstruction,
The synchronization journal file records change contents when a record insertion / deletion / update operation is performed on a partition to which the transaction is being transferred, and stores the changed contents in the synchronization journal file, which is batch processed by the journal file synchronization transaction immediately before partitioning through the pre- Database management system.
Wherein the hash value of the key column in each partition and the node number corresponding to the hash value of the key column are minimized in the online database management system.
And a journal file synchronization module for synchronizing the synchronization journal file in real time with the first partition. The online database management system minimizes the performance degradation of a transaction when a table is reconfigured.
(a) partitioning a plurality of partitions into a partition hash map when the table is created;
(b) selecting a first partition to be moved among the plurality of partitions partitioned in step (a);
(c) generating a synchronization journal file to record a record insert / delete / change history in the first partition selected in step (b);
(d) migrating the at least one second partition to be migrated to the first partition where the synchronization journal file is created in step (c);
(e) deleting the second partition when migration of the second partition to the first partition is completed; And
(f) deleting the synchronization journal file generated in the first partition,
The synchronization journal file includes:
A journal record is a set of journal records, each time a transaction inserts / deletes / changes a record, a journal record is recorded one by one. During the partition transfer process, the record of the change in insertion / deletion / Whenever this record is inserted / deleted / changed, journal records are recorded one by one,
The journal record includes a transaction ID, a system change number (SCN), information of a changed record, and a record ID,
(D) synchronizing the synchronization journal file in real time with the first partition,
The synchronizing journal file records changes in the journal file when a record insertion / deletion / update operation is performed on a partition to which the transaction is being transferred, and performs batch processing by a journal file synchronization transaction immediately before partitioning through a pre- And,
Wherein the step (e) is performed when the second partition is deleted, and the deleted second partition is returned to the table space.
Wherein a hash value of a key column of each partition and a node number corresponding to a hash value of the key column are minimized.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020160095225A KR101875763B1 (en) | 2016-07-27 | 2016-07-27 | The database management system and method for preventing performance degradation of transaction when table reconfiguring |
| PCT/KR2016/008261 WO2018021593A1 (en) | 2016-07-27 | 2016-07-28 | Online database management system and method for minimizing performance degradation of transaction when reconfiguring table |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020160095225A KR101875763B1 (en) | 2016-07-27 | 2016-07-27 | The database management system and method for preventing performance degradation of transaction when table reconfiguring |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| KR20180012436A KR20180012436A (en) | 2018-02-06 |
| KR101875763B1 true KR101875763B1 (en) | 2018-08-07 |
Family
ID=61017370
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| KR1020160095225A KR101875763B1 (en) | 2016-07-27 | 2016-07-27 | The database management system and method for preventing performance degradation of transaction when table reconfiguring |
Country Status (2)
| Country | Link |
|---|---|
| KR (1) | KR101875763B1 (en) |
| WO (1) | WO2018021593A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11086840B2 (en) | 2018-12-07 | 2021-08-10 | Snowflake Inc. | Transactional streaming of change tracking data |
| CN109840051B (en) * | 2018-12-27 | 2020-08-07 | 华为技术有限公司 | Data storage method and device of storage system |
| CN111723073B (en) * | 2019-03-19 | 2023-09-01 | 北京沃东天骏信息技术有限公司 | Data storage processing method, device, processing system and storage medium |
| CN110046348B (en) * | 2019-03-19 | 2021-05-25 | 西安理工大学 | Method for recognizing main body in subway design specification based on rules and dictionaries |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007293828A (en) * | 2006-04-14 | 2007-11-08 | Hitachi Ltd | System and method for processing a plurality of kinds of event markers of continuous data protection |
| US20100235606A1 (en) * | 2009-03-11 | 2010-09-16 | Oracle America, Inc. | Composite hash and list partitioning of database tables |
| KR101069350B1 (en) * | 2003-05-09 | 2011-10-04 | 마이크로소프트 코포레이션 | Architecture for partition computation and propagation of changes in data replication |
| US20140164443A1 (en) * | 2012-12-06 | 2014-06-12 | Microsoft Corporation | Database scale-out |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8560584B2 (en) * | 2010-12-15 | 2013-10-15 | Teradata Us, Inc. | Database partition management |
| KR101544356B1 (en) * | 2012-10-31 | 2015-08-13 | 삼성에스디에스 주식회사 | Distributed database managing method and composition node thereof supporting dynamic sharding based on the metadata and data transaction quantity |
-
2016
- 2016-07-27 KR KR1020160095225A patent/KR101875763B1/en active IP Right Grant
- 2016-07-28 WO PCT/KR2016/008261 patent/WO2018021593A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101069350B1 (en) * | 2003-05-09 | 2011-10-04 | 마이크로소프트 코포레이션 | Architecture for partition computation and propagation of changes in data replication |
| JP2007293828A (en) * | 2006-04-14 | 2007-11-08 | Hitachi Ltd | System and method for processing a plurality of kinds of event markers of continuous data protection |
| US20100235606A1 (en) * | 2009-03-11 | 2010-09-16 | Oracle America, Inc. | Composite hash and list partitioning of database tables |
| US20140164443A1 (en) * | 2012-12-06 | 2014-06-12 | Microsoft Corporation | Database scale-out |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018021593A1 (en) | 2018-02-01 |
| KR20180012436A (en) | 2018-02-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11468015B2 (en) | Storage and synchronization of metadata in a distributed storage system | |
| KR101875763B1 (en) | The database management system and method for preventing performance degradation of transaction when table reconfiguring | |
| US9280487B2 (en) | Methods and apparatus for data processing using data compression, linked lists and de-duplication techniques | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| CN103875229B (en) | asynchronous replication method, device and system | |
| EP2498476B1 (en) | Massively scalable object storage system | |
| CN105183400B (en) | It is a kind of based on content addressed object storage method and system | |
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| CN106201771B (en) | Data-storage system and data read-write method | |
| CN105138571B (en) | Distributed file system and method for storing massive small files | |
| US20070094269A1 (en) | Systems and methods for distributed system scanning | |
| US20160142249A1 (en) | Method, medium, and system for configuring a new node in a distributed memory network | |
| WO2017088705A1 (en) | Data processing method and device | |
| US20220035786A1 (en) | Distributed database management system with dynamically split b-tree indexes | |
| US12013814B2 (en) | Managing snapshotting of a dataset using an ordered set of B+ trees | |
| US20160139980A1 (en) | Erasure-coding extents in an append-only storage system | |
| KR20190063835A (en) | System for processing real-time data modification of in-memory database | |
| CN108762982B (en) | A kind of database restoring method, apparatus and system | |
| CN113449065A (en) | Data deduplication-oriented decentralized storage method and storage device | |
| US7200625B2 (en) | System and method to enhance availability of a relational database | |
| US8612717B2 (en) | Storage system | |
| US7668846B1 (en) | Data reconstruction from shared update log | |
| CN113076298A (en) | Distributed small file storage system | |
| CN111143366B (en) | High-efficiency storage method for massive large object data | |
| KR101748913B1 (en) | Cluster management method and data storage system for selecting gateway in distributed storage environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A201 | Request for examination | ||
| E902 | Notification of reason for refusal | ||
| E90F | Notification of reason for final refusal | ||
| E701 | Decision to grant or registration of patent right | ||
| GRNT | Written decision to grant |