CN114443601A - Method and device for processing object log - Google Patents
Method and device for processing object log Download PDFInfo
- Publication number
- CN114443601A CN114443601A CN202111583557.9A CN202111583557A CN114443601A CN 114443601 A CN114443601 A CN 114443601A CN 202111583557 A CN202111583557 A CN 202111583557A CN 114443601 A CN114443601 A CN 114443601A
- Authority
- CN
- China
- Prior art keywords
- data page
- data
- updated
- incremental
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000004590 computer program Methods 0.000 claims description 9
- 238000003672 processing method Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a method and a device for processing an object log. The method comprises the following steps: after receiving all the thing logs sent by a master node in a cloud database, traversing all the thing logs in sequence; for any traversed transaction log, executing the following steps: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated; and storing the incremental data corresponding to the data page identification to be updated by utilizing the data page identification to be updated. Therefore, the original data page data corresponding to the page identification to be updated does not need to be acquired from the cloud database, and the original data page data does not need to be updated. Therefore, a large amount of processing time is saved, and the situation that the object log sent to the slave node by the master node is generated, and the processing result of the object log cannot be inquired by the slave node is reduced.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method and an apparatus for processing an object log.
Background
The cloud database provides a highly available and high-performance relational database service, faces to an enterprise complex OLTP (online Transaction Processing) scenario, and has the advantages mainly focusing On the integrity of Structured Query Language (SQL) specification and rich and diverse data types. The cloud database with the read-write separation improves the overall throughput of the cloud database by increasing the query requests of the slave nodes.
In the prior art, after a slave node in a cloud database receives each object log sent by a master node of the cloud database, the following operations are sequentially performed on each object log: analyzing the object log to obtain a data page identifier to be updated and incremental data, then acquiring original data page data corresponding to the data page identifier to be updated from the cloud database by the slave node, then updating the original data page data by utilizing the incremental data, and storing the updated original data page data in the slave node. However, this method results in a long time for the slave node to process each transaction log, and when there are a large number of transaction logs, the slave node takes a lot of time to process each transaction log. This may cause the object log transmitted from the master node to the slave node to be not queried by the slave node as a result of the processing of the object log.
Disclosure of Invention
The exemplary embodiments of the present disclosure provide a method and an apparatus for processing a transaction log, which are used to shorten the processing time of the transaction log, reduce the occurrence of the transaction log sent from a master node to a slave node, and when the slave node cannot inquire the processing result of the transaction log.
A first aspect of the present disclosure provides a transaction log processing method, applied to a slave node in a cloud database, where the method includes:
after receiving all object logs sent by a master node in a cloud database, traversing all object logs in sequence;
for any traversed transaction log, executing the following steps:
analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
and storing the incremental data corresponding to the data page identification to be updated by utilizing the data page identification to be updated.
In this embodiment, each traversed object log is analyzed to obtain a to-be-updated data page identifier and incremental data corresponding to the to-be-updated data page identifier, and then the incremental data corresponding to the to-be-updated data page identifier is stored by using the to-be-updated data page identifier. Therefore, the processing of the transaction log is completed, and in the embodiment, the original data page data corresponding to the page identifier to be updated does not need to be acquired from the cloud database, and the original data page data does not need to be updated. Therefore, a large amount of processing time is saved, and the situation that the object log sent to the slave node by the master node is generated, and the processing result of the object log cannot be inquired by the slave node is reduced.
In one embodiment, after storing the incremental data corresponding to the data page identifier to be updated by using the data page identifier to be updated, the method further includes:
responding to a query request sent by a user, and determining a target data page identification based on the query request;
acquiring original data page data corresponding to the target data page identification from a cloud database;
combining the incremental data corresponding to the target data page identification with the original data page data corresponding to the target data page identification to obtain target data page data;
and sending the target data page data to the terminal equipment corresponding to the user.
In this embodiment, the incremental data corresponding to the target data page identifier and the original data page data corresponding to the target data page identifier are merged to obtain target data page data, and the target data page data is sent to the terminal device corresponding to the user. Thus, data consistency is guaranteed.
In an embodiment, before the combining the incremental data corresponding to the target data page identifier and the original data page data corresponding to the target data page identifier to obtain the target data page data, the method further includes:
and determining that the storage time of the incremental data is greater than the storage time of the original data page data in the cloud database.
In this embodiment, before the incremental data corresponding to the target data page identifier and the raw data page data corresponding to the target data page identifier are merged, it is required to determine that the storage time of the incremental data is longer than the storage time of the raw data page data in the cloud database, so as to ensure the accuracy of the obtained target data page data.
In one embodiment, the method further comprises:
and if the storage time of the incremental data is not more than the storage time of the original data page data, sending the original data page data to the terminal equipment corresponding to the user.
In this embodiment, if the storage time of the incremental data is not longer than the storage time of the raw data page data, the raw data page data is sent to the terminal device corresponding to the user, so that the accuracy of query is improved.
In one embodiment, the method further comprises:
periodically acquiring target data page refreshing time of a main node, wherein the target data page refreshing time is the data page refreshing time which has the minimum time interval with the current time and has the latest time in the data page refreshing time;
and deleting the incremental data with the storage time smaller than the target data page refreshing time.
In this embodiment, the incremental data whose storage time is less than the target data page refresh time is deleted, so that useless incremental data is cleared, and the memory usage rate is reduced.
A second aspect of the present disclosure provides an apparatus for processing transaction logs, the apparatus comprising:
the traversal module is used for traversing all the object logs in sequence after receiving all the object logs sent by the main node in the cloud database;
the analysis module is used for executing the following steps aiming at any traversed object log: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
and the storage module is used for storing the incremental data corresponding to the data page identification to be updated by utilizing the data page identification to be updated.
In one embodiment, the apparatus further comprises:
a target data page identifier determining module, configured to, after storing the incremental data corresponding to the data page identifier to be updated by using the data page identifier to be updated, respond to an inquiry request sent by a user, and determine a target data page identifier based on the inquiry request;
the raw data page data acquisition module is used for acquiring raw data page data corresponding to the target data page identifier from a cloud database;
the merging module is used for merging the incremental data corresponding to the target data page identification and the original data page data corresponding to the target data page identification to obtain target data page data;
and the first sending module is used for sending the target data page data to the terminal equipment corresponding to the user.
In one embodiment, the apparatus further comprises:
and the storage time comparison module is used for determining that the storage time of the incremental data is longer than the storage time of the raw data page data in the cloud database before merging the incremental data corresponding to the target data page identifier and the raw data page data corresponding to the target data page identifier to obtain the target data page data.
In one embodiment, the apparatus further comprises:
and the second sending module is used for sending the original data page data to the terminal equipment corresponding to the user if the storage time of the incremental data is not more than the storage time of the original data page data.
In one embodiment, the apparatus further comprises:
the device comprises a target data page refreshing time acquisition module, a main node and a data page refreshing module, wherein the target data page refreshing time acquisition module is used for periodically acquiring the target data page refreshing time of the main node, and the target data page refreshing time is the data page refreshing time which has the minimum time interval with the current time and the latest time in the data page refreshing time;
and the deleting module is used for deleting the incremental data of which the storage time is less than the target data page refreshing time.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions for execution by the at least one processor; the instructions are executable by the at least one processor to enable the at least one processor to perform the method of the first aspect.
According to a fourth aspect provided by an embodiment of the present disclosure, there is provided a computer storage medium storing a computer program for executing the method according to the first aspect.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic diagram of a suitable scenario in accordance with an embodiment of the present disclosure;
FIG. 2 is a flow diagram of a method for processing transaction logs according to one embodiment of the present disclosure;
FIG. 3 is a second flowchart of a transaction log processing method according to an embodiment of the disclosure;
FIG. 4 is a schematic flow diagram of querying target data page data according to one embodiment of the present disclosure;
FIG. 5 is a third flowchart of a method for processing transaction logs according to an embodiment of the present disclosure;
FIG. 6 is a processing apparatus for transaction logs according to one embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
The term "and/or" in the embodiments of the present disclosure describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B, and may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The application scenario described in the embodiment of the present disclosure is for more clearly illustrating the technical solution of the embodiment of the present disclosure, and does not form a limitation on the technical solution provided in the embodiment of the present disclosure, and as a person of ordinary skill in the art knows, with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present disclosure is also applicable to similar technical problems. In the description of the present disclosure, the term "plurality" means two or more unless otherwise specified.
In the prior art, after receiving each transaction log sent by a master node, a slave node executes the following operations for each transaction log in sequence: analyzing the object log to obtain a data page identifier to be updated and incremental data, then acquiring original data page data corresponding to the data page identifier to be updated from the cloud database by the slave node, then updating the original data page data by utilizing the incremental data, and storing the updated original data page data in the slave node. However, this method results in a long time for the slave node to process each transaction log, and when there are a large number of transaction logs, the slave node takes a lot of time to process each transaction log. This may cause the object log transmitted from the master node to the slave node to be not queried by the slave node as a result of the processing of the object log.
For example, after the master node sends the object logs t 1-t 6 to the slave node, the slave node processes the object log t1, firstly analyzes the object log t1 to obtain a data page identifier to be updated and incremental data, then the slave node acquires original data page data corresponding to the data page identifier to be updated from the cloud database, then updates the original data page data by using the incremental data, and stores the updated original data page data in the page cache management module of the slave node. Then transaction log t2 is processed, and so on. However, since the slave node processes each transaction log for too long, the transaction log transmitted from the master node to the slave node may occur during the processing, but the slave node may not inquire the processing result of the transaction log. If the query is made at this time when the node is processed only to t3, the query is executed to the processing result of the object log corresponding to t1 to t3, but the query cannot be executed to the processing result of the object log corresponding to t4 to t 6.
Therefore, the present disclosure provides a method for processing an object log, where the object log is analyzed by traversing each object log to obtain a to-be-updated data page identifier and incremental data corresponding to the to-be-updated data page identifier, and then the to-be-updated data page identifier is used to store the incremental data corresponding to the to-be-updated data page identifier. Therefore, the processing of the transaction log is completed, and in the embodiment, the original data page data corresponding to the page identifier to be updated does not need to be acquired from the cloud database, and the original data page data does not need to be updated. Therefore, a large amount of processing is saved, and the situation that the object log sent to the slave node by the master node is generated, and the processing result of the object log cannot be inquired by the slave node is reduced. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, an application scenario of the transaction log processing method includes a slave node server 110 and a master node server 120, where the slave node server 110 and the master node server 120 may be implemented by a single server or by multiple servers. The slave node server 110 and the master node server 120 may be implemented by physical servers or virtual servers.
In a possible application scenario, after receiving each transaction log sent by the master node server 120 in the cloud database from the node server 110, traversing each transaction log in sequence; the following steps are executed for any traversed transaction log from the node server 110: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated; and storing the incremental data corresponding to the data page identification to be updated by using the data page identification to be updated. Therefore, the method and the device do not need to acquire the original data page data corresponding to the page identification to be updated from the cloud database, and do not need to update the original data page data. Therefore, a large amount of processing is saved, and the situation that the object log sent to the slave node by the master node is generated, and the processing result of the object log cannot be inquired by the slave node is reduced.
As shown in fig. 2, a flow chart of the transaction log processing method of the present disclosure may include the following steps:
step 201: after receiving all the thing logs sent by a master node in a cloud database, traversing all the thing logs in sequence;
in this embodiment, each transaction log is traversed according to the sequence of receiving the transaction logs. The specific traversal order may be set according to an actual situation, and this embodiment is not limited herein.
Step 202: for any traversed transaction log, executing the following steps: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
step 203: and storing the incremental data corresponding to the data page identification to be updated by utilizing the data page identification to be updated.
In the implementation, the hash table is used to store the incremental data, and the key value is the data page identifier to be updated corresponding to the incremental data.
For example, as shown in fig. 3, after receiving each transaction log sent by the master node in the cloud database, the slave node sequentially traverses each transaction log, wherein for any traversed transaction log, the following steps are performed: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated; and then the slave node stores the incremental data corresponding to the data page identification to be updated by using the data page identification to be updated.
Therefore, in the embodiment, the original data page data corresponding to the page identifier to be updated does not need to be acquired from the cloud database, and the original data page data does not need to be updated. Therefore, a large amount of processing time is saved, and the situation that the object log sent to the slave node by the master node is generated and the processing result of the object log cannot be inquired by the slave node is reduced.
As shown in fig. 4, a schematic flowchart of the process for querying the target data page data may include the following steps:
step 401: responding to a query request sent by a user, and determining a target data page identifier based on the query request;
wherein the query request includes the target data page identifier.
Step 402: acquiring original data page data corresponding to the target data page identification from a cloud database;
and the cloud database stores the data page identification and the original data page data correspondingly.
Step 403: merging the incremental data corresponding to the target data page identification and the original data page data corresponding to the target data page identification to obtain target data page data;
to improve the accuracy of the target data page data, before performing step 403, in one embodiment, it is determined that the storage time of the delta data is greater than the storage time of the raw data page data in the cloud database.
For example, the target data page data includes data page data 1 and data page data 2. If the storage time of the original data page data of the data page data 1 in the cloud database is T1, and the storage time of the incremental data of the data page data 1 is T2. If T1< T2, then step 303 is performed.
Step 404: and sending the target data page data to the terminal equipment corresponding to the user.
In an embodiment, if the storage time of the incremental data is not greater than the storage time of the raw data page data, the raw data page data is sent to the terminal device corresponding to the user.
For example, if the storage time of the original data page data of the data page data 2 in the cloud database is T3, and the storage time of the incremental data of the data page data 2 is T4. And if T4< T3, sending the original data page data of the data page data 2 to the terminal equipment corresponding to the user.
In order to save the storage space, in one implementation, target data page refreshing time of the master node is periodically acquired, wherein the target data page refreshing time is data page refreshing time which has the minimum time interval with the current time and has the latest time in the data page refreshing time; and deleting the incremental data with the storage time smaller than the target data page refreshing time.
For example, if the target data page refresh time is 6:00, the incremental data in the slave node and the storage time of each incremental data are respectively: (incremental data A, 6:14), (incremental data B, 5:45), (incremental data C, 7: 00). The storage time of the incremental data B is determined to be less than the target data page refreshing time, and the incremental data B is deleted from the node.
For further understanding of the technical solution of the present disclosure, the following detailed description with reference to fig. 5 may include the following steps:
step 501: after receiving all the thing logs sent by a master node in a cloud database, traversing all the thing logs in sequence;
step 502: for any traversed transaction log, executing the following steps: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
step 503: storing incremental data corresponding to the data page identification to be updated by using the data page identification to be updated;
step 504: responding to a query request sent by a user, and determining a target data page identification based on the query request;
step 505: acquiring original data page data corresponding to the target data page identification from a cloud database;
step 506: judging whether the storage time of the incremental data is longer than the storage time of the original data page data in the cloud database, if so, executing a step 507, and if not, executing a step 509;
step 507: merging the incremental data corresponding to the target data page identification and the original data page data corresponding to the target data page identification to obtain target data page data;
step 508: sending the target data page data to terminal equipment corresponding to the user;
step 509: and sending the original data page data to the terminal equipment corresponding to the user.
Based on the same disclosure concept, the transaction log processing method of the present disclosure as described above can also be implemented by a transaction log processing apparatus. The effect of the processing device of the transaction log is similar to that of the method, and is not described herein again.
Fig. 6 is a schematic structural diagram of a transaction log processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 6, a transaction log processing device 600 of the present disclosure may include a traversal module 610, a parsing module 620, and a storage module 630.
The traversal module 610 is configured to traverse the object logs in sequence after receiving the object logs sent by the master node in the cloud database;
the parsing module 620 is configured to execute the following steps for any traversed transaction log: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
a storage module 630, configured to store, by using the to-be-updated data page identifier, incremental data corresponding to the to-be-updated data page identifier.
In one embodiment, the apparatus further comprises:
a target data page identifier determining module 640, configured to, after storing the incremental data corresponding to the data page identifier to be updated by using the data page identifier to be updated, respond to an inquiry request sent by a user, and determine a target data page identifier based on the inquiry request;
a raw data page data obtaining module 650, configured to obtain raw data page data corresponding to the target data page identifier from a cloud database;
a merging module 660, configured to merge the incremental data corresponding to the target data page identifier and the original data page data corresponding to the target data page identifier to obtain target data page data;
a first sending module 670, configured to send the target data page data to a terminal device corresponding to the user.
In one embodiment, the apparatus further comprises:
a storage time comparison module 680, configured to determine that storage time of the incremental data is longer than storage time of the raw data page data in the cloud database before merging the incremental data corresponding to the target data page identifier with the raw data page data corresponding to the target data page identifier to obtain target data page data.
In one embodiment, the apparatus further comprises:
a second sending module 690, configured to send the raw data page data to the terminal device corresponding to the user if the storage time of the incremental data is not greater than the storage time of the raw data page data.
In one embodiment, the apparatus further comprises:
a target data page refresh time acquisition module 691, configured to periodically acquire a target data page refresh time of the master node, where the target data page refresh time is a data page refresh time that has a minimum time interval with the current time and is latest in the refresh times of the data pages;
a delete module 692 for deleting delta data having a storage time less than the target data page refresh time.
After a method and an apparatus for processing an event log according to an exemplary embodiment of the present disclosure are introduced, an electronic device according to another exemplary embodiment of the present disclosure is introduced.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
In some possible implementations, an electronic device in accordance with the present disclosure may include at least one processor, and at least one computer storage medium. Wherein the computer storage medium stores program code which, when executed by the processor, causes the processor to perform the steps in the method for processing transaction logs according to various exemplary embodiments of the present disclosure described above in this specification. For example, the processor may perform step 201 and 203 as shown in FIG. 2.
An electronic device 700 according to this embodiment of the disclosure is described below with reference to fig. 7. The electronic device 700 shown in fig. 7 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, the electronic device 700 is represented in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to: the at least one processor 701, the at least one computer storage medium 702, and the bus 703 that connects the various system components (including the computer storage medium 702 and the processor 701).
The computer storage media 702 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM)721 and/or cache storage media 722, and may further include read-only computer storage media (ROM) 723.
The electronic device 700 may also communicate with one or more external devices 704 (e.g., keyboard, pointing device, etc.), with one or more devices that enable a user to interact with the electronic device 700, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 700 to communicate with one or more other electronic devices. Such communication may occur via input/output (I/O) interfaces 705. Also, the electronic device 700 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 706. As shown, the network adapter 706 communicates with the other modules for the electronic device 700 over a bus 703. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
In some possible embodiments, various aspects of a method for processing a transaction log provided by the present disclosure may also be implemented in the form of a program product including program code for causing a computer device to perform the steps of the method for processing a transaction log according to various exemplary embodiments of the present disclosure described above in this specification when the program product is run on the computer device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable diskette, a hard disk, a random access computer storage media (RAM), a read-only computer storage media (ROM), an erasable programmable read-only computer storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only computer storage media (CD-ROM), an optical computer storage media piece, a magnetic computer storage media piece, or any suitable combination of the foregoing.
The program product of the processing of transaction logs of embodiments of the present disclosure may employ a portable compact disc read-only computer storage medium (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present disclosure is not so limited, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (for example, through the internet using an internet service provider).
It should be noted that although several modules of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the modules described above may be embodied in one module, in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module described above may be further divided into embodiments by a plurality of modules.
Further, while the operations of the disclosed methods are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk computer storage media, CD-ROMs, optical computer storage media, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable computer storage medium produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications can be made in the present disclosure without departing from the spirit and scope of the disclosure. Thus, if such modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include such modifications and variations as well.
Claims (10)
1. A transaction log processing method is applied to a slave node in a cloud database, and comprises the following steps:
after receiving all the thing logs sent by a master node in a cloud database, traversing all the thing logs in sequence;
for any traversed transaction log, executing the following steps:
analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
and storing the incremental data corresponding to the data page identification to be updated by utilizing the data page identification to be updated.
2. The method according to claim 1, wherein after storing the incremental data corresponding to the data page identifier to be updated by using the data page identifier to be updated, the method further comprises:
responding to a query request sent by a user, and determining a target data page identification based on the query request;
acquiring original data page data corresponding to the target data page identification from a cloud database;
merging the incremental data corresponding to the target data page identification and the original data page data corresponding to the target data page identification to obtain target data page data;
and sending the target data page data to the terminal equipment corresponding to the user.
3. The method according to claim 2, wherein before merging the incremental data corresponding to the target data page identifier with the raw data page data corresponding to the target data page identifier to obtain the target data page data, the method further comprises:
and determining that the storage time of the incremental data is greater than the storage time of the original data page data in the cloud database.
4. The method of claim 3, further comprising:
and if the storage time of the incremental data is not more than the storage time of the original data page data, sending the original data page data to the terminal equipment corresponding to the user.
5. The method of claim 1, further comprising:
periodically acquiring target data page refreshing time of a main node, wherein the target data page refreshing time is the data page refreshing time which has the minimum time interval with the current time and has the latest time in the data page refreshing time;
and deleting the incremental data with the storage time smaller than the target data page refreshing time.
6. An apparatus for transaction log processing, the apparatus comprising:
the traversal module is used for traversing all the object logs in sequence after receiving all the object logs sent by the main node in the cloud database;
the analysis module is used for executing the following steps aiming at any traversed transaction log: analyzing the object log to obtain a data page identifier to be updated and incremental data corresponding to the data page identifier to be updated;
and the storage module is used for storing the incremental data corresponding to the data page identification to be updated by utilizing the data page identification to be updated.
7. The apparatus of claim 6, further comprising:
a target data page identifier determining module, configured to, after storing the incremental data corresponding to the data page identifier to be updated by using the data page identifier to be updated, respond to an inquiry request sent by a user, and determine a target data page identifier based on the inquiry request;
the raw data page data acquisition module is used for acquiring raw data page data corresponding to the target data page identifier from a cloud database;
the merging module is used for merging the incremental data corresponding to the target data page identification and the original data page data corresponding to the target data page identification to obtain target data page data;
and the first sending module is used for sending the target data page data to the terminal equipment corresponding to the user.
8. The apparatus of claim 7, further comprising:
and the storage time comparison module is used for determining that the storage time of the incremental data is longer than the storage time of the raw data page data in the cloud database before merging the incremental data corresponding to the target data page identifier and the raw data page data corresponding to the target data page identifier to obtain the target data page data.
9. An electronic device comprising at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions for execution by the at least one processor; the instructions are executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
10. A computer storage medium, characterized in that the computer storage medium stores a computer program for performing the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111583557.9A CN114443601A (en) | 2021-12-22 | 2021-12-22 | Method and device for processing object log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111583557.9A CN114443601A (en) | 2021-12-22 | 2021-12-22 | Method and device for processing object log |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114443601A true CN114443601A (en) | 2022-05-06 |
Family
ID=81362909
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111583557.9A Pending CN114443601A (en) | 2021-12-22 | 2021-12-22 | Method and device for processing object log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114443601A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241175A (en) * | 2018-06-28 | 2019-01-18 | 东软集团股份有限公司 | Method of data synchronization, device, storage medium and electronic equipment |
| CN110019066A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Data base processing method and device, system |
| CN113791935A (en) * | 2021-09-06 | 2021-12-14 | 广州宝云信息科技有限公司 | Data backup method, network node and system |
-
2021
- 2021-12-22 CN CN202111583557.9A patent/CN114443601A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019066A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Data base processing method and device, system |
| CN109241175A (en) * | 2018-06-28 | 2019-01-18 | 东软集团股份有限公司 | Method of data synchronization, device, storage medium and electronic equipment |
| CN113791935A (en) * | 2021-09-06 | 2021-12-14 | 广州宝云信息科技有限公司 | Data backup method, network node and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908997B (en) | Data blood relationship construction method and device, server and readable storage medium | |
| US9037677B2 (en) | Update protocol for client-side routing information | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN106897322B (en) | A kind of access method and device of database and file system | |
| US9251227B2 (en) | Intelligently provisioning cloud information services | |
| US20170075775A1 (en) | System and Method for Time Parameter Based Database Restoration | |
| US20200364241A1 (en) | Method for data synchronization between a source database system and target database system | |
| US20240095241A1 (en) | Data search method and apparatus, and device | |
| AU2021244852B2 (en) | Offloading statistics collection | |
| CN111143390A (en) | Method and device for updating metadata | |
| US10678789B2 (en) | Batch data query method and apparatus | |
| WO2021031583A1 (en) | Method and apparatus for executing statements, server and storage medium | |
| US9229969B2 (en) | Management of searches in a database system | |
| CN109388651B (en) | Data processing method and device | |
| CN113297222B (en) | Report data acquisition method and device, electronic equipment and storage medium | |
| US11321374B2 (en) | External storage of unstructured database objects | |
| CN109977097B (en) | Network configuration management method and system based on database | |
| CN114443601A (en) | Method and device for processing object log | |
| CN112148762A (en) | Statistical method and device for real-time data stream | |
| CN113268483B (en) | Request processing method and device, electronic equipment and storage medium | |
| CN115292313A (en) | Pseudo-column implementation method and device, electronic equipment and storage medium | |
| CN113568892A (en) | Method and equipment for carrying out data query on data source based on memory calculation | |
| CN113760950A (en) | Index data query method and device, electronic equipment and storage medium | |
| CN111880840A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN112286992B (en) | Query method, distributed system, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |