CN115757642A - Data synchronization method and device based on filing log file - Google Patents
Data synchronization method and device based on filing log file Download PDFInfo
- Publication number
- CN115757642A CN115757642A CN202211528606.3A CN202211528606A CN115757642A CN 115757642 A CN115757642 A CN 115757642A CN 202211528606 A CN202211528606 A CN 202211528606A CN 115757642 A CN115757642 A CN 115757642A
- Authority
- CN
- China
- Prior art keywords
- data table
- target data
- message queue
- target
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data synchronization method and a data synchronization device based on an archived log file, which can be used in the financial field or other technical fields, wherein the method comprises the following steps: monitoring an archive log file of a source database; acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance; and sending the change data corresponding to the target data table to a message queue server, so that the message queue server publishes the change data corresponding to the target data table to a corresponding message queue, and the change data corresponding to the target data table is synchronized to a corresponding target database. The device is used for executing the method. The data synchronization method and device based on the archived log files provided by the embodiment of the invention improve the data synchronization efficiency.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data synchronization method and device based on an archive log file.
Background
At present, in large financial enterprises such as internet finance and banks, the condition that data synchronization is needed to be carried out on upstream and downstream data is often met in the research and development process.
In the prior art, the upstream data and the downstream data are synchronized, usually the downstream data demander acquires the upstream data at regular time, and the data synchronization can be performed through the data file, if the data which the data demander needs to synchronize depends on the generation of the upstream data file, but the upstream data file is not generated, the data synchronization needs to be performed after the generation of the upstream data file, and the timeliness is poor.
Disclosure of Invention
For solving the problems in the prior art, embodiments of the present invention provide a data synchronization method and apparatus based on an archive log file, which can at least partially solve the problems in the prior art.
In a first aspect, the present invention provides a data synchronization method based on an archive log file, including:
monitoring an archive log file of a source database;
acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance;
and sending the change data corresponding to the target data table to a message queue server, so that the message queue server publishes the change data corresponding to the target data table to a corresponding message queue, and the change data corresponding to the target data table is synchronized to a corresponding target database.
In a second aspect, the present invention provides a data synchronization apparatus based on an archive log file, including:
the monitoring module is used for monitoring the filing log file of the source database;
the acquisition module is used for acquiring the change data corresponding to the target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance;
and the sending module is used for sending the change data corresponding to the target data table to a message queue server so that the message queue server can issue the change data corresponding to the target data table to a corresponding message queue to synchronize the change data corresponding to the target data table to a corresponding target database.
In a third aspect, the present invention provides a data synchronization system based on an archived log file, for implementing the data synchronization method based on the archived log file according to any of the above embodiments, including a database server, a console, a message queue server, and a message queue management and control platform, where:
the database server is respectively in communication connection with the console and the message queue server, the console is in communication connection with the message queue control platform, and the message queue control platform is in communication connection with the message queue server;
the console is used for configuring a target data table and sending the target data table to the database server; the database server is used for archiving a log file of a source database to acquire change data corresponding to the target data table and send the change data to the message queue server; the message queue server is used for publishing the change data corresponding to the target data table to a corresponding message queue so as to synchronize the change data corresponding to the target data table to a corresponding target database; the message queue management and control platform is used for receiving a message subscription request, generating the target data table based on the message subscription request and sending the target data table to the console.
In a fourth aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the archive log file-based data synchronization method according to any of the above embodiments.
In a fifth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the archive log file-based data synchronization method according to any one of the above embodiments.
In a sixth aspect, the present invention provides a computer program product comprising a computer program, which when executed by a processor implements the archive log file-based data synchronization method according to any of the above embodiments.
The data synchronization method and device based on the filing log file provided by the embodiment of the invention can monitor the filing log file of the source database, acquire the change data corresponding to the target data table from the filing log file according to the target data table, and send the change data corresponding to the target data table to the message queue server, so that the message queue server can issue the change data corresponding to the target data table to the corresponding message queue, so as to synchronize the change data corresponding to the target data table to the corresponding target database, and can synchronize the changed data of the source database to the target database in time, thereby improving the data synchronization efficiency.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic structural diagram of a data synchronization system based on archive log files according to a first embodiment of the present invention.
Fig. 2 is a flowchart illustrating a data synchronization method based on archive log files according to a second embodiment of the present invention.
Fig. 3 is a flowchart illustrating a data synchronization method based on archive log files according to a third embodiment of the present invention.
Fig. 4 is a flowchart illustrating a data synchronization method based on archive log files according to a fourth embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a data synchronization apparatus based on archive log files according to a fifth embodiment of the present invention.
Fig. 6 is a schematic structural diagram of a data synchronization apparatus based on archive log files according to a sixth embodiment of the present invention.
Fig. 7 is a schematic structural diagram of a data synchronization apparatus based on archive log files according to a seventh embodiment of the present invention.
Fig. 8 is a schematic structural diagram of a data synchronization apparatus based on archive log files according to an eighth embodiment of the present invention.
Fig. 9 is a schematic structural diagram of a data synchronization apparatus based on archive log files according to a ninth embodiment of the present invention.
Fig. 10 is a schematic physical structure diagram of an electronic device according to a tenth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict. In the technical scheme of the application, the data acquisition, storage, use, processing and the like all accord with relevant regulations of national laws and regulations.
In order to facilitate understanding of the technical solutions provided in the present application, the following first describes relevant contents of the technical solutions in the present application.
And archiving a log file, and reserving all transaction redo history records, which means that a backup is reserved for all transactions performed by the database so as to perform data recovery when the database is abnormal. And a producer and a data provider in the data copying scene upload the data to a specified theme.
Consumers, downstream in the data replication scenario, data consumers, consume data from a specified topic.
The problem of data synchronization caused by the dependence of upstream and downstream data in the traditional stock projects of large financial enterprises such as banks in the research and development process is often solved, and the problems that the data synchronization timeliness is poor, the time consumption is long, the synchronization cost is high and the like, and the data requirements cannot be quickly responded are easily caused because the dependence between data is often tree-shaped or net-shaped. In order to solve the above problems encountered by large financial enterprises such as finance, communication and banking in data dependence and data synchronization, and to achieve data synchronization quickly, with high quality and efficiency, improve data synchronization efficiency and reduce data synchronization cost, the data synchronization method based on the filing log file provided by the embodiment of the invention analyzes the change data in the filing log file to perform data synchronization in a quasi-real time manner, so that the time consumption of data synchronization is reduced and the efficiency of data synchronization is improved.
Fig. 1 is a schematic structural diagram of a data synchronization system based on an archive log file according to a first embodiment of the present invention, and as shown in fig. 1, the data synchronization system based on an archive log file according to the embodiment of the present invention includes a database server 1, a console 2, a message queue server 3, and a message queue management and control platform 4, where:
the database server 1 is respectively in communication connection with the console 2 and the message queue server 3, the console 2 is in communication connection with the message queue control platform 4, and the message queue control platform 4 is in communication connection with the message queue server 3. The console 2 and the message queue management and control platform 4 may be implemented by a server.
And the user sends a message subscription request to the message queue management and control platform 4 through the user terminal, wherein the message subscription request is used for applying for a theme corresponding to the data table needing to be copied, and the message subscription request comprises the name of the data table needing to be subjected to data synchronization. After the application is passed, a corresponding message queue is generated in the message queue server 3 and corresponds to the theme. One data table of the source database corresponds to one theme, and the theme name format adopts the following format: the application cluster name, database table name. The message queue management and control platform 4 sends the data table to be synchronized to the console 2.
The console 2 is provided with a data table to be synchronized with data, and the data table is provided to the database server 1 as a target data table. The database server 1 is configured to execute the data synchronization method based on the archived log file according to the embodiment of the present invention, monitor data changes of the archived log file, screen change data corresponding to a target data table that needs data synchronization, and send the change data to the message queue server 3 to publish the received change data to a corresponding message queue.
The following describes a specific implementation process of the data synchronization method based on the archived log files, which is provided by the embodiment of the present invention, by taking a database server as an execution subject.
Fig. 2 is a schematic flowchart of a data synchronization method based on an archive log file according to an embodiment of the present invention, and as shown in fig. 2, the data synchronization method based on the archive log file according to the embodiment of the present invention includes:
s201, monitoring an archiving log file of a source database;
specifically, the database server may obtain an archive log file of the source database, and may monitor data changes of the archive log file through the log parsing component. The source database refers to a database which needs to export data. The archive log file records data changes of the target data table.
S202, acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance;
specifically, if the data in the archive log file changes, the database server may query whether there is change data corresponding to a target data table from the change data in the archive log file, and if there is change data in the target data table, may obtain the change data corresponding to the target data table. The target data table is obtained in advance, and the target data table may include one data table or a plurality of data tables.
For example, if data of a certain data table of the source database changes, a record is made in the archive log file, and the changed data of the data table may be stored in the archive log file corresponding to the name of the data table. The database server inquires whether the change data of the target data table exists in the change data of the filing log file according to the data table name of the target data table, if the change data corresponding to the data table name of the target data table is inquired, the change data of the target data table is indicated to be changed, and the change data corresponding to the target data table can be obtained according to the data table name of the target data table.
S203, sending the change data corresponding to the target data table to a message queue server, so that the message queue server issues the change data corresponding to the target data table to a corresponding message queue, and synchronizing the change data corresponding to the target data table to a corresponding target database.
Specifically, after obtaining the change data corresponding to the target data table, the database server may send the change data corresponding to the target data table to a message queue server. And the message queue server matches the change data corresponding to the target data table to a corresponding message queue and publishes the change data to the corresponding message queue. The messages in the message queue can be pushed to a target database subscribed to the message queue, and the change data corresponding to the target database is synchronized to the target database. The target database refers to a database for importing data.
The data synchronization method based on the filing log file provided by the embodiment of the invention can monitor the filing log file of the source database, acquire the change data corresponding to the target data table from the filing log file according to the target data table, and send the change data corresponding to the target data table to the message queue server, so that the message queue server can issue the change data corresponding to the target data table to the corresponding message queue, so as to synchronize the change data corresponding to the target data table to the corresponding target database, and can synchronize the changed data of the source database to the target database in time, thereby improving the data synchronization efficiency. In addition, the data synchronization can be completed only through the configuration work of the message queue management and control platform and the console without research and development of a data synchronization interface, and the data synchronization cost is saved.
Fig. 3 is a schematic flowchart of a data synchronization method based on an archive log file according to a third embodiment of the present invention, and as shown in fig. 3, on the basis of the foregoing embodiments, further, the sending the change data corresponding to the target data table to a message queue server includes:
s301, converting the change data corresponding to the target data table into JSON character strings in a row unit to obtain the release data of each data table in the target data table;
in particular, JSON (JavaScript Object notification) is a lightweight data exchange format that is easy for machine parsing and generation. And the database server converts the change data corresponding to the target data table into JSON character strings in a row unit, so that the release data of each data table in the target data table can be obtained. And the change data corresponding to the target data table is converted into the JSON character string, so that the transmission efficiency of the data is improved. The data can be converted into the JSON character string through a third-party plug-in, and the third-party plug-in is selected according to actual needs.
S302, obtaining a theme name corresponding to the release data of each data table based on the data table name of each data table in the target data table and the application cluster name corresponding to the data table name;
specifically, for each data table in the target data table, the database server may obtain a data table name of the data table, obtain a corresponding application cluster name according to the data table name, and then obtain a topic name corresponding to the published data of the data table name according to the data table name and the application cluster name corresponding to the data table name.
The name of the database table, namely the naming format of the subject name is as follows: application cluster name database table name. And after the database server obtains the data table name and the application cluster name corresponding to the data table name, combining the topic name corresponding to the release data of the data table according to the naming format of the topic name.
S303, sending the publishing data and the corresponding subject name of each data table to the message queue server.
Specifically, the database server sends the publishing data and the corresponding subject name of each data table in the target data table to the message queue server. And for each data table in the target data table, the message queue server matches the topic name corresponding to the published data of the data table with the topic of each message queue, and if the topic corresponding to a certain message queue is the same as the topic name corresponding to the published data of the data table, the published data of the data table is published to the message queue.
On the basis of the foregoing embodiments, further, the acquiring, according to a target data table, change data corresponding to the target data table from the archive log file includes:
and inquiring the change data corresponding to each data table name from the filing log file according to the data table name of each data table in the target data table.
Specifically, when storing the change data of each data table, the archive log file associates the change data of each data table with the data table name of the data table. And for each data table in the target data table, the database server inquires the change data corresponding to the name of the data table from the filing log file according to the name of the data table. And using the data table name corresponding change data of the data table in the inquired target data table as the change data corresponding to the target data table.
Fig. 4 is a schematic flow chart of a data synchronization method based on an archive log file according to a fourth embodiment of the present invention, and as shown in fig. 4, on the basis of the foregoing embodiments, further, the data synchronization method based on the archive log file according to the embodiment of the present invention further includes:
s401, receiving source head table configuration information sent by a control console, wherein the source head table configuration information comprises each source head table;
specifically, which data tables in the source database need to be synchronized with the target database may be set in the console, and these data tables that need to be synchronized are the source header tables. And the console carries the set source table in the source table configuration information and sends the source table to the database server. The database server receives the source header table configuration information, which includes each source header table.
It will be appreciated that the source table to be copied may be determined by the consumer, which data tables the consumer subscribes to, and which data tables are data synchronized. The source header table may be set based on the data table to which the consumer subscribes. In the embodiment of the present invention, the source database may be regarded as a producer, and the target database may be regarded as a consumer.
S402, taking each source table as the target data table.
Specifically, after receiving the source header table configuration information, the database server obtains each source header table from the source header table configuration information, and takes each source header table as the target data table.
On the basis of the foregoing embodiments, before sending the change data corresponding to the target data table to the message queue server, the method further includes:
and desensitizing the change data corresponding to the target data table.
Specifically, since some data in the source database is sensitive data, such as information of a mobile phone number, a bank card number, and the like, data conversion or modification is required to prevent leakage of the sensitive data. In order to avoid leakage of sensitive data, before sending the change data corresponding to the target data table to the message queue server, the database server performs desensitization processing on the change data corresponding to the target data table.
On the basis of the above embodiments, further, an embodiment of the present invention provides a method further including:
and encrypting the changed data corresponding to the target data table.
Specifically, because some data in the source database is data that needs to be kept secret, for data security, the database server may encrypt the changed data corresponding to the target data table, and the specific encryption mode is set according to actual needs, which is not limited in the embodiment of the present invention.
It can be understood that the change data corresponding to the target data table may be subjected to both desensitization processing and encryption processing, and is selected according to actual needs, which is not limited in the embodiment of the present invention.
On the basis of the above embodiments, further, the subject of the message queue is named by an application cluster name and a data table name.
Specifically, the topic of the message queue is named by an application cluster name and a data table name, so that the change data corresponding to the data table from the source database is published into the message queue.
The bank collects the assets of the whole row of data together by establishing a data lake platform, and provides data support for unified data analysis, sedimentation and deep excavation of the assets of the whole row of data. Each business system of the bank, which is used as the upstream of the data lake platform, can periodically provide data to the data lake platform.
By the data synchronization method based on the filing log file, provided by the embodiment of the invention, the database of each business system of a bank can be used as a source database, and the database of a data lake platform can be used as a target database. And synchronizing the change data of the target data table to the database of the data lake platform in quasi real time by using the archive log file.
The data synchronization method based on the archived log files provided by the embodiment of the invention can consume and store data in real time through subscribing the corresponding topics, thereby achieving the purpose of copying the data in real time among heterogeneous databases. The problem of data synchronization efficiency of large-scale financial enterprises such as finance, communication and banks in data dependence and data synchronization is solved, and data synchronization cost is reduced.
The data synchronization method based on the archival log file provided by the embodiment of the invention has high data synchronization efficiency: the data provider does not need developers to research and develop a data synchronization interface, and can complete data synchronization only through configuration work of a message queue management and control platform and a console; the data synchronization accuracy is high: by automatically analyzing the filing log file of the relational database, the condition that the conversion result is inconsistent with the original data due to the factor of human error is avoided without human intervention; the data synchronization cost is low: the scene of data synchronization is realized through an automation tool, the requirement on the technical capability of personnel is low, and professional technicians are not needed; the universality is high: the method is suitable for all applications, and data synchronization software does not need to be specially developed for a certain application; non-invasive: data synchronization can be achieved by providing limited configuration information without intruding on items.
Fig. 5 is a schematic structural diagram of a data synchronization apparatus based on an archive log file according to a fifth embodiment of the present invention, and as shown in fig. 5, the data synchronization apparatus based on the archive log file according to the embodiment of the present invention includes a monitoring module 501, an obtaining module 502, and a sending module 503, where:
the monitoring module 501 is configured to monitor an archive log file of a source database; the obtaining module 502 is configured to obtain change data corresponding to a target data table from the archive log file according to the target data table; wherein the target data table is obtained in advance; the sending module 503 is configured to send the change data corresponding to the target data table to a message queue server, so that the message queue server issues the change data corresponding to the target data table to a corresponding message queue, so as to synchronize the change data corresponding to the target data table to a corresponding target database.
Specifically, the monitoring module 501 may obtain an archive log file of the source database, and may monitor data changes of the archive log file through a log analysis component. The source database refers to a database which needs to export data. The archive log file records data changes of the target data table.
If the data of the archive log file changes, the obtaining module 502 may query whether there is changed data corresponding to the target data table from the changed data of the archive log file, and if there is changed data of the target data table, obtain the changed data corresponding to the target data table. The target data table is obtained in advance, and the target data table may include one data table or a plurality of data tables.
After obtaining the change data corresponding to the target data table, the sending module 503 may send the change data corresponding to the target data table to the message queue server. And the message queue server matches the change data corresponding to the target data table to the corresponding message queue and releases the change data to the corresponding message queue. The messages in the message queue can be pushed to a target database subscribed to the message queue, and the change data corresponding to the target data table is synchronized to the target database. The target database refers to a database for importing data.
The data synchronization device based on the filing log file provided by the embodiment of the invention can monitor the filing log file of the source database, acquire the change data corresponding to the target data table from the filing log file according to the target data table, and send the change data corresponding to the target data table to the message queue server, so that the message queue server can issue the change data corresponding to the target data table to the corresponding message queue, so as to synchronize the change data corresponding to the target data table to the corresponding target database, and can synchronize the changed data of the source database to the target database in time, thereby improving the data synchronization efficiency. In addition, the data synchronization can be completed only through the configuration work of the message queue management and control platform and the console without research and development of a data synchronization interface, and the data synchronization cost is saved.
Fig. 6 is a schematic structural diagram of a data synchronization apparatus based on an archive log file according to a sixth embodiment of the present invention, and as shown in fig. 6, on the basis of the foregoing embodiments, further, the sending module 503 includes a converting unit 5031, an obtaining unit 5032, and a sending unit 5033, where:
the converting unit 5031 is configured to convert the change data corresponding to the target data table into a JSON character string in a row unit, and obtain release data of each data table in the target data table; the obtaining unit 5032 is configured to obtain a topic name corresponding to the published data of each data table based on the data table name of each data table in the target data table and the application cluster name corresponding to the data table name; the sending unit 5033 is configured to send the publishing data and the corresponding topic name of each data table to the message queue server.
On the basis of the foregoing embodiments, further, the obtaining module 502 is specifically configured to:
and inquiring the filing log file according to the data table name of each data table in the target data table to obtain the change data corresponding to each data table name.
Fig. 7 is a schematic structural diagram of a data synchronization apparatus based on an archive log file according to a seventh embodiment of the present invention, and as shown in fig. 7, on the basis of the foregoing embodiments, further, the data synchronization apparatus based on the archive log file according to the embodiment of the present invention further includes a receiving module 504 and an acting module 505, where:
the receiving module 504 is configured to receive source table configuration information sent by a console, where the source table configuration information includes each source table; act as block 505 for treating each source table as the target data table.
Fig. 8 is a schematic structural diagram of a data synchronization apparatus based on an archive log file according to an eighth embodiment of the present invention, and as shown in fig. 8, the data synchronization apparatus based on the archive log file according to the embodiment of the present invention further includes a desensitization module 506, where:
the desensitization module 506 is configured to perform desensitization processing on the change data corresponding to the target data table.
Fig. 9 is a schematic structural diagram of a data synchronization apparatus based on an archive log file according to a ninth embodiment of the present invention, and as shown in fig. 9, on the basis of the foregoing embodiments, further, the data synchronization apparatus based on the archive log file according to the embodiment of the present invention further includes an encryption module 507, where:
the encryption module 507 is configured to encrypt the change data corresponding to the target data table.
On the basis of the above embodiments, further, the message queue is named by an application cluster name and a data table name.
The embodiment of the apparatus provided in the embodiment of the present invention may be specifically configured to execute the processing flows of the above method embodiments, and the functions of the apparatus are not described herein again, and refer to the detailed description of the above method embodiments.
It should be noted that the data synchronization method and apparatus based on the archived log file provided by the embodiment of the present invention may be used in the financial field, and may also be used in any technical field other than the financial field.
Fig. 10 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 10, the electronic device may include: a processor (processor) 1001, a communication Interface (communication Interface) 1002, a memory (memory) 1003 and a communication bus 1004, wherein the processor 1001, the communication Interface 1002 and the memory 1003 complete communication with each other through the communication bus 1004. Processor 1001 may call logic instructions in memory 1003 to perform the following method: monitoring an archiving log file of a source database; acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance; and sending the change data corresponding to the target data table to a message queue server, so that the message queue server publishes the change data corresponding to the target data table to a corresponding message queue, and the change data corresponding to the target data table is synchronized to a corresponding target database.
In addition, the logic instructions in the memory 1003 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments, for example, comprising: monitoring an archive log file of a source database; acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance; and sending the change data corresponding to the target data table to a message queue server, so that the message queue server publishes the change data corresponding to the target data table to a corresponding message queue, and the change data corresponding to the target data table is synchronized to a corresponding target database.
The present embodiment provides a computer-readable storage medium, which stores a computer program, where the computer program causes the computer to execute the method provided by the above method embodiments, for example, the method includes: monitoring an archive log file of a source database; acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance; and sending the change data corresponding to the target data table to a message queue server, so that the message queue server publishes the change data corresponding to the target data table to a corresponding message queue, and the change data corresponding to the target data table is synchronized to a corresponding target database.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A data synchronization method based on an archive log file is characterized by comprising the following steps:
monitoring an archive log file of a source database;
acquiring change data corresponding to a target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance;
and sending the change data corresponding to the target data table to a message queue server, so that the message queue server publishes the change data corresponding to the target data table to a corresponding message queue, and the change data corresponding to the target data table is synchronized to a corresponding target database.
2. The method of claim 1, wherein sending the change data corresponding to the target data table to a message queue server comprises:
converting the change data corresponding to the target data table into JSON character strings in a row unit to obtain the release data of each data table in the target data table;
obtaining a theme name corresponding to the published data of each data table based on the data table name of each data table in the target data table and the application cluster name corresponding to the data table name;
and sending the publishing data and the corresponding subject name of each data table to the message queue server.
3. The method of claim 1, wherein the obtaining change data corresponding to the target data table from the archive log file according to the target data table comprises:
and inquiring the filing log file according to the data table name of each data table in the target data table to obtain the change data corresponding to each data table name.
4. The method of claim 1, further comprising:
receiving source head table configuration information sent by a control console, wherein the source head table configuration information comprises each source head table;
and taking each source table as the target data table.
5. The method of claim 1, further comprising, before sending the change data corresponding to the target data table to a message queue server:
desensitizing the changed data corresponding to the target data table and/or encrypting the changed data corresponding to the target data table.
6. The method of any of claims 1 to 5, wherein the message queue is named with an application cluster name and a data table name.
7. An archived log file-based data synchronization system for implementing the archived log file-based data synchronization method according to any one of claims 1 to 6, comprising a database server, a console, a message queue server and a message queue management platform, wherein:
the database server is respectively in communication connection with the console and the message queue server, the console is in communication connection with the message queue control platform, and the message queue control platform is in communication connection with the message queue server;
the console is used for configuring a target data table and sending the target data table to the database server; the database server is used for monitoring the filing log file of a source database to acquire the change data corresponding to the target data table and sending the change data to the message queue server; the message queue server is used for publishing the change data corresponding to the target data table to a corresponding message queue so as to synchronize the change data corresponding to the target data table to a corresponding target database; the message queue management and control platform is used for receiving a message subscription request, generating the target data table based on the message subscription request and sending the target data table to the console.
8. A data synchronization apparatus based on archive log files, comprising:
the monitoring module is used for monitoring the filing log file of the source database;
the acquisition module is used for acquiring the change data corresponding to the target data table from the filing log file according to the target data table; wherein the target data table is obtained in advance;
and the sending module is used for sending the change data corresponding to the target data table to a message queue server so that the message queue server can issue the change data corresponding to the target data table to a corresponding message queue to synchronize the change data corresponding to the target data table to a corresponding target database.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 6 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211528606.3A CN115757642A (en) | 2022-11-30 | 2022-11-30 | Data synchronization method and device based on filing log file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211528606.3A CN115757642A (en) | 2022-11-30 | 2022-11-30 | Data synchronization method and device based on filing log file |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115757642A true CN115757642A (en) | 2023-03-07 |
Family
ID=85342027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211528606.3A Pending CN115757642A (en) | 2022-11-30 | 2022-11-30 | Data synchronization method and device based on filing log file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115757642A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116541470A (en) * | 2023-07-07 | 2023-08-04 | 深圳创维智慧科技有限公司 | Synchronization method, device, equipment and medium of database read-only standby library |
-
2022
- 2022-11-30 CN CN202211528606.3A patent/CN115757642A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116541470A (en) * | 2023-07-07 | 2023-08-04 | 深圳创维智慧科技有限公司 | Synchronization method, device, equipment and medium of database read-only standby library |
| CN116541470B (en) * | 2023-07-07 | 2024-02-13 | 深圳创维智慧科技有限公司 | Synchronization method, device, equipment and medium of database read-only standby library |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022126974A1 (en) | Kafka-based incremental data synchronization method and apparatus, device, and medium | |
| CN108536761B (en) | Report data query method and server | |
| US11356337B2 (en) | System and method for tracking service requests | |
| US9069800B2 (en) | Parallel database backup and restore | |
| CN110825420A (en) | Configuration parameter updating method, device, equipment and storage medium for distributed cluster | |
| KR20210005043A (en) | Input and output schema mapping | |
| US10122665B2 (en) | Distributed synchronization data in a message management service | |
| US12019634B1 (en) | Reassigning a processing node from downloading to searching a data group | |
| CN112559475B (en) | Data real-time capturing and transmitting method and system | |
| CN104657497A (en) | Mass electricity information concurrent computation system and method based on distributed computation | |
| US11892976B2 (en) | Enhanced search performance using data model summaries stored in a remote data store | |
| CN103235811A (en) | Data storage method and device | |
| CN110781197B (en) | Hive offline synchronous verification method and device and electronic equipment | |
| CN111460038A (en) | Quasi-real-time data synchronization method and device | |
| CN112073395B (en) | File distribution method and device | |
| AU2014315494A1 (en) | Automatically generating certification documents | |
| CN111563083B (en) | Report data query method, device and system | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| CN113434312A (en) | Data blood relationship processing method and device | |
| CN115757642A (en) | Data synchronization method and device based on filing log file | |
| US11841827B2 (en) | Facilitating generation of data model summaries | |
| US9092338B1 (en) | Multi-level caching event lookup | |
| KR20220086677A (en) | Systems and methods for blockchain-based backup and recovery | |
| CN110941658A (en) | Data export method, device, server and storage medium | |
| CN113449042B (en) | Automatic data warehouse separation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |