CN110209518A - A kind of multi-data source daily record data, which is concentrated, collects storage method and device - Google Patents
A kind of multi-data source daily record data, which is concentrated, collects storage method and device Download PDFInfo
- Publication number
- CN110209518A CN110209518A CN201910343502.7A CN201910343502A CN110209518A CN 110209518 A CN110209518 A CN 110209518A CN 201910343502 A CN201910343502 A CN 201910343502A CN 110209518 A CN110209518 A CN 110209518A
- Authority
- CN
- China
- Prior art keywords
- log
- daily record
- record data
- server
- collection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 65
- 238000009434 installation Methods 0.000 claims abstract description 17
- 230000001360 synchronised effect Effects 0.000 claims description 14
- 230000033228 biological regulation Effects 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 12
- 230000001960 triggered effect Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 235000013399 edible fruits Nutrition 0.000 claims 2
- 230000008030 elimination Effects 0.000 abstract description 5
- 238000003379 elimination reaction Methods 0.000 abstract description 5
- 230000007257 malfunction Effects 0.000 abstract description 5
- 238000013024 troubleshooting Methods 0.000 abstract description 5
- 238000007405 data analysis Methods 0.000 abstract description 2
- 230000002159 abnormal effect Effects 0.000 description 5
- 239000012141 concentrate Substances 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000004321 preservation Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a kind of multi-data source daily record data concentration collection storage method, and the installation log collection system client on each application server collects the daily record data of each application server by result collection system client and is sent to log processing system；Log processing system first saves original daily record data duplication portion into log storage system after receiving daily record data；Original daily record data is screened again, is cut, is classified and merging treatment, and will treated that new daily record data is saved in log storage system；Log index is established in log display systems, and daily record data is obtained from log storage system according to log index and is shown.The present invention also provides devices corresponding to a kind of method.It realizes the collection storage and retrieval of multi-data source log through the invention, can be improved troubleshooting efficiency, promote data analysis capabilities, and lay a good foundation for the malfunction elimination in later period.
Description
Technical field
The present invention relates to log processing field, in particular to a kind of multi-data source daily record data concentrate collect storage method and
Device.
Background technique
Distributed Application is the application program being made of the component under the different running environment for running on separation, usually
Got up on different platforms by the network interconnection.Typical Distributed Application has two ends (Client/Server), three ends
(client/middleware/server) and the end n (client/multiple middleware/multiple server).
Currently, Distributed Application cluster has been widely used.There is exception or failure in Distributed Application cluster
When, in order to reduce influence caused by exception or failure, technical staff requires to go to carry out abnormal investigation or failure at the first time
Positioning.It is all to carry out hardware resource detection, operating system shape by manually successively logging in server node in groups under conventional situation
State inspection, using log analysis.But due to server cluster there are region dispersibility, investigation step it is various etc. because
Therefore element when specifically being checked or being positioned, usually requires to consume a large amount of time, cannot achieve at all at the first time
Failure is repaired, this also causes enterprise's SLA index to decline, and has seriously affected the experience of system user, while also resulting in enterprise
Economic loss.
Summary of the invention
One of the technical problem to be solved in the present invention is to provide a kind of multi-data source daily record data concentration collection storage side
Method is realized the collection storage and retrieval of multi-data source log by this method, troubleshooting efficiency can be improved, promote data point
Analysis ability, and lay a good foundation for the malfunction elimination in later period.
The present invention is realized in one of technical problem: a kind of multi-data source daily record data concentration collection storage method,
Described method includes following steps:
Step S1, the installation log collection system client on each application server, passes through result collection system client
It collects the daily record data of each application server and is sent to log processing system；
Step S2, log processing system is first a to day by original daily record data duplication after receiving daily record data
It is saved in will storage system；Original daily record data is screened again, is cut, is classified and merging treatment, and will place
New daily record data after reason is saved in log storage system；
Step S3, log index is established in log display systems, and is obtained from log storage system according to log index
Daily record data is taken to be shown.
Further, the step S2 further include:
When handling original daily record data, log processing system is to the log content attribute in new daily record data
It is scanned, and Auto-matching alarm regulation, and log early warning system transmission alert content is triggered according to alarm regulation and is accused
It is alert；The alert content includes application server IP, log alias, log path and log content.
Further, the step S1 specifically:
The installation log collection system client on each application server, and the configuration file of result collection system is carried out
The parameter of parameter configuration, configuration includes at least local IP address, journal file path, the merging of log multirow, Collection Rules, log
The address of processing system and log interval time；
The application program of running log collection system, just whether application program reads configuration file, and check configuration file
Really, it if it is not, then stopping the operation of application program, and sends prompting message and gives result collection system client；If it is, pressing
Corresponding journal file is scanned according to the Collection Rules configured in configuration file, and the daily record data scanned is sent to day in real time
Will processing system.
Further, a local server node and at least one server are provided in the log storage system
From node；
The step S2 is specifically included:
Step S21, log processing system is a to log by original daily record data duplication after receiving daily record data
Storage system, log storage system save original daily record data to local server node, while by original log
Data are synchronized to each server from node；
Step S22, log processing system carries out attributes extraction to original daily record data, merges rule according to the attribute of setting
Then to merge new daily record data, and classified according to the classification type of setting to new daily record data；Wherein, the attribute of extraction
Including application server IP, log alias, log path, Log Types, server name, log content, log rank or
Custom Attributes；
Step S23, will treated new daily record data is sent to log storage system, log storage system is by new log number
According to saving on local server node, while new daily record data is synchronized to each server from node.
Further, the step S3 specifically:
Log display systems are scanned the new daily record data being stored in log storage system, and according to user in day
The search rule set in the interface of will display systems indexes to establish log, to get corresponding day from log storage system
Will data；
Log display systems are shown according to different attributes to the daily record data of acquisition or log display systems
The user behavior of log is counted and shown by introducing the third-party library IP.
The second technical problem to be solved by the present invention is to provide a kind of multi-data source daily record data concentration collection storage dress
It sets, the collection storage and retrieval of multi-data source log is realized by the device, troubleshooting efficiency can be improved, promote data point
Analysis ability, and lay a good foundation for the malfunction elimination in later period.
The present invention is realized in the twos' of technical problem: a kind of multi-data source daily record data concentration collection storage device,
Described device includes log collection module, log processing module and log display module；
The log collection module passes through log for the installation log collection system client on each application server
Collection system client collects the daily record data of each application server and is sent to log processing system；
The log processing module, for log processing system after receiving daily record data, first by original log number
It is saved according to duplication portion into log storage system；Original daily record data is screened, is cut, classified and closed again
And it handles, and new daily record data is saved in log storage system by treated；
The log display module is indexed for establishing log index in log display systems, and according to log from day
Daily record data is obtained in will storage system to be shown.
Further, the log processing module is also used to when handling original daily record data, log processing
System is scanned the log content attribute in new daily record data, and Auto-matching alarm regulation, and is touched according to alarm regulation
Hair log early warning system sends alert content and is alerted；The alert content includes application server IP, log alias, log
Path and log content.
Further, the log collection module specifically:
The installation log collection system client on each application server, and the configuration file of result collection system is carried out
The parameter of parameter configuration, configuration includes at least local IP address, journal file path, the merging of log multirow, Collection Rules, log
The address of processing system and log interval time；
The application program of running log collection system, just whether application program reads configuration file, and check configuration file
Really, it if it is not, then stopping the operation of application program, and sends prompting message and gives result collection system client；If it is, pressing
Corresponding journal file is scanned according to the Collection Rules configured in configuration file, and the daily record data scanned is sent to day in real time
Will processing system.
Further, a local server node and at least one server are provided in the log storage system
From node；
The log processing module specifically includes Log backup unit, journal processing unit and log storage unit；
The Log backup unit, for log processing system after receiving daily record data, by original daily record data
Duplication is a to give log storage system, and log storage system saves original daily record data to local server node, together
When original daily record data is synchronized to each server from node；
The journal processing unit carries out attributes extraction to original daily record data for log processing system, according to setting
Fixed attribute merges rule to merge new daily record data, and is classified according to the classification type of setting to new daily record data；
Wherein, the attribute of extraction include application server IP, log alias, log path, Log Types, server name, in log
Appearance, log rank or Custom Attributes；
The log storage unit, for new daily record data to be sent to log storage system, log storage by treated
System saves new daily record data to local server node, while new daily record data is synchronized to each server from node.
Further, the log display module specifically:
Log display systems are scanned the new daily record data being stored in log storage system, and according to user in day
The search rule set in the interface of will display systems indexes to establish log, to get corresponding day from log storage system
Will data；
Log display systems are shown according to different attributes to the daily record data of acquisition or log display systems
The user behavior of log is counted and shown by introducing the third-party library IP.
The present invention has the advantage that passing through client of the deployment for log collection on application server cluster in advance
End, with all kinds of resources on real-time monitoring server；Again by configuring the Collection Rules of client come to the abnormal number perceived
It is reported according to being collected, while abnormal data is filtered, classifies, stores and early warning server-side is unified, it not only can be very
The collection storage and retrieval of good realization multi-data source log, improve troubleshooting efficiency, promote data analysis capabilities, Er Qieke
It is laid a good foundation with the malfunction elimination well for the later period.
Detailed description of the invention
The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.
Fig. 1 is system architecture diagram of the present invention.
Fig. 2 is that a kind of multi-data source daily record data of the present invention concentrates the execution flow chart for collecting storage method.
Specific embodiment
It please refers to shown in Fig. 1 and Fig. 2, a kind of multi-data source daily record data of the present invention concentrates the preferable reality for collecting storage method
Example is applied, described method includes following steps:
Step S1, the installation log collection system client on each application server, passes through result collection system client
It collects the daily record data of each application server and is sent to log processing system；
Step S2, log processing system is first a to day by original daily record data duplication after receiving daily record data
It is saved in will storage system；Original daily record data is screened again, is cut, is classified and merging treatment, and will place
New daily record data after reason is saved in log storage system；
Step S3, log index is established in log display systems, and is obtained from log storage system according to log index
Daily record data is taken to be shown.
In the present invention, the step S2 further include:
When handling original daily record data, log processing system is to the log content attribute in new daily record data
It is scanned, and Auto-matching alarm regulation, and log early warning system transmission alert content is triggered according to alarm regulation and is accused
It is alert, to notify corresponding technical staff or administrator to handle in time alarm；In the specific implementation, such as when being matched to
Reach corresponding log rank (such as severity level), triggering log early warning system is alerted；For another example, continuous more when being matched to
When a log rank is ERROR, triggering log early warning system is alerted；Certainly, example provided above is only for
Illustrative, the present invention not and is only limitted to this, and in actual setting, corresponding alarm can also be arranged according to actual needs
Rule.The alert content includes application server IP (ip), log alias (index), log path (source) and day
Will content (message).In the present invention, log early warning system can support following Three models to realize alarm notification:
A: it is alarmed by third party's mailbox to send mail；
B: call wechat interface to send alarm notification by wechat；
C: the interface of nail nail office software is called to carry out alarm notification.
Certainly, the present invention is not limited to this, in the specific implementation, it is logical can also to increase other alarms as needed
Know mode, such as alarm notification can also be sent by short message, QQ etc..
The step S1 specifically:
The installation log collection system client on each application server, to be used for collector journal data, log collection system
The installation kit for client of uniting is a compiled binary file, can directly be executed on the application server, and in installation kit
It is interior to carry out parameter configuration to configuration file；Parameter configuration is carried out to the configuration file of result collection system, the parameter of configuration is extremely
It less include local IP address, journal file path, the merging of log multirow, Collection Rules, the address of log processing system and day
Will interval time, certainly, enumerated above is only that some more common parameter configurations in the specific implementation can be with root
Increase other parameter configurations according to actual needs；
The application program of running log collection system, just whether application program reads configuration file, and check configuration file
Really, it if it is not, then stopping the operation of application program, and sends prompting message and gives result collection system client；If it is, pressing
Corresponding journal file is scanned according to the Collection Rules configured in configuration file, and the daily record data scanned is sent to day in real time
Will processing system；Wherein, the journal file range of scanning includes: using log, operating system log and hardware resource day
Will.
A local server node (Master) and at least one server are provided in the log storage system
From node (slave)；
The step S2 is specifically included:
Step S21, log processing system is a to log by original daily record data duplication after receiving daily record data
Storage system, log storage system save original daily record data to local server node, while by original log
Data are synchronized to each server from node；
In the present invention, the daily record data for retaining a most original in log storage system by elder generation, may make subsequent
It can be convenient progress log to trace to the source；And specifically when carrying out the preservation of log data, need first in local server node
An init file is established, and original daily record data is stored in init file, log data is in init text
The format saved in part folder are as follows: Log Names _ server ip _ timestamp .log, wherein timestamp can be as accurate as millisecond, example
Such as: IQWebService_127.0.0.1_20190325161905.log.Meanwhile local server node can automaticly inspect and work as
Preceding online server is synchronized to each server from node from node, and by original daily record data.
Step S22, log processing system carries out attributes extraction to original daily record data, merges rule according to the attribute of setting
Then to merge new daily record data, and classified according to the classification type of setting to new daily record data；Wherein, the attribute of extraction
Including application server IP (ip), log alias (index), log path (source), Log Types (type), server name
Claim (hostname), log content (message), log rank (level) or Custom Attributes；
In the specific implementation, highest priority of the Custom Attributes in log processing system, log processing system are not
Custom Attributes can be filtered out；Attribute, which merges rule, to be set according to actual needs, for example, setting
Attribute merges rule are as follows: log alias+application server IP+ log path+log content+Custom Attributes, then specific
When implementation, rule is just merged according to the attribute and merges into new daily record data；Classification type is to apply by default
Server ip is come what is classified, and certainly, the present invention is not limited to this, in the specific implementation, can also be according to actual needs
Other classification types are arranged.
Step S23, will treated new daily record data is sent to log storage system, log storage system is by new log number
According to saving on local server node, while new daily record data is synchronized to each server from node.In specific implementation, it needs
A main file first is established in local server node, and new daily record data is stored in main file, new day
The format that will data save in main file are as follows: Log Names _ server ip _ timestamp .log.
The step S3 specifically:
Log display systems are scanned the new daily record data being stored in log storage system, and according to user in day
The search rule set in the interface of will display systems indexes to establish log, to get corresponding day from log storage system
Will data；By default, search rule is the index to establish log alias property (index), and shows in log and be
Index can be retrieved in the search box of system, and can support regular expression to be retrieved；Certainly, it is being embodied
When, other search rules can also be established according to actual needs.
Log display systems are shown the daily record data of acquisition according to different attributes, for example, can be according to not
With attribute draw bar chart, pie chart, line chart etc., and user is showed to check；
Or log display systems are counted and are shown to the user behavior of log by introducing the third-party library IP,
Such as web log, the case where the case where Global Subscriber access can be counted, page request etc., and with bar chart, cake
Figure, line chart etc. are shown.
In the specific implementation, log display systems can also provide user authority management, specifically: pass through
UserManager function authorizes user, and system default adds in background data base table user administrator's note
Record；Showing interface creates user's list, and list can be written in background data base user table for userManager function after submission,
Using md5 encryption storage, logging in every time all can go to match md5 value and user name in database user password.
Refering to Figure 1, a kind of multi-data source daily record data of the present invention concentrates the preferred embodiment for collecting storage device,
Described device includes log collection module, log processing module and log display module；
The log collection module passes through log for the installation log collection system client on each application server
Collection system client collects the daily record data of each application server and is sent to log processing system；
The log processing module, for log processing system after receiving daily record data, first by original log number
It is saved according to duplication portion into log storage system；Original daily record data is screened, is cut, classified and closed again
And it handles, and new daily record data is saved in log storage system by treated；
The log display module is indexed for establishing log index in log display systems, and according to log from day
Daily record data is obtained in will storage system to be shown.
In the present invention, the log processing module is also used to when handling original daily record data, at log
Reason system is scanned the log content attribute in new daily record data, and Auto-matching alarm regulation, and according to alarm regulation
Triggering log early warning system sends alert content and is alerted, to notify corresponding technical staff or administrator in time to alarm
It is handled；In the specific implementation, such as when matching reaches corresponding log rank (such as severity level), just triggering log is pre-
Alert system is alerted；For another example, when being matched to continuous multiple log ranks and being ERROR, just trigger log early warning system into
Row alarm；Certainly, example provided above is only for illustrative, and the present invention not and is only limitted to this, in actual setting,
Corresponding alarm regulation can also be set according to actual needs.The alert content includes application server IP (ip), log
Alias (index), log path (source) and log content (message).In the present invention, log early warning system can be with
Following Three models are supported to realize alarm notification:
A: it is alarmed by third party's mailbox to send mail；
B: call wechat interface to send alarm notification by wechat；
C: the interface of nail nail office software is called to carry out alarm notification.
Certainly, the present invention is not limited to this, in the specific implementation, it is logical can also to increase other alarms as needed
Know mode, such as alarm notification can also be sent by short message, QQ etc..
The log collection module specifically:
The installation log collection system client on each application server, to be used for collector journal data, log collection system
The installation kit for client of uniting is a compiled binary file, can directly be executed on the application server, and in installation kit
It is interior to carry out parameter configuration to configuration file；Parameter configuration is carried out to the configuration file of result collection system, the parameter of configuration is extremely
It less include local IP address, journal file path, the merging of log multirow, Collection Rules, the address of log processing system and day
Will interval time, certainly, enumerated above is only that some more common parameter configurations in the specific implementation can be with root
Increase other parameter configurations according to actual needs；
The application program of running log collection system, just whether application program reads configuration file, and check configuration file
Really, it if it is not, then stopping the operation of application program, and sends prompting message and gives result collection system client；If it is, pressing
Corresponding journal file is scanned according to the Collection Rules configured in configuration file, and the daily record data scanned is sent to day in real time
Will processing system；Wherein, the journal file range of scanning includes: using log, operating system log and hardware resource day
Will.
A local server node (Master) and at least one server are provided in the log storage system
From node (slave)；
The log processing module specifically includes Log backup unit, journal processing unit and log storage unit；
The Log backup unit, for log processing system after receiving daily record data, by original daily record data
Duplication is a to give log storage system, and log storage system saves original daily record data to local server node, together
When original daily record data is synchronized to each server from node；
In the present invention, the daily record data for retaining a most original in log storage system by elder generation, may make subsequent
It can be convenient progress log to trace to the source；And specifically when carrying out the preservation of log data, need first in local server node
An init file is established, and original daily record data is stored in init file, log data is in init text
The format saved in part folder are as follows: Log Names _ server ip _ timestamp .log, wherein timestamp can be as accurate as millisecond, example
Such as: IQWebService_127.0.0.1_20190325161905.log.Meanwhile local server node can automaticly inspect and work as
Preceding online server is synchronized to each server from node from node, and by original daily record data.
The journal processing unit carries out attributes extraction to original daily record data for log processing system, according to setting
Fixed attribute merges rule to merge new daily record data, and is classified according to the classification type of setting to new daily record data；
Wherein, the attribute of extraction includes application server IP (ip), log alias (index), log path (source), Log Types
(type), server name (hostname), log content (message), log rank (level) or Custom Attributes；
In the specific implementation, highest priority of the Custom Attributes in log processing system, log processing system are not
Custom Attributes can be filtered out；Attribute, which merges rule, to be set according to actual needs, for example, setting
Attribute merges rule are as follows: log alias+application server IP+ log path+log content+Custom Attributes, then specific
When implementation, rule is just merged according to the attribute and merges into new daily record data；Classification type is to apply by default
Server ip is come what is classified, and certainly, the present invention is not limited to this, in the specific implementation, can also be according to actual needs
Other classification types are arranged.
The log storage unit, for new daily record data to be sent to log storage system, log storage by treated
System saves new daily record data to local server node, while new daily record data is synchronized to each server from node.
In specific implementation, it needs first to establish a main file in local server node, and new daily record data is stored in
In main file, format that new daily record data saves in main file are as follows: Log Names _ server ip _ timestamp
.log。
The log display module specifically:
Log display systems are scanned the new daily record data being stored in log storage system, and according to user in day
The search rule set in the interface of will display systems indexes to establish log, to get corresponding day from log storage system
Will data；By default, search rule is the index to establish log alias property (index), and shows in log and be
Index can be retrieved in the search box of system, and can support regular expression to be retrieved；Certainly, it is being embodied
When, other search rules can also be established according to actual needs.
Log display systems are shown the daily record data of acquisition according to different attributes, for example, can be according to not
With attribute draw bar chart, pie chart, line chart etc., and user is showed to check；
Or log display systems are counted and are shown to the user behavior of log by introducing the third-party library IP,
Such as web log, the case where the case where Global Subscriber access can be counted, page request etc., and with bar chart, cake
Figure, line chart etc. are shown.
In the specific implementation, log display systems can also provide user authority management, specifically: pass through
UserManager function authorizes user, and system default adds in background data base table user administrator's note
Record；Showing interface creates user's list, and list can be written in background data base user table for userManager function after submission,
Using md5 encryption storage, logging in every time all can go to match md5 value and user name in database user password.
In conclusion the present invention has the advantage that by the way that deployment is used for log receipts on application server cluster in advance
The client of collection, with all kinds of resources on real-time monitoring server；Again by configuring the Collection Rules of client come to perceiving
Abnormal data be collected and report, while abnormal data be filtered, classify, store and early warning server-side is unified, no
It only can be very good to realize the collection storage of multi-data source log and retrieve, improve troubleshooting efficiency, promote data and analyze energy
Power, and can be very good to lay a good foundation for the malfunction elimination in later period.
Although specific embodiments of the present invention have been described above, those familiar with the art should be managed
Solution, we are merely exemplary described specific embodiment, rather than for the restriction to the scope of the present invention, it is familiar with this
The technical staff in field should be covered of the invention according to modification and variation equivalent made by spirit of the invention
In scope of the claimed protection.
Claims (10)
1. a kind of multi-data source daily record data, which is concentrated, collects storage method, it is characterised in that: described method includes following steps:
Step S1, the installation log collection system client on each application server, is collected by result collection system client
The daily record data of each application server is simultaneously sent to log processing system；
Step S2, log processing system first deposits original daily record data duplication portion to log after receiving daily record data
It is saved in storage system；Original daily record data is screened again, is cut, is classified and merging treatment, and will be after processing
New daily record data be saved in log storage system；
Step S3, log index is established in log display systems, and day is obtained from log storage system according to log index
Will data are shown.
2. a kind of multi-data source daily record data according to claim 1, which is concentrated, collects storage method, it is characterised in that: described
Step S2 further include:
When handling original daily record data, log processing system carries out the log content attribute in new daily record data
Scanning, and Auto-matching alarm regulation, and log early warning system transmission alert content is triggered according to alarm regulation and is alerted；Institute
Stating alert content includes application server IP, log alias, log path and log content.
3. a kind of multi-data source daily record data according to claim 1, which is concentrated, collects storage method, it is characterised in that: described
Step S1 specifically:
The installation log collection system client on each application server, and parameter is carried out to the configuration file of result collection system
The parameter of configuration, configuration includes at least local IP address, journal file path, the merging of log multirow, Collection Rules, log processing
The address of system and log interval time；
The application program of running log collection system, application program reads configuration file, and checks whether configuration file is correct, such as
Fruit is no, then stops the operation of application program, and sends prompting message and give result collection system client；If it is, according to matching
It sets the Collection Rules configured in file and scans corresponding journal file, and the daily record data scanned is sent at log in real time
Reason system.
4. a kind of multi-data source daily record data according to claim 1, which is concentrated, collects storage method, it is characterised in that: described
A local server node and at least one server are provided in log storage system from node；
The step S2 is specifically included:
Step S21, original daily record data duplication portion is given log to store by log processing system after receiving daily record data
System, log storage system save original daily record data to local server node, while by original daily record data
Each server is synchronized to from node；
Step S22, log processing system carries out attributes extraction to original daily record data, merges rule according to the attribute of setting and comes
Merge new daily record data, and is classified according to the classification type of setting to new daily record data；Wherein, the attribute of extraction includes
Application server IP, log alias, log path, Log Types, server name, log content, log rank are made by oneself
Adopted attribute；
Step S23, by treated, new daily record data is sent to log storage system, and log storage system protects new daily record data
It deposits to local server node, while new daily record data is synchronized to each server from node.
5. a kind of multi-data source daily record data according to claim 1, which is concentrated, collects storage method, it is characterised in that: described
Step S3 specifically:
Log display systems are scanned the new daily record data being stored in log storage system, and according to user in log exhibition
Show the search rule set in the interface of system to establish log index, to get corresponding log number from log storage system
According to；
Log display systems are shown according to different attributes to the daily record data of acquisition or log display systems pass through
The third-party library IP is introduced the user behavior of log is counted and be shown.
6. a kind of multi-data source daily record data, which is concentrated, collects storage device, it is characterised in that: described device includes log collection mould
Block, log processing module and log display module；
The log collection module passes through log collection for the installation log collection system client on each application server
System client collects the daily record data of each application server and is sent to log processing system；
The log processing module first answers original daily record data for log processing system after receiving daily record data
System portion is saved into log storage system；Original daily record data is screened again, is cut, is classified and merging at
Reason, and new daily record data is saved in log storage system by treated；
The log display module for establishing log index in log display systems, and is indexed from log according to log and is deposited
Daily record data is obtained in storage system to be shown.
7. a kind of multi-data source daily record data according to claim 6, which is concentrated, collects storage device, it is characterised in that: described
Log processing module is also used to when handling original daily record data, and log processing system is in new daily record data
Log content attribute is scanned, and Auto-matching alarm regulation, and is triggered log early warning system according to alarm regulation and sent police
Report content is alerted；The alert content includes application server IP, log alias, log path and log content.
8. a kind of multi-data source daily record data according to claim 6, which is concentrated, collects storage device, it is characterised in that: described
Log collection module specifically:
The installation log collection system client on each application server, and parameter is carried out to the configuration file of result collection system
The parameter of configuration, configuration includes at least local IP address, journal file path, the merging of log multirow, Collection Rules, log processing
The address of system and log interval time；
The application program of running log collection system, application program reads configuration file, and checks whether configuration file is correct, such as
Fruit is no, then stops the operation of application program, and sends prompting message and give result collection system client；If it is, according to matching
It sets the Collection Rules configured in file and scans corresponding journal file, and the daily record data scanned is sent at log in real time
Reason system.
9. a kind of multi-data source daily record data according to claim 6, which is concentrated, collects storage device, it is characterised in that: described
A local server node and at least one server are provided in log storage system from node；
The log processing module specifically includes Log backup unit, journal processing unit and log storage unit；
The Log backup unit replicates original daily record data for log processing system after receiving daily record data
Portion gives log storage system, and log storage system saves original daily record data to local server node, simultaneously will
Original daily record data is synchronized to each server from node；
The journal processing unit carries out attributes extraction to original daily record data for log processing system, according to setting
Attribute merges rule to merge new daily record data, and is classified according to the classification type of setting to new daily record data；Wherein,
The attribute of extraction includes application server IP, log alias, log path, Log Types, server name, log content, day
Will rank or Custom Attributes；
The log storage unit, for new daily record data to be sent to log storage system, log storage system by treated
New daily record data is saved to local server node, while new daily record data is synchronized to each server from node.
10. a kind of multi-data source daily record data according to claim 6, which is concentrated, collects storage device, it is characterised in that:
The log display module specifically:
Log display systems are scanned the new daily record data being stored in log storage system, and according to user in log exhibition
Show the search rule set in the interface of system to establish log index, to get corresponding log number from log storage system
According to；
Log display systems are shown according to different attributes to the daily record data of acquisition or log display systems pass through
The third-party library IP is introduced the user behavior of log is counted and be shown.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910343502.7A CN110209518A (en) | 2019-04-26 | 2019-04-26 | A kind of multi-data source daily record data, which is concentrated, collects storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910343502.7A CN110209518A (en) | 2019-04-26 | 2019-04-26 | A kind of multi-data source daily record data, which is concentrated, collects storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110209518A true CN110209518A (en) | 2019-09-06 |
Family
ID=67786606
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910343502.7A Pending CN110209518A (en) | 2019-04-26 | 2019-04-26 | A kind of multi-data source daily record data, which is concentrated, collects storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209518A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780857A (en) * | 2019-10-23 | 2020-02-11 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN110958149A (en) * | 2019-12-09 | 2020-04-03 | 南京富士通南大软件技术有限公司 | Method for automatically aggregating and displaying OpenStack logs |
| CN111368039A (en) * | 2020-03-10 | 2020-07-03 | 澳克多普有限公司 | Data management system |
| CN111782473A (en) * | 2020-06-30 | 2020-10-16 | 中国工商银行股份有限公司 | Distributed log data processing method, device and system |
| CN112506761A (en) * | 2020-11-26 | 2021-03-16 | 福州智象信息技术有限公司 | Production environment server interface debugging method and system |
| CN113138971A (en) * | 2021-04-30 | 2021-07-20 | 深圳市度申科技有限公司 | Visual log analysis method, device and system |
| CN113377623A (en) * | 2021-07-02 | 2021-09-10 | 华青融天(北京)软件股份有限公司 | Automatic generation method and device of alarm rule and electronic equipment |
| CN113590556A (en) * | 2021-07-30 | 2021-11-02 | 中国工商银行股份有限公司 | Database-based log processing method, device and equipment |
| US11966323B2 (en) | 2021-01-05 | 2024-04-23 | Red Hat, Inc. | Troubleshooting software services based on system calls |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102750326A (en) * | 2012-05-30 | 2012-10-24 | 浪潮电子信息产业股份有限公司 | Log management optimization method of cluster system based on downsizing strategy |
| CN102981943A (en) * | 2012-10-29 | 2013-03-20 | 新浪技术（中国）有限公司 | Method and system for monitoring application logs |
| US20160378577A1 (en) * | 2015-04-03 | 2016-12-29 | Oracle International Corporation | Method and system for implementing an operating system hook in a log analytics system |
| CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
| CN108399199A (en) * | 2018-01-30 | 2018-08-14 | 武汉大学 | A kind of collection of the application software running log based on Spark and service processing system and method |
-
2019
- 2019-04-26 CN CN201910343502.7A patent/CN110209518A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102750326A (en) * | 2012-05-30 | 2012-10-24 | 浪潮电子信息产业股份有限公司 | Log management optimization method of cluster system based on downsizing strategy |
| CN102981943A (en) * | 2012-10-29 | 2013-03-20 | 新浪技术（中国）有限公司 | Method and system for monitoring application logs |
| US20160378577A1 (en) * | 2015-04-03 | 2016-12-29 | Oracle International Corporation | Method and system for implementing an operating system hook in a log analytics system |
| CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
| CN108399199A (en) * | 2018-01-30 | 2018-08-14 | 武汉大学 | A kind of collection of the application software running log based on Spark and service processing system and method |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780857A (en) * | 2019-10-23 | 2020-02-11 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN110780857B (en) * | 2019-10-23 | 2024-01-30 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN110958149B (en) * | 2019-12-09 | 2022-08-16 | 南京富士通南大软件技术有限公司 | Method for automatically aggregating and displaying OpenStack logs |
| CN110958149A (en) * | 2019-12-09 | 2020-04-03 | 南京富士通南大软件技术有限公司 | Method for automatically aggregating and displaying OpenStack logs |
| CN111368039A (en) * | 2020-03-10 | 2020-07-03 | 澳克多普有限公司 | Data management system |
| CN111368039B (en) * | 2020-03-10 | 2024-03-22 | 澳克多普有限公司 | Data management system |
| CN111782473A (en) * | 2020-06-30 | 2020-10-16 | 中国工商银行股份有限公司 | Distributed log data processing method, device and system |
| CN112506761B (en) * | 2020-11-26 | 2022-07-05 | 福州智象信息技术有限公司 | Production environment server interface debugging method and system |
| CN112506761A (en) * | 2020-11-26 | 2021-03-16 | 福州智象信息技术有限公司 | Production environment server interface debugging method and system |
| US11966323B2 (en) | 2021-01-05 | 2024-04-23 | Red Hat, Inc. | Troubleshooting software services based on system calls |
| CN113138971A (en) * | 2021-04-30 | 2021-07-20 | 深圳市度申科技有限公司 | Visual log analysis method, device and system |
| CN113377623A (en) * | 2021-07-02 | 2021-09-10 | 华青融天(北京)软件股份有限公司 | Automatic generation method and device of alarm rule and electronic equipment |
| CN113377623B (en) * | 2021-07-02 | 2024-05-28 | 华青融天(北京)软件股份有限公司 | Automatic generation method and device of alarm rules and electronic equipment |
| CN113590556A (en) * | 2021-07-30 | 2021-11-02 | 中国工商银行股份有限公司 | Database-based log processing method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209518A (en) | A kind of multi-data source daily record data, which is concentrated, collects storage method and device | |
| CN110493348B (en) | Intelligent monitoring alarm system based on Internet of things | |
| CN109542733B (en) | High-reliability real-time log collection and visual retrieval method | |
| CN105159964B (en) | A kind of log monitoring method and system | |
| CN107832196B (en) | Monitoring device and monitoring method for abnormal content of real-time log | |
| CN108833137A (en) | A kind of flexibility micro services Monitoring framework framework | |
| CN106571960B (en) | Log collection management system and method | |
| CN107508722B (en) | Service monitoring method and device | |
| CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
| CN111885012A (en) | Network situation perception method and system based on information acquisition of various network devices | |
| CN107612779B (en) | Secondary safety protection network equipment of dispatching data network and service operation monitoring system | |
| CN109871392B (en) | Slow sql real-time data acquisition method under distributed application system | |
| CN104022903A (en) | One-stop automatic operation and maintaining system | |
| CN107229556A (en) | Log Analysis System based on elastic components | |
| CN110083391A (en) | Call request monitoring method, device, equipment and storage medium | |
| CN111259073A (en) | Intelligent business system running state studying and judging system based on logs, flow and business access | |
| CN111782345B (en) | Container cloud platform log collection and analysis alarm method | |
| Vaarandi et al. | Event log analysis with the LogCluster tool | |
| CN107239502B (en) | Database error daily record monitoring system based on salt | |
| CN110224865A (en) | A kind of log warning system based on Stream Processing | |
| CN110336863B (en) | Data reporting method and system | |
| CN108390907B (en) | Management monitoring system and method based on Hadoop cluster | |
| CN111258971A (en) | Application state monitoring alarm system and method based on access log | |
| CN114328124A (en) | Method and device for business monitoring, storage medium and electronic device | |
| CN108390782A (en) | A kind of centralization application system performance question synthesis analysis method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190906 |