TWI685736B - Method for remotely clearing abnormal status of racks applied in data center - Google Patents

Abstract

A method for remotely clearing abnormal status of racks is disclosed and includes following steps: obtaining each information of a rack management controller (RMC) and multiple baseboard management controllers (BMCs) of a rack regularly by a management system; recording each operating action performed by manager through the management system; analyzing the information and the operating action by the management system for determining whether any RMC or BMC is under one of multiple default attention-statuses; and, automatically performing a remotely service re-starting procedure to one of the RMC and the BMCs for preventing the RMC or the BMC from occurring an abnormal status when the RMC or the BMC is determined keeping a connection with the management system but about to occur the abnormal status.

Description

The remote elimination method of the abnormal state of the cabinet used in the data center (2)

The invention relates to a data center, in particular to a method for analyzing and eliminating the abnormal state of the cabinet in the data center.

Generally speaking, a data center usually uses intelligent platform management interface (Intelligent Platform Management Interface, IPMI) to control the rack management controller (RMC) and baseboard of the cabinets, endpoint servers and other equipment in the data center The management controller (Baseboard Management Controller, BMC) performs remote management.

No matter which method is used for remote management, as long as the RMC or BMC of any cabinet or endpoint server is abnormal, the administrator will receive many warning letters. However, it is generally difficult for managers to directly know the real problem of the status through these warning letters in the first place, and it is often necessary to continue to advance with time until the hundreds of warning letters are received and the connection with the device is lost. An abnormality occurred in RMC and BMC.

What's more, even if some management platforms collect error messages from different monitoring pipelines and submit failure assessment reports to managers after aggregation, such monitoring methods still need to be managed by The manager makes the final judgment and decides the processing method. However, as long as human factors intervene, the possibility of misjudgment cannot be completely avoided.

In view of this, the field really needs to develop a novel system and method that can automatically implement a remote repair mechanism for RMC and BMC in an abnormal state, thereby strengthening the monitoring capabilities of the data center, so that the cabinet management can be highly automated, while Reduce the time lost by human judgment, and avoid human misjudgment.

The main purpose of the present invention is to provide a remote elimination method for the abnormal state of the cabinet used in the data center. When it is judged that the cabinet management controller or the baseboard management controller is connected normally but it is judged that the abnormal state may be imminent, To prevent the cabinet management controller or baseboard management controller from entering the abnormal state.

In order to achieve the above-mentioned object, the method of the present invention is that a cabinet server management system periodically obtains various information of a cabinet management controller and a plurality of baseboard management controllers in a cabinet remotely, and records a manager’s pass The operation actions performed by the cabinet server management system. The rack server management system analyzes the above information and operation behaviors to determine whether the rack management controller or each baseboard management controller in the rack is in one of a variety of preset states of interest.

If it is determined that the connection between any rack management controller or baseboard management controller and the rack server management system is normal, but it is determined that an abnormal state may soon occur, the rack server management system automatically implements a remote service restart mechanism to Prevent the cabinet management controller or each baseboard management controller from entering an abnormal state.

Compared with the related art, the method of the present invention is analyzed by the rack server management system connected to the rack and automatically implements the remote service restart mechanism. There is no need to wait for the manager to artificially determine the abnormal state, which can greatly reduce the management cost. The monitoring of the cabinet does not need human intervention, and is not affected by distance and time.

1‧‧‧Data Center

2‧‧‧ Cabinet

21‧‧‧ Cabinet management controller

211, 221‧‧‧ network interface controller

22‧‧‧Baseboard management controller

23‧‧‧Internal network switch

24‧‧‧Internal hardware circuit

3‧‧‧ Cabinet server management system

31‧‧‧ Database

4‧‧‧Public network switch

S11~S15, S21~S28‧‧‧‧Collection steps

S31~S39‧‧‧‧Analysis and elimination steps

S41~S47, S51~S58, S61~S66, S71~S80

FIG. 1 is a schematic diagram of the data center of the present invention.

FIG. 2 is a first specific embodiment of a block diagram of a cabinet of the present invention.

FIG. 3A is a first specific embodiment of the data collection flowchart of the present invention.

FIG. 3B is a second specific embodiment of the data collection flowchart of the present invention.

FIG. 4 is a first specific embodiment of the analysis and elimination flowchart of the present invention.

FIG. 5 is a first specific embodiment of the first type of attention state elimination flowchart of the present invention.

FIG. 6 is a second specific embodiment of the first type of attention state elimination flowchart of the present invention.

FIG. 7 is a first specific embodiment of a second type of attention state elimination flowchart of the present invention.

FIG. 8 is a first specific embodiment of the third type of attention state elimination flowchart of the present invention.

The following is a detailed description of a preferred embodiment of the present invention with reference to the drawings.

The present invention discloses a remote elimination method for abnormal conditions of cabinets (hereinafter referred to as the elimination method in the specification). The elimination method is mainly used in a data center to help managers automatically monitor, analyze, and exclude data in the data center. Abnormal state.

Refer to FIG. 1 for a schematic diagram of the data center of the present invention. As shown in FIG. 1, the data center 1 according to the present invention mainly includes a plurality of cabinets 2 and a cabinet server management system 3 (hereinafter referred to as a management system 3 for short) connected to the plurality of cabinets 2 from a remote end. The management system 3 may be installed inside or outside the data center 1 and connected to the public network switch 4 via the network, and then connected to the plurality of cabinets 2 in the data center 1 via the public network switch 4.

The management system 3 of the present invention can monitor the plural cabinets 2 in the data center 1 in real time, obtain various information of the plural cabinets 2, and analyze the information. When it is found that any cabinet 2 has an abnormal state or an abnormal state is about to occur, the management system 3 of the present invention can automatically implement a corresponding processing mechanism to eliminate the situation. In this way, the present invention can eliminate the abnormal state that has occurred in the cabinet 2 or prevent the abnormal state that may occur soon without the need for human intervention, greatly reducing human misjudgment, and increasing processing speed.

In an embodiment, the management system 3 may be a personal computer or a cloud server, which has one or more central processing units (not shown in the figure). After the management system 3 is activated, it can be connected to a plurality of cabinets 2 in the data center 1 through a public network cross machine 4, and one or more central processing units can execute specific applications and algorithms to achieve these The monitoring, data analysis and elimination of abnormal status of cabinet 2.

The management system 3 also has a data base 31 for temporarily storing or permanently storing various information obtained from the plural cabinets 2 in the data center 1. In the embodiment of FIG. 1, the database 31 is built in the management system 3. In other embodiments, the management system 3 may also be externally connected to one or more databases 31 without limitation.

Referring to FIG. 2, it is a first specific embodiment of the block diagram of the cabinet of the present invention. In the embodiment of FIG. 2, a single cabinet 2 in the data center 1 is connected to the management system 3 as an example for description. However, the data center 1 may be provided with multiple cabinets 2 according to actual needs, instead of the figure. 2 is limited.

As shown in FIG. 2, the cabinet 2 of the present invention mainly includes at least one Rack Management Controller (RMC) 21, and multiple endpoint servers 220 connected to the RMC 21, wherein each endpoint server 220 is provided with At least one baseboard management controller (Baseboard Management Controller, BMC) 22.

The RMC21 is an embedded system, which is installed in the cabinet 2 and assists in processing all external hardware devices (cooling fans, various sensors or power supplies, etc.) of the cabinet 2 through various hardware circuits. Communicate and communicate with BMC22 of all endpoint servers 220 in cabinet 2. The BMC 22 is also an embedded system, which is installed in the endpoint server 220 and assists in processing all external communications of the internal hardware devices (various sensors and other devices) of the endpoint server 220.

In this embodiment, the RMC 21 connects to the BMCs 22 of all the endpoint servers 220 in the cabinet 2 through internal hardware lines 24, and communicates with the BMCs 22 to control the endpoint servers 220 and obtain the required information. In this embodiment, the endpoint server may be, for example, an upright server (Tower Model Server) or a blade server (Blade Server), but it is not limited.

As shown in FIG. 2, each endpoint server 220 installed in the cabinet 2 has a fixed position number (such as #1, #2, #n, etc. in FIG. 2). When the endpoint server 220 or When the external network function of BMC22 fails, RMC21 can be connected to the specified location in cabinet 2 (such as #1, #2, #n above) through internal hardware line 24, and then serve with the endpoint at the specified location 220 and BMC22 communication. In this way, even if the endpoint server 220 or the BMC 22 loses the network connection, the cabinet 2 can still monitor and manage each BMC 22 through the RMC 21 and eliminate the abnormal condition of each BMC 22.

In addition, the RMC 21 of the present invention is provided with a network interface controller (NIC) 211, and each BMC 22 is also provided with a network interface controller 221, respectively. The RMC 21 is connected to the internal network switch 23 inside the cabinet 2 through the NIC 211, and each BMC 22 is connected to the internal network switch 23 through the respective NIC 221. The cabinet 2 is connected to the public network switch 4 through the internal network switch 23, and establishes a network connection with the management system 3 through the public network switch 4. In this way, the management system 3 can remotely access the cabinet 2 in the data center 1 via the network, thereby inquiring and acquiring all the information of all RMC21 and BMC22 in the cabinet 2 and storing it in the database 31.

The main technical feature of the present invention is that the management system 3 can regularly access the cabinet 2 via the network and obtain all the information of all RMC21 and BMC22 in the cabinet 2 (such as status data, event log, system resource utilization rate, The sensing values of the internal sensors of the endpoint server 220, etc.) are used to actively analyze whether the RMC21 and the BMC22 have an abnormal state, or an abnormal state is about to occur. When the management system 3 considers it necessary after analysis, it can actively implement the corresponding mechanism at the remote end to directly rule out the abnormal state of the RMC21 and/or BMC22 at the remote end, or prevent the RMC21 and/or BMC22 from entering the Abnormal state.

The technical solution of the present invention can perform abnormal state processing without human intervention at all, greatly reducing the possibility of human misjudgment, and can make the monitoring of the cabinet 2 highly automated.

Please refer to FIG. 3A for the first specific embodiment of the data collection flowchart of the present invention.

As shown in FIG. 3A, if the manager wants to monitor the cabinet 2 in the data center 1, the manager can directly start the remote management system 3 (step S11). When the management system 3 is started, it will take the initiative to remotely access the cabinet 2 in the data center 1 (take the single cabinet 2 in FIG. 2 as an example) RMC21 and all BMC22 (step S12). In addition, the management system 3 obtains the information of the RMC 21 and all the BMC 22 in the cabinet 2 through remote access (step S13), and then stores the obtained information in the local data 31 (step S14).

Specifically, in this embodiment, the management system 3 actively accesses the cabinet 2 regularly after startup, that is, the access operations, information acquisition operations, and storage operations in steps S12, S13, and S14 are regarded as routine programs after startup (routine ). When the above routine is executed, it is continuously determined whether the management system 3 is shut down (step S15), and before the management system 3 is shut down, the above steps S12 to S14 are continuously performed to continuously monitor the RMC21 and the BMC22 in the cabinet 2.

Referring to FIG. 3B, it is a second specific embodiment of the data collection flowchart of the present invention.

In this embodiment, after the manager activates the management system 3 (step S21), the management system 3 may provide an operation interface (step S22). Through this operation interface, the administrator can log in to the management system 3, and use the management system 3 to remotely monitor and control the information of each cabinet 2 in the data center 1. In this embodiment, the operation interface may be a physical interface or a web interface (Web) interface, which is not limited.

After the operation interface is provided, the management system 3 continuously determines whether the operation performed by the manager is accepted through the operation interface (step S23). If the operation of the manager is indeed accepted, the management system 3 performs remote management on the cabinet 2 and the RMC 21 and the BMC 22 in the cabinet 2 from the remote according to the operation behavior of the manager (step S24). Next, the management system 3 can record the above-mentioned operation behavior of the manager (step S25), and can also obtain and record the management system 3, the cabinet 2, each endpoint server 220, and the RMC21, BMC22 generated by the remote management Feedback information such as feedback, system parameters and execution data (step S26). Finally, the management system 3 also stores the operation behavior and feedback information in the database 31 (step S27) to facilitate subsequent analysis actions for abnormal states.

Similarly, the management system 3 of the present embodiment regards the actions of steps S22 to S27 as the activated routine. During the execution of the above routine, continuously determine whether the management system 3 is shut down (step S28), and continue to execute the above steps S22 to S27 before the management system 3 is shut down, to continuously monitor and analyze the operation actions performed by the manager to the cabinet 2 The impact of RMC21 and BMC22.

Please refer to FIG. 4 for the first specific embodiment of the analysis and elimination flowchart of the present invention.

As shown in FIG. 4, in this embodiment, the management system 3 periodically accesses the database 31 (step S31), and obtains the RMC21 and BMC22 information in the cabinet 2 from the database 31, the operation behavior of the manager, and Various feedback information (step S32), and analyze it. With the above data, the management system 3 can analyze whether the RMC 21 and each BMC 22 in the cabinet 2 are in one of the preset multiple attention states (step S33).

In an embodiment, the management system 3 can obtain various information of the RMC 21 and BMC 22 in the cabinet 2 in real time, obtain the operation behavior of the manager from the operation interface in real time, and analyze accordingly. In another embodiment, the management system 3 may periodically store the above-mentioned data into the database 31 through step S14 of FIG. 3A and step S27 of FIG. 3B, and periodically read the above-mentioned data from the database 31 for analysis. Not limited.

In one embodiment, the above information of RMC21 and BMC22 can be, for example, status data (such as currently in working mode or update mode, IP address, MAC address, subnet mask, gateway IP address) , The number of IPMI sessions, etc.), event logs, etc., and the above operations may be, for example, data query operations, update operations, and resets performed by a manager for a specific cabinet 2, endpoint server 220, or RMC21, BMC22 Homework, etc., but not limited. Through the above information, the Department of Management The system 3 can analyze whether the RMC 21 or the BMC 22 in need of real-time rescue is currently in the cabinet 2 by executing a corresponding algorithm.

In the embodiment of FIG. 4, the management system 3 can preset at least three types of attention states, including a first-type attention state, a second-type attention state, and a third-type attention state, of which the three types of attention states are Corresponding to different abnormal conditions of RMC21/BMC22, it is necessary for the management system 3 to implement different mechanisms directly at the remote end to eliminate or prevent them.

As shown in Figure 4, if the management system 3 analyzes based on the above data (mainly based on status data, event logs, and manager's operating behavior), it is found that any RMC21 or BMC22 has been in an abnormal state, but has not been lost from the management system 3 Connected, it will be determined that the RMC21 or BMC22 is in the first type of attention state (step S34). When any RMC21 or BMC22 is found in the first type of attention state, the management system 3 can automatically implement a remote recovery mechanism for the RMC21 or BMC22 in the first type of attention state to remotely release the abnormal state of RMC21 or BMC22 (step S37) .

If the management system 3 analyzes based on the above data (mainly based on the RMC21 and BMC22 status data) and finds that any RMC21 or BMC22 is connected to the management system 3 normally, but judges that an abnormal state may occur soon, it will be determined that this RMC21 or BMC22 It is in the second type of attention state (step S35). When any RMC21 or BMC22 is found to be in the second type of state of concern, the management system 3 can automatically implement a remote service restart mechanism for the RMC21 or BMC22 in the second type of state of attention, so as to remotely prevent the RMC21 or BMC22 from entering a possible abnormal state ( Step S38).

If the management system 3 analyzes based on the above data (mainly based on the status data, the manager's operation behavior and various feedback information), it is found that any BMC22 has lost the network connection (that is, the management system 3 cannot remotely access this directly BMC22), it will be determined that this BMC22 is in the third type of attention state (step S36). When any BMC22 is found to be in the third type of attention state, the management system 3 A remote rescue mechanism can be automatically implemented on the BMC22 in the third type of state of concern to remotely exclude the state where the BMC22 loses connection, and restore the network connection of the BMC22 to normal (step S39).

The following paragraph discusses the first type of attention state.

Since some RMC21/BMC22 do not have a Basic Input/Output System (BIOS), they need to use the Network Time Protocol (NTP) service provided by an external server or a hardware clock chip Provide real-time clock (Real-time Clock, RTC) service to set the time to achieve time synchronization with other devices.

As mentioned above, if a system event occurs before the time synchronization procedure of RMC21 or BMC22 is completed, although the system event will still be recorded in the event log of RMC21 and BMC22, the time field of the system event will not be recorded The correct time of the event, and only the words "Pre-init" will be recorded. If there is no correct event occurrence time, the administrator cannot use the event log as a reference indicator of the system event, which will result in a judgment error. In addition, if the RMC21 and the BMC22 need to perform a reset (Reset) operation, it may also cause an error or abnormality in the recording time of the event of the above system event.

Referring to FIG. 5, it is a first specific embodiment of the first type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 periodically accesses the database 31 (step S41) to obtain the status data and event logs of the RMC21 and BMC22 in the cabinet 2 from the database 31, and determine the status of the RMC21 and BMC22 Change (step S42).

In this embodiment, the management system 3 mainly determines whether there is any system event whose event occurrence time is unknown or wrong in the obtained event log (step S43). If all system events in the event log record the correct event occurrence time, the management system 3 does not actively implement any action.

If after analysis, the management system 3 finds that any RMC21 or BMC22 has a system event whose time is unknown or wrong, the management system 3 will regard the RMC21 or BMC22 as being in the first type of attention state (step S44), that is, It is believed that the RMC21 or BMC22 is in an abnormal state, but the network connection has not been lost.

In an embodiment, the management system 3 can mainly record when the event occurrence time of any system event in the event log is recorded as "Pre-init" or similar words (ie, the event occurrence time of the system event cannot be correctly stated) To determine whether the event time of the system event is unknown or wrong. In another embodiment, the management system 3 can mainly find from the event log that any RMC21 or BMC22 has a system event whose event occurrence time is unknown, and find from the status data that the RMC21 or BMC22 has not completed the time synchronization procedure or needs When performing the reset operation, it is determined that the event occurrence time of the system event is unknown or wrong.

After the management system 3 determines in step S44 that an RMC21 or BMC22 is in the first type of attention state, the management system 3 first obtains the time stamp of the access event log (step S45), and uses this time stamp as the system event The spare time identification information is stored in the database 31 (step S46). In an embodiment, the management system 3 uses the time of reading the event log as the timestamp for the current access to the database 31. In another embodiment, the management system 3 uses the time for remotely accessing the cabinet 2 and obtaining the event log from the RMC 21 and the BMC 22 as the time stamp, but it is not limited.

For example, the original content of the event log may be as shown in the following table:

If the management system 3 accesses the event log at 11:32:23 pm on December 22, 2018, and finds that the occurrence time of the event 2 is wrong, the management system 3 can take the initiative Describe the bit time identification information, and modify the content of the event log or generate a new event log. The new event log can be shown in the following table, for example:

When the administrator logs in to the management system 3 through the operation interface and queries the event log in the management system 3, the management system 3 can display the standby time identification information as event 2 as shown in the table above When the event occurred. In this way, even if a system event occurs before RMC21 or BMC22 time synchronization is not completed, the management system 3 can still set a recognizable standby time for the system event, so that the management system 3 and the administrator can Interpretation of the event, and thereby strengthen the effect of remote recovery.

After step S46, the management system 3 may further issue a control command (such as a first control command) to the RMC21 or BMC22 in the first type of attention state through the network to perform a time correction procedure on the RMC21 or BMC22 having an abnormal state with a time error (Step S47). In one embodiment, the time correction procedure is to control RMC21 or BMC22 to perform time correction through the NTP service. In another embodiment, the time correction procedure is to force the RMC21 or BMC22 to perform the reset operation, but it is not limited.

The following paragraphs continue to explain other possible types of concerns.

Due to the large number of cabinets 2 inside the data center 1, it is difficult to manually update one by one when the manager has a need for updating. Therefore, when the administrator wants to update the RMC21 and BMC22 in the cabinet 2 (such as firmware update), he can operate the management system 3 to pass the management Manage the relevant program code of the system 3 to send the update command and the latest version of the firmware, thereby simultaneously updating the RMC21 and BMC22 of multiple cabinets 2 in the data center 1 at the same time.

If problems such as network congestion or network signal instability cause network connection interruption during the update process, making some RMC21 and BMC22 unable to follow the normal update process to complete the update operation, it may cause the update operation to fail. However, some RMC21 and BMC22 will only cause the system to not operate normally after the update operation fails, but the network connection has not been lost (for example, it cannot be restored to the working mode after entering the update mode). End intervention to eliminate abnormal conditions.

Referring to FIG. 6, it is a second specific embodiment of the first type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 also regularly accesses the database 31 (step S51), so as to obtain the status data and event logs of the RMC21 and BMC22 in the cabinet 2 from the database 31, and at the same time obtain the implementation of the administrator through the operation interface Operation behavior, and judge the state change of RMC21 and BMC22 (step S52).

In this embodiment, the management system 3 can first analyze the status data and event logs of the RMC21 and BMC22 to determine whether any of the RMC21 and BMC22 update operations have expired (step S54) or an error has occurred, and determine the Whether the network connection of RMC21 or BMC22 where the update operation has timed out or an error has occurred is normal (step S55). If, after analysis, the management system 3 finds that the update operation of any RMC21 or BMC22 has timed out or an error has occurred but the network connection is still normal, the RMC21 or BMC22 may be regarded as being in the first type of attention state (step S56 ), that is, in an abnormal state, but the connection has not been lost.

More specifically, after the above step S52, the management system 3 may first determine whether the manager has performed an update operation on the RMC21 and/or BMC22 in the cabinet 2 according to the operation behavior (step S53). Moreover, after confirming that the manager has performed the update operation, the management system 3 continues to execute Steps S54 and S55 are used to determine whether the update operations of these RMC21 and BMC22 have timed out or an error has occurred, and whether the network connection is normal.

The RMC21 and BMC22 will automatically enter the update mode after accepting the update operation performed by the administrator. At this time, RMC21 and BMC22 will set the flag that has entered the update mode in the status data. When the peripheral device communicates with RMC21 and BMC22 and reads the update mode mark, it will automatically stop interacting with this RMC21 and BMC22. Therefore, as long as the update operation of RMC21 and BMC22 fails and it is impossible to leave the update mode, the RMC21 and BMC22 cannot operate normally. When the management system 3 finds that any RMC21 and BMC22 have accepted the update operation, the update operation has timed out or an error has occurred, but the network connection has not been lost, it can be determined that the RMC21 and BMC22 are in the first attention state.

After step S56, the management system 3 may further issue a control command (such as a second control command) to the RMC21 or BMC22 in the first type of attention state via the network to force the RMC21 or BMC22 that failed the update operation to leave the update mode (step S57).

As described above, in the case where the update operation referred to in this embodiment fails (ie, it is impossible to leave the update mode), the RMC21 and BMC22 can still receive and process related commands, but the peripheral device is reading the update mode flag (flag ) Will automatically stop the interaction with RMC21, BMC22. In this embodiment, the management system 3 has judged that the RMC21 and BMC22 have abnormal statuses, so it ignores the above-mentioned flag and forces the RMC21 and BMC22 to leave the update mode by issuing control commands.

After step S57, the management system 3 may further issue another control command (for example, a third control command) to the RMC21 or BMC22 that has left the update mode through the network to force the RMC21 or BMC22 to perform a reset operation, or implement the operation again. The update operation is described (step S58). In this way, the management system 3 can ensure that the RMC21 and BMC22 have resumed normal operation, and the firmware or software is at the latest version after the update is completed.

The following paragraphs continue to discuss the second type of state of interest.

The RMC21 and BMC22 in the present invention are an embedded system (Embbeded System), so even if the endpoint server 220 in the cabinet 2 is not turned on, the management system 3 can still realize remote booting and remote control by communicating with the RMC21 and BMC22 Remote management functions such as shutdown and device status check.

Generally speaking, when implementing remote management procedures, managers can use the Intelligent Platform Management Interface (IPMI) tool program on the management system 3 to send IPMI commands through the network to communicate with the RMC21 in the cabinet 2 , BMC22 communication. In the case of using the IPMI tool program, each command transmission needs to establish an IPMI session with the destination's RMC21 and BMC22, so as to communicate with the destination's RMC21 and BMC22. Specifically, after the IPMI session is established, the management system 3 can communicate with the RMC21, BMC22, and the underlying hardware devices of the cabinet 2 and the endpoint server 220 through the network to obtain the execution result of the command (such as obtaining firmware Version, sensing values of all sensors in the endpoint server 220, etc.).

However, the computing resources of the embedded system itself are quite limited. In addition to the basic resource consumption required for operation, communication with RMC21, communication with BMC22, and responses to various monitoring systems in the data center 1 will be further consumed. Computing resources for embedded systems.

In addition, when the administrator implements remote management procedures for each RMC21 and BMC22 through the management system 3, it also consumes the computing resources of RMC21 and BMC22. The most obvious is that the number of IPMI sessions of RMC21 and BMC22 is greatly increased, making RMC21, BMC22 appears to be under-responsive or request timeout. At this time, although the RMC21 and the BMC22 have not yet experienced an abnormal state, the management system 3 may need to intervene remotely to prevent the RMC21 and the BMC22 from having an abnormal state in the future and affecting the operation of the cabinet 2.

Referring to FIG. 7, it is a first specific embodiment of the second type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 also regularly accesses the database 31 (step S61) to The state data of RMC21 and BMC22 in the cabinet 2 is acquired from the database 31, and the state change of RMC21 and BMC22 is judged (step S62). In one embodiment, the management system 3 in step S62 mainly obtains the current total IPMI session of the RMC21 and each BMC22. In another embodiment, the management system 3 obtains the current system resource utilization rate of the RMC 21 and each BMC 22 at the same time in step S62.

After step S63, the management system 3 determines whether the total number of IPMI sessions of any RMC21 and BMC22 is higher than the first threshold (step S63), and when the total number of IPMI sessions of any RMC21 and BMC22 is higher than the first threshold, it is determined The RMC21 and the BMC22 are in the second state of attention (step S65), that is, the connection of the RMC21 or the BMC22 is normal, but it is determined that an abnormal state may soon occur.

It is worth mentioning that if the management system 3 obtains the system resource utilization rate of RMC21 and each BMC22 at the same time in step S62, the management system 3 can simultaneously determine whether there is any RMC21, BMC22 system resource utilization rate higher than the second Threshold value (step S64). In this situation, the management system 3 determines that the current total number of IPMI sessions is higher than the first threshold, and the RMC21 or BMC22 whose system resource utilization rate is higher than the second threshold is in the second state of concern.

In one embodiment, the system resource usage rate is the usage rate of the central processing unit or memory of RMC21 and BMC22. In another embodiment, the system resource usage rate is mainly used within RMC21 and BMC22 to provide system resources used by various services (such as HyperText Transfer Protocol (HTTP) service or IPMI service, etc.) Usage rate, but not limited.

When the management system 3 determines that an RMC21 or BMC22 is in the second type of attention state, the management system 3 can further issue control commands (such as a fourth control instruction) through the network to the RMC21 or BMC22 in the second type of attention state, so that all The RMC21 or BMC22 restarts the IPMI service (step S66). In this way, RMC21 and BMC22 can clear the currently accumulated IPMI session to avoid the occurrence of abnormal conditions.

In an embodiment, the fourth control command is a reset command. The management system 3 sends a reset command to the RMC21 or BMC22 in the second type of attention state through the network to force the RMC21 or BMC22 to perform the reset operation. In this way, the reset RMC21 and BMC22 can directly restart the IPMI service. However, the above is only one specific embodiment of the present invention, but not limited to the above.

Through the above technical solution, the management system 3 can find that the abnormal state of RMC21 or BMC22 may soon occur through analysis. Therefore, it can actively implement a service restart mechanism at the remote end to avoid the abnormal state of RMC21 or BMC22 from affecting the operation of the cabinet 2.

The following paragraphs continue to discuss the third type of state of interest.

As mentioned above, the management system 3 of the present invention mainly communicates with the RMC21 and BMC22 in the cabinet 2 in the data center 1 through the network, and the administrator also implements remote management procedures for these RMC21 and BMC22 through the network. Therefore, when the BMC 22 in the cabinet 2 loses the network connection, the management system 3 cannot communicate with the BMC 22, and the manager cannot manage the BMC 22. In this embodiment, the abnormal condition of the BMC22 losing the network connection may be caused by the incorrect IP address setting.

Generally speaking, the BMC22 in the cabinet 2 may be configured to use a dynamic IP address (that is, the network mode of the BMC22 is set to the dynamic IP mode) or a static IP address (that is, the network mode of the BMC22 is set to the static IP mode). If the network mode of BMC22 is dynamic IP mode, a dynamic host configuration protocol (DHCP) server (not shown) in the data center 1 can be used to proactively assign a set of dynamic IP addresses to BMC22 . If the network mode of the BMC 22 is the static IP mode, the administrator can set a set of static IP addresses from the BMC 22 through the operation interface of the management system 3.

To perform network configuration operations on BMC22 to set a set of available static IP addresses, the administrator needs to issue at least four commands to BMC22 via management system 3 (that is, four IPMI sessions need to be established), including: (1) settings The network mode of BMC22 is static IP mode; (2) set static IP address; (3) set subnet mask (netmask); (4) set gateway IP address.

As mentioned above, if the static IP address set by the administrator is wrong (for example, it overlaps with one of the multiple dynamic IP addresses assigned by the DHCP server), or the gateway IP address is set incorrectly, then In an environment where multiple sub-domains coexist, or an environment that requires a gateway to communicate, the BMC 22 will not be able to connect to the management system 3. For the management system 3, although the endpoint server 220 to which the BMC22 belongs still exists, but because the management system 3 loses the connection with the BMC22, the BMC22 (and the endpoint server to which it belongs 220) Management. At this time, the management system 3 may need to intervene remotely to restore the BMC 22 to the network connection.

Referring to FIG. 8, it is a first specific embodiment of the third type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 periodically accesses the database 31 (step S71), so as to obtain the status data of each BMC 22 in the cabinet 2 from the database 31, the operation behavior implemented by the manager through the management system 3, And the management system 3 obtains various pieces of feedback information based on the operation behavior, and judges the state change of the BMC 22 (step S72).

In one embodiment, the status data obtained by the management system 3 in step S72 includes at least the network mode (static IP mode or dynamic IP mode) of each BMC 22, the currently used static IP address, subnet mask and gate The IP address of the router is not limited. In addition, the feedback information obtained by the management system 3 in step S72 mainly includes the feedback generated by the management system 3, the cabinet 2 and each endpoint server 220 (and each BMC 22) based on this operation behavior when the operation behavior is implemented. Data such as parameters and execution data are not limited.

After step S72, the management system 3 first determines whether any BMC 22 in the cabinet 2 has lost the connection with the management system 3 based on the status data and feedback information (step S73), and determines the manager based on the operation behavior Whether the network setting operation has just been performed for any BMC 22 in the cabinet 2 (step S74). If after analysis, it is found that the administrator has just performed a network setting operation on a certain BMC22, and this BMC22 loses connection after the network setting operation, the management system 3 can regard this BMC22 as being in the third category Attention state (step S75), that is, the BMC22 has lost connection.

It is worth mentioning that in the aforementioned step S73, the management system 3 can mainly be set to a static IP mode in any BMC22 network mode, and the static IP address of this BMC22 and the multiple groups allocated by the DHCP server When one of the dynamic IP addresses is repeated, it is judged that the BMC22 has lost the network connection (the connection has been lost, or may be lost).

In addition, in the aforementioned step S73, the management system 3 can also determine that the BMC22 loses the network connection when the network mode of any BMC22 is set to the static IP mode, and the gateway IP address of the BMC22 is set incorrectly (The connection has been lost, or may be lost). However, the above are only some specific implementation examples of the present invention, but should not be limited to the above.

After step S75, the management system 3 can determine that a certain BMC22 is in the third type of attention state. Then, the management system 3 determines what RMC21 is mainly responsible for the BMC22 in the data center 1 (step S76), and controls the RMC21 Check the endpoint server 220 to which the BMC 22 belongs through the internal hardware circuit 24 of the cabinet 2 (step S77) to confirm whether this endpoint server 220 exists (step S78).

As shown in FIG. 2, the RMC 21 in a cabinet 2 can mainly be physically connected to the BMC 22 in all the endpoint servers 220 in the cabinet 2 through internal hardware lines 24. Therefore, even if the BMC 22 loses the network Road connection, RMC21 in the same cabinet 2 can still communicate with BMC22 through internal hardware circuit 24.

If it is determined in the above step S78 that the endpoint server 220 does not exist (for example, it has been pulled away from the cabinet 2 or has been damaged), the management system 3 correspondingly issues a warning signal (step S79). In an embodiment, the management system 3 can issue a warning signal (such as text, light or sound) through the operation interface to warn the manager. In another embodiment, the management system 3 can send a warning signal (such as a text message, email, or communication software) to the administrator through the network to achieve the warning function.

If it is determined in the above step S78 that the endpoint server 220 still exists, the management system 3 controls the RMC 21 to send a set of IPMI commands to the BMC 22 through the internal hardware line 24, so that the BMC 22 resumes the network connection ( Step S80). In one embodiment, the management system 3 may send an IPMI command to the BMC22 via RMC21 to reset the static IP address of the BMC22, or reset the IP address of the gateway of the BMC22, thereby Make the BMC22 restore the connection with the management system 3.

Through the above technical solution, the management system 3 can actively implement a rescue mechanism on the BMC 22 remotely after the BMC 22 loses the connection, so that the BMC 22 can restore the network connection.

In the method of the present invention, the management system 3 can automatically collect the required information and analyze the status of all RMC21 and BMC22, and at the same time, when any RMC21 and BMC22 are in one of a variety of states of attention, a corresponding mechanism is automatically implemented to eliminate the abnormal state. In this way, the technical solution of the present invention can greatly reduce the management cost, and also makes the monitoring of the data center 1 without human intervention, and is not affected by distance and time.

The above is only a preferred specific example of the present invention, and the patent scope of the present invention is not limited by this, so all equivalent changes in applying the content of the present invention are included in the scope of the present invention in the same way. Bright.

S31~S39‧‧‧‧Analysis and elimination steps

Claims (7)

A remote elimination method for abnormal state of a cabinet is applied to a data center having a cabinet and a cabinet server management system connected to the cabinet from a remote side, wherein the cabinet has a rack management controller (RMC) ) And a plurality of endpoint servers, each of which has a baseboard management controller (BMC), the remote exclusion method includes: a) the rack server management system regularly accesses a database To obtain the status data of the RMC and each BMC, and determine the status change of the RMC and each BMC; b) according to the status data and the status change, determine whether one of the RMC and each BMC is in the preset One of multiple attention states; and c) when it is judged that any RMC or BMC is in a second type of attention state among the multiple attention states, the rack server management system automatically checks for the second type of attention state The RMC or the BMC implements a remote service restart mechanism to prevent the RMC or the BMC from entering an abnormal state, where the second type of attention state refers to the normal connection between the RMC or the BMC and the rack server management system, but It is judged that the abnormal state may be imminent; wherein step a) is to obtain the total number of sessions of the RMC and the current Intelligent Platform Management Interface (IPMI) of each BMC, and step b) is When the total number of IPMI sessions of any RMC or BMC is higher than a first threshold, it is determined that the RMC or BMC is in the second type of attention state, and the step c) is a control command issued by the rack server management system To the RMC or the BMC in the second type of attention state, so that the RMC or the BMC restarts the IPMI service. The remote elimination method of the abnormal state of the cabinet as described in claim 1, further comprising the following steps: a01) the cabinet server management system is started; a02) after the step a01), the cabinet server management system regularly and actively accesses remotely The RMC and each BMC in the cabinet; a03) obtain the status data of the RMC and each BMC; a04) store the status data in the database; and a05) continue until the cabinet server management system is shut down Perform this step a02) to this step a04). The remote elimination method of the abnormal state of the cabinet as described in claim 1, further comprising the following steps: a11) the cabinet server management system is started; a12) after the step a11), the cabinet server management system provides an operation interface A13) When accepting an operation behavior of a manager through the operation interface, implement a remote management procedure for the RMC and each BMC according to the content of the operation behavior; a14) obtain feedback information corresponding to the remote management procedure A15) store the operation behavior and the feedback information to the database; and a16) continue to execute the steps a12) to a15) before the rack server management system is shut down. The remote elimination method for the abnormal state of the cabinet according to claim 1, wherein the step a) also obtains a system resource utilization rate of the RMC and each BMC, and the step b) is the IPMI of any RMC or BMC When the total number of sessions is higher than the first threshold and the system resource utilization rate is higher than a second threshold, it is determined that the RMC or the BMC is in the second type of attention state. The remote elimination method for the abnormal state of the cabinet according to claim 4, wherein the system resource utilization rate is the utilization rate of the central processing unit or memory of the RMC or the BMC. The method for remotely eliminating the abnormal state of the cabinet according to claim 4, wherein the system resource usage rate is the system resource usage rate of the RMC or the BMC mainly used to provide HTTP services or IPMI services. The remote elimination method for the abnormal state of the cabinet as described in claim 1, wherein the step c) is that the cabinet server management system issues a reset command to the RMC or the BMC in the second type of state of concern, Force the RMC or the BMC to perform a reset operation to restart the IPMI service.

Images