JP7179947B2 - Storage system and storage control method - Google Patents

Description

The present invention relates to a storage system and its control method.

In an information processing system that requires high reliability, it is common to use a plurality of servers to make the system redundant. However, in the case of such a redundant configuration, it is necessary to prepare a spare server to replace the failed server in order to restore redundancy after a server failure. Since the spare server does not perform processing during normal times, the utilization efficiency of the server decreases.

On the other hand, in recent years, there has been an increasing number of configurations in which servers are virtualized using virtualization technology, thereby improving utilization efficiency of physical servers and reducing the number of physical servers. An invention related to redundancy of virtual machines is disclosed in Patent Document 1, for example. Patent Literature 1 discloses a technique of arranging, on a plurality of physical servers, a plurality of active virtual machines and a standby virtual machine provided to make the active virtual machines redundant. there is According to such virtual machine placement technology, when one of the redundant virtual machines is lost due to a physical server failure, the lost virtual machine is copied to another physical server to reconstruct the redundant configuration. By doing so, it is possible to restore redundancy without preparing a spare physical server.

JP 2014-75027 A

In information processing systems that require high reliability, for example, in order to stably operate the base processing inside the system, such as the operation base for performing redundant operation, regardless of the amount of information processed by the system, Some require a certain amount of information processing resources such as CPU cores and memory. For example, in order for a storage system using virtualization technology to operate stably, it requires a certain amount of information processing resources regardless of the number of volumes.

In addition, in a configuration that operates multiple independent systems on a single server using virtual machines, containers, multi-processes, etc., when operating a system that requires a certain amount of information processing resources at a minimum, It is necessary to avoid being affected by other systems running on the same server. Therefore, it is common to reserve information processing resources necessary for the system and assign them to the system in a fixed manner.

However, when applying the arrangement technology of Patent Document 1 to such a system, the minimum information processing resources required by the system with reduced redundancy must be left in the physical server to be reconstructed. There is Therefore, in order to reliably recover redundancy, it is necessary to reserve information processing resources for redundancy recovery in the physical server in advance. The information processing resources reserved for redundancy recovery are not used unless the redundancy is lowered due to a failure or the like, so the utilization efficiency of the physical server is lowered and the system construction cost is increased.

Software Defined Storage (SDS), which is a storage system using virtualization technology, is required to be highly reliable, but it is also required to construct a low-cost information processing system using relatively inexpensive servers. ing.

An object of the present invention is to provide a low-cost storage system and storage control method while ensuring system availability.

In order to solve the above problems, a storage system according to one preferred aspect of the present invention is a storage system having a plurality of storage nodes forming a cluster. Each storage node includes a storage device for storing data and a It has a cluster control unit that performs control, and a storage control unit that uses a storage device to provide a host device with a storage area in units of volumes, and stores data in the storage device in response to an IO request from the host device. The storage controllers form a storage controller group with the storage controllers of other storage nodes in the cluster, and one storage controller in the storage controller group receives IO from the host device as a storage controller in active mode. The remaining storage controllers in the storage controller group are configured as standby mode storage controllers to switch to active mode if the active mode storage controller is lost, and to take over the processing of the active mode storage controllers. When one of the storage nodes configured to take over is removed from the storage system, the cluster controllers of the other remaining storage nodes are configured using the storage controller of the removed storage node. A storage that acquires multiple volumes that a storage controller group is in charge of, determines the save destination storage controller group for each of the acquired multiple volumes, and uses the storage controller of the removed storage node. A plurality of volumes that are in charge of the storage control unit group configured using the storage control unit of the storage node to be removed are transferred from the storage control unit that constitutes the control unit group to the storage control units of the multiple storage control unit groups of the backup destination. Disperse and evacuate.

The present invention eliminates the need for reserved information processing resources for guaranteeing the possibility of recovering redundancy, thereby improving the utilization efficiency of physical servers.

1 is a block diagram showing the overall configuration of an information processing system according to Example 1; FIG. 3 is a block diagram showing the detailed configuration of a storage node; FIG. 1 is a diagram showing the logical configuration of a storage system according to Example 1; FIG. 4 is a diagram for explaining data management according to the first embodiment; FIG. FIG. 10 is a diagram showing an example of a storage node management table; FIG. FIG. 10 is a diagram showing an example of a storage control unit management table; FIG. FIG. 10 is a diagram showing an example of a volume management table; FIG. FIG. 10 illustrates an example of a logical chunk management table; FIG. FIG. 4 is a diagram showing an example of a physical chunk management table; FIG. It is a figure explaining the subject of this invention. It is a figure which shows an example of the program and management information which are stored in memory. FIG. 4 is a diagram showing an example of processing of a failure recovery program (1); FIG. 10 is a diagram showing an example of processing of a failure recovery program (2); FIG. 10 is a diagram showing an example of processing of the volume saving program (1); FIG. FIG. 10 is a diagram showing an example of processing of a volume save destination determination program (1); FIG. FIG. 10 is a diagram showing an example of processing of a storage control unit pair deletion program (1); FIG. 10 is a diagram showing an example of processing of a storage node removal program (1); FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the following description and drawings are examples for explaining the present invention, and are appropriately omitted and simplified for clarity of explanation, and limit the technical scope of the present invention. is not.

In the following description, expressions such as "table", "table", "list", and "queue" will be used to describe various types of information, but various types of information may be expressed in data structures other than these. Therefore, it is sometimes simply called "information" to indicate that it does not depend on the data structure. Expressions such as "identification information", "identifier", "name", "name", "ID", and "number" are used when explaining the contents of various types of information, but these can be replaced with each other. be.

In the following explanation, the subject may be "program", but the program is a processor (for example, CPU (Central Processing Unit) or GPU (Graphics Processing Unit)). Since it is performed while appropriately using storage resources (for example, memory) and interface devices (for example, communication device), the processor may be the subject of the explanation. Similarly, the subject of processing performed by executing a program may be, for example, a controller, device, system, computer, node, storage device, server, client, or host having a processor. Also, part or all of the program may be processed using a hardware circuit.

Various programs may be installed in each computer by a program distribution server or storage media. Also, in the following description, two or more programs may be implemented as one program, and conversely, one program may be implemented as two or more programs.

The first embodiment will be described in detail below with reference to FIGS. 1 to 16. FIG.

FIG. 1 is a diagram showing the physical configuration of the information processing system according to the first embodiment. This information processing system comprises one or more host devices 100, one or more management terminals 110, and a multi-node configuration storage system 200 composed of two or more storage nodes 200. FIG. Each host device 100, management terminal 110, and each storage node 210 are connected via a network 300 composed of, for example, Fiber Channel, Ethernet (registered trademark), wireless LAN (Local Area Network), or InfiniBand. be done. Although not shown, the network 300 may include various relay devices such as network switches and gateways. A dedicated network between the storage nodes 210 may be provided separately, or each host device 100, management terminal 110, and storage node 210 may be connected to a network other than these.

The host device 100 is a server device for performing various business processes by executing installed application programs. The host device 100 transmits a data read request or data write request to the storage node 210 via the network 300 in response to a request from the application program being executed. Note that the host device 100 may be a virtual server device such as a virtual machine or a container.

The management terminal 110 is a client device for the administrator of the storage system to perform various setting operations and monitor the status of the storage system 200 . The management terminal 110 may be a mobile terminal such as a smart phone or a tablet terminal, and some of the host devices 100 may also serve as the management terminal.

The storage system 200 is a server device that provides storage areas for reading and writing data to the host device 100 . The storage node 210 that constitutes the storage system 200 may be a virtual server device such as a virtual machine or a container. It may be arranged in the same physical server device.

FIG. 2 is a diagram showing the detailed configuration of the storage node 210. As shown in FIG. The storage node 210 comprises a CPU 211 , a memory 212 , a storage device 213 and a communication device 214 , which are connected via an internal network 215 as a server device. However, FIG. 2 is an example of a storage node, and the present invention is not limited to the configuration shown in the figure, and all or any of the CPU 211, memory 212, storage device 213, and communication device 214 may be plural. .

The CPU 211 is a control device that controls the operation of the entire storage node 210, and executes various programs stored in the memory 212 to perform various processes. The memory 212 stores, for example, control information used in the storage node 210, programs executed by the CPU 211, data accessed by the host device, and the like. The memory 212 is generally composed of DRAM (Dynamic RAM (Random Access Memory)), but may be composed of storage media other than DRAM such as MRAM (Magnetoresistive RAM), ReRAM (Resistive RAM), PCM (Phase Change Memory), and NAND. may be

The storage device 213 is a device having a physical storage area, and is composed of a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), an SCM (Storage Class Memory), or an optical disk. be done. Although SAS (Serial Attached SCSI) and NVMe (Non-Volatile Memory Express) are described as interfaces for accessing the storage device 213, for example, SATA (Serial ATA), USB (Universal Serial Bus), etc. are also available. It may be an interface other than

Generally, in a multi-node configuration storage system, a copy of data is stored in another storage node 210 to protect data in case of a node failure. A high reliability technique such as RAID (Redundant Array of Independent Disks) may be used by bundling a plurality of storage devices 213 within a node.

The communication device 214 is connected to the host device 100, other storage nodes 210, the management terminal 110 for managing the storage system 200, etc. via the network 300, and communicates with the host device 100, the management terminal 110, and the other storage nodes 210. to mediate communications between In FIG. 2, the communication device 214 is shared by the communication for the host device 100, the communication for the management terminal 110, and the communication for the other storage node 210, but different communication devices are provided for each communication. good too.

FIG. 3 is a diagram showing the logical configuration of the storage system according to the first embodiment. The cluster control unit 216 is software that controls the entire storage system composed of multiple storage nodes. The cluster controller 216 has two operational roles: master and worker. The worker role cluster control unit 216b performs various controls and status monitoring within the storage node according to instructions from the master role cluster control unit 216a. Processing, processing of various setting operations via the management terminal 110, and notification when a failure or the like occurs. Note that the master includes the functions of the worker.

There is always one cluster controller operating as the master role in the cluster, and the other cluster controllers operate as worker roles. The cluster control unit of the master role and the cluster control unit of the worker role monitor each other for life and death through inter-storage node communication or the like. When the cluster control unit is lost due to a storage node failure or the like, the cluster control unit of the master role determines that a storage node failure has occurred and performs failure recovery processing. Details of the failure recovery process will be described later with reference to the drawings.

If the cluster controller of the master role is lost, any one of the cluster controllers of the worker roles in the cluster will switch to the master role. Regarding the selection of the cluster control unit that switches from the cluster control units of a plurality of worker roles to the master role, a technique and function generally called "leader selection" is used, so the explanation is omitted.

The storage control unit 219 is implemented by software that controls various types of volumes provided to the host device as storage areas. The storage control unit 219 has a function of providing storage areas to the host device in units of volumes using a storage device, and storing data in the storage device in response to IO (Input/Output) requests from the host device. It also has a function (migration function) to move a volume that was in charge of a certain storage control unit to another storage control unit.

The storage controller 219 has two modes of operation: active and standby. An active-mode storage control unit 219a placed in a certain storage node forms a pair (called a storage control unit pair 217) with a standby-mode storage control unit 219b placed in a different storage node in the cluster and operates. . In addition to a storage control unit pair, when one active mode storage control unit is associated with a plurality of standby mode storage control units, a storage control unit group is formed. Normally, the active mode storage controller 219a processes IO requests from the host device. The standby mode storage control unit 219b stands by in preparation for loss of the active mode storage control unit 219a due to a storage node failure or the like. If the active mode storage control unit 219a is lost, the standby mode storage control unit 219b switches to the active mode and takes over processing such as IO requests. Both of the two storage controllers that make up the storage controller pair may be operating in active mode, and two or more storage controllers that make up the storage controller group may be operating in active mode. good. In this case, additional processing such as exclusive control is required between the storage controllers in active mode. Also, if the active mode storage controller is lost due to a storage node failure, etc., the surviving active mode storage controller will be You can take over the process.

As shown in FIG. 3, one storage node may have two or more storage controllers. In addition, by arranging the same number of active mode and standby mode storage control units in one storage node, it is possible to equalize the utilization rate of information processing resources such as CPU and memory among storage nodes. .

The data redundancy unit 218 is realized by software for preventing data loss due to storage node failure by making data redundant among the plurality of storage nodes 210 and storing it in the storage device. As a data redundancy method, for example, a method of storing duplicates of data in different storage nodes 210, a method of distributing and storing parity in a plurality of storage nodes 210, and the like are conceivable. Although not shown in the figure, in addition to data redundancy between storage nodes, data redundancy such as RAID may be performed within a node in preparation for a failure of a storage device within the storage node.

As described above, the first embodiment relates to a storage system using a plurality of physical servers as storage nodes as an application example of virtualization technology for virtualizing servers. In such a storage system, the active (active system) and standby (backup system) storage control units that provide volumes as storage areas to the host device are arranged between different storage nodes for redundancy. Furthermore, in order to improve the processing performance of the entire storage system, one storage node has a plurality of active and standby storage controller pairs.

FIG. 4 is a diagram for explaining an overview of data management according to the first embodiment. FIG. 4 shows the case of processing a write request from the host device.

A data redundancy unit 218 replicates data between multiple storage nodes using chunks. A physical chunk 222 is a physical storage area created by dividing a storage device in a storage node into one or more small areas (for example, 42 MB) of a predetermined capacity. A logical chunk 221 is a logical chunk associated with one or more physical chunks. A logical chunk 221 is associated with a block 223 of a volume 220, which will be described later, and stores data written by the host device. Two or more physical chunks 222 created in different storage nodes are associated with one logical chunk 221, and data written to the logical chunk 221 is stored in all associated physical chunks 222. data redundancy. In FIG. 4, data is stored in physical chunks 222 of each storage node 210 in which active mode and standby mode storage controllers 219 are arranged. In this way, since data is saved (data locality is ensured) in the storage node 210 in which the active mode and standby mode storage controllers 219 are arranged, the data is transferred to the storage device that provides the volume to the host device. When there is a read request, there is no need to read data from other storage nodes, ensuring high responsiveness.

If data locality is not guaranteed, data may be stored in physical chunks of any two storage nodes. For example, when the free space of the storage device of the storage node in which the storage control unit 219 is arranged is insufficient, a process of saving to a physical chunk of a storage node with sufficient free space of the storage device may be performed.

The volume 220 is a virtual storage area provided to the host device 100 by the storage control unit 219, and the host device 100 issues a data write request to the volume. The volume 220 is created by the administrator of the storage system 200 issuing a volume creation instruction to the storage system 200 via the management terminal 110 . The storage control unit 219 to be the creation destination of the volume 220 may be specified by the administrator at the time of volume creation, and the cluster control unit 216a of the master role that receives the volume creation instruction determines the free storage capacity of each storage node and each storage. It may be selected based on the CPU usage rate of the control unit.

A volume itself does not have a physical storage area, and logical chunks 221 are allocated in response to write requests from the host system 100 and data is logically written to the logical chunks 221 . The volume 220 is managed by dividing the storage area into one or more blocks 223 of a predetermined capacity from the beginning. This block is associated, for example, with a logical chunk on a one-to-one basis. Immediately after the volume is created, no logical chunk is associated with any block, and when the host system 100 writes data to the volume 220, the block corresponding to the area where the data is written If the logical chunk 221 is not associated with 223, processing for creating the logical chunk 221 and associating the block 223 with the logical chunk 221 is performed.

The processing of IO requests from the host device 100 is handled by the active mode storage control unit 219a. When a new logical chunk is created and the blocks and logical chunks are associated with each other, the information indicating the correspondence is transferred to the storage control unit 219b in standby mode. One storage control pair 217 is composed of the storage control unit 219a in active mode and the storage control unit 219b in standby mode. As shown in FIG. 4, the standby mode storage controller 219 of storage node 0 forms a storage control pair 217 with the storage controllers of storage nodes other than storage node 1, and the standby mode storage controller of storage node 1 Unit 219 forms a storage control pair 217 with the storage controllers of storage nodes other than storage node 0 .

Data written to the logical chunks 221 is written to the physical chunks 222 by the data redundancy unit 218 according to the correspondence relationship between the logical chunks 221 and the physical chunks 222 . The example of FIG. 4 shows the case of making data redundant by duplicating (duplicating) physical chunks, and the data written from the host system 100 is stored in physical chunks of "storage node 0" and "storage node 1". written in chunks. When the physical chunks are tripled, or when redundancy is achieved by using RAID or erasure coding between storage nodes, the data redundancy part duplicates physical chunks and generates parity according to the redundancy method. conduct. In FIG. 4, blocks and logical chunks have the same capacity and are associated one-to-one, and henceforth the explanation will proceed on the assumption that blocks and logical chunks are also attached one-to-one. However, for example, two or more blocks made up of one or more volumes may be associated with one logical chunk.

Although not shown in FIG. 4, when “storage node 0” fails, the storage control unit and data redundancy unit of “storage node 1” take over the processing. information and information about logical chunks, and when information is updated in one of the storage nodes, the updated contents are synchronously transferred to the other storage node and the information is updated. The details of each piece of information will be described with reference to a diagram in which management information is described.

Next, management information (management table) for controlling the storage system according to the first embodiment will be described. Various types of management information may be referenced and set by the administrator of the storage system 200 via the management terminal 110. FIG.

FIG. 5 is an example of the storage node management table 256. As shown in FIG. The storage node management table may be expressed in a data structure other than tabular format. Therefore, it is sometimes simply called "information" to indicate that it does not depend on the data structure. The storage node management table is information for managing the operating status of the storage node and various information processing resources of the storage node. The storage node management table is stored in the memory of the storage node in which the master role cluster controller operates. The storage node management table 256 manages records including storage node ID 2561, role 2562, operating status 2563, number of CPU cores 2564, amount of memory 2565, communication band utilization rate 2566, total storage capacity 2567, total storage usage 2568. do.

A storage node ID 2561 is an ID that uniquely identifies a storage node, and is a unique ID in the entire storage system. The role 2562 is information indicating the operation role (master, worker) of the cluster control unit operating in the storage node. The operating status 2563 is information indicating whether the storage node is operating normally. The number of CPU cores 2564 and the amount of memory 2565 are information indicating the number of CPU cores and the memory capacity of the storage node, respectively. The communication band utilization rate 2566 is information indicating the band utilization rate of the communication device installed in the storage node. The storage device total capacity 2567 is the total capacity of the storage devices installed in the storage node. The storage device total usage 2568 is the total amount of capacity actually used among the capacities of the storage devices installed in the storage node. A storage node with a storage node ID of 1 indicates that it is operating as a master role cluster controller.

The communication band utilization rate 2566 and the total storage device usage 2568 are information periodically acquired by the master role cluster control unit from the worker role cluster control units operating in each storage control unit. Although omitted, each storage node manages storage node information collected by the cluster controller of the master role. Also, if the cluster controller of the relevant storage node is not operating due to a storage node failure, etc., the cluster controller of the master role will determine that a failure has occurred in the relevant storage node, and the operation of the storage node management table will be performed. Change state to failure. FIG. 4 shows that a storage node with a storage node ID of 0 has failed. Also, when the communication band utilization rate and the total storage device usage could not be acquired due to a storage node failure, "NA" is indicated.

FIG. 6 is an example of the storage controller management table 257. As shown in FIG. The storage control unit management table 257 may be expressed in a data structure other than the tabular format. Therefore, it is sometimes simply called "information" to indicate that it does not depend on the data structure. The storage controller management table 257 is information for managing the pair relationship of the storage controllers, the relationship between the storage controllers and the storage nodes, and the operating status of the storage controllers. The storage controller management table 257 is stored in the memory of the storage node in which the master role cluster controller operates. The storage control unit management table 257 is a record containing a storage control unit ID 2571, a storage control unit pair ID 2572, a storage node ID 2573, an operation mode 2574, an allocated CPU core number 2575, an allocated memory amount 2576, a CPU usage rate 2577, and a memory usage amount 2578. to manage.

The storage controller ID 2571 is an ID that uniquely identifies the storage controller, and is a unique ID in the entire storage system. The storage controller pair ID 2572 is an ID for uniquely identifying the storage controller pair to which the storage controller belongs. The storage node ID 2573 is an ID for uniquely identifying the ID of the storage node in which the storage control unit is arranged. The operation mode 2574 is information indicating whether the operation mode of the storage control unit is active or standby.

In Figure 6, a fixed amount of CPU cores and memory is assigned to each storage control unit, and the number of allocated CPU cores 2575 and the amount of allocated memory 2576 are allocated from the storage node to the storage control unit. This information indicates the number of CPU cores and the amount of memory used. The CPU usage rate 2577 is information indicating the average usage rate of each CPU core assigned to the storage control unit. The memory usage 2578 is information indicating the amount of memory actually used out of the memory allocated to the storage control unit.

In FIG. 6, the storage controller with the storage controller ID 2571 of "0" is operating in the active mode in the storage node with the storage node ID of "0", and is in the standby mode in the storage node with the storage node ID of "1". A storage controller pair with a storage controller ID of "1" and a storage controller pair with a storage controller pair ID of "0".

The CPU usage rate 2577 and the memory usage amount 2578 are information periodically obtained by the master role cluster control unit 216a from each storage control unit via the worker role cluster control unit 216b operating in each storage node. In FIG. 6, "NA" indicates that the cluster controller of the storage node is not operating due to a storage node failure or the like.

FIG. 7 is an example of the volume management table 261. As shown in FIG. The volume management table 261 may be expressed in a data structure other than the tabular format. Therefore, it is sometimes simply called "information" to indicate that it does not depend on the data structure. The volume management table 261 manages the relationship between volumes and storage controller pairs, the relationship between blocks in volumes and logical chunks, and the IO amount per unit time for each volume. A volume management table 261 is stored in the memory of each storage node. The volume management table manages records including volume ID 2611, capacity 2612, used capacity 2613, storage controller ID 2614, block ID 2615, logical chunk ID 2616, and IO amount 2617. The volume management table 261 can be referenced by the storage controller.

A volume ID 2611 is an ID for uniquely identifying a volume. A volume is a resource provided to a host device and has a unique ID in the entire storage system. The capacity 2612 is information indicating the capacity of the volume. The used capacity 2613 is information indicating the capacity of the physical storage area actually used by the volume. The used capacity 2613 can also be calculated by multiplying the number of blocks to which logical chunks are allocated by the block size. The storage control unit pair ID 2614 is an ID for uniquely identifying the storage control unit pair in charge of processing IO requests from the host system to the volume. Block ID 2615 is block position information from the head of the volume.

The logical chunk ID 2616 is an ID for uniquely identifying the logical chunk associated with the block of the volume. By combining the storage control unit pair ID 2614 and the logical chunk ID 2616, it is possible to uniquely identify the logical chunk associated with the block of the volume. The IO amount 2617 is information representing the IO amount per unit time for each volume.

In FIG. 7, the volume with the volume ID 2611 of "0" is handled by the storage control unit pair with the storage control unit pair ID 2614 of "0", and the block with the block ID 2615 of "0" is in charge of processing the IO request from the host device. A logical chunk with a logical chunk ID 2616 of "0" is associated.

In this way, the volume management table 261 associates and manages each volume with a storage controller pair. There is a one-to-one correspondence between one volume and a pair of storage controllers that are in charge of IO requests from the host system for that volume. The storage controllers that make up the storage controller pair are specified in the storage controller management table in FIG. It has a corresponding configuration as a storage control unit that processes IO.

FIG. 7 shows a case where blocks and logical chunks are associated one-to-one. ID column added.

FIG. 8 is an example of the logical chunk management table 271. As shown in FIG. The logical chunk management table 271 may be expressed in a data structure other than tabular format. Therefore, it is sometimes simply called "information" to indicate that it does not depend on the data structure. The logical chunk management table 271 is information for managing the relationship between logical chunks and physical chunks and the storage controller pairs associated with the logical chunks. A logical chunk management table 271 is stored in the memory of each storage node. The logical chunk management table 271 includes a logical chunk ID 2711, a storage controller pair ID 2712, a storage node ID (master) 2713, a physical chunk ID (master) 2714, a storage node ID (mirror) 2715, and a physical chunk ID (mirror) 2716. Manage records. The logical chunk management table can be referenced by the data redundancy unit 218. FIG.

A logical chunk ID 2711 is an ID for uniquely identifying a logical chunk. A logical chunk is a resource associated with a storage controller pair and is a unique ID within the storage controller pair. The storage controller pair ID 2712 is an ID for uniquely identifying the storage controller pair associated with the relevant logical chunk. A storage node ID (master) 2713 is an ID for uniquely identifying a storage node. A physical chunk ID (master) 2714 is an ID for uniquely identifying a physical chunk. By combining the storage node ID and the physical chunk ID, it is possible to uniquely identify the physical chunk associated with the relevant logical chunk. The storage node ID (mirror) 2715 and physical chunk ID (mirror) 2716 are information for identifying redundant (mirror) physical chunks in case of failure. In FIG. 8, the logical chunk with the logical chunk ID 2711 of "0" is associated with the storage control unit pair "0", and the physical chunk ID 2714 of the storage node with the storage node ID (master) 2713 of "0" is " 0” and the physical chunk of the storage node whose storage node ID (mirror) 2715 is “1” and whose physical chunk ID 2716 is “1”.

The example of FIG. 8 shows an example of a logical chunk table in the case of making data redundant by duplicating (duplicating) physical chunks. In other words, one set of storage controller pairs corresponds to one logical chunk, and physical chunks correspond to each of a plurality of master and mirror storage nodes. The structure of the logical chunk management table may be changed according to the data redundancy method, such as tripled physical chunks or application of RAID or erasure coding between storage nodes.

FIG. 9 is an example of the physical chunk management table 272. As shown in FIG. The physical chunk management table 272 may be expressed in a data structure other than tabular format. Therefore, it is sometimes simply called "information" to indicate that it does not depend on the data structure. The physical chunk management table 272 manages addresses of storage devices corresponding to physical chunks. A physical chunk management table is stored in the memory of each storage node. The physical chunk management table manages records including physical chunk ID 2721 , storage device ID 2722 , and storage device offset 2723 . The physical chunk management table 272 can be referenced by the data redundancy unit 218. FIG.

A physical chunk ID 2721 is an ID for uniquely identifying a physical chunk. A physical chunk is a resource within a storage node and has a unique ID within the storage node. The storage device ID 2722 is an ID for identifying each storage device within the storage node. The storage device offset 2723 is the address of the storage device to which the head of the physical chunk identified by the physical chunk ID corresponds. In FIG. 9, the physical chunk whose physical chunk ID 2721 is "0" is stored in the storage device whose storage device ID 2722 is "0", and the start address of the physical chunk is "0x0000" indicated by the offset 2723 in the storage device. is.

FIG. 10 is a conceptual diagram explaining the problem of the present invention. FIG. 10 shows a case where a failure occurs in "storage node 0" in the configuration of FIG.

The master role cluster control unit 216a monitors the life and death of each storage node through periodic communication with the worker role cluster control unit 216b operating in each storage node.

When a storage node failure is detected, first, the management terminal 110 notifies the administrator of the storage system 200 of the failure. Subsequently, in order to take over the processing of IO requests from the host device to the volume that the storage control unit was in charge of, the standby unit that formed a pair with the active mode storage control unit 219a operating in the storage node The mode storage control unit 219b is instructed to switch to the active mode. In FIG. 10, the operation mode of the "storage control unit 1" 219b, which forms a pair with the "storage control unit 0" 219a, is switched from standby to active.

Next, in order to recover the redundancy of the storage controller pair 217, the standby mode storage controller 219c is reconfigured in the normal storage node. The master role cluster control unit 216a selects a storage node that has free information processing resources such as CPU cores and memory required to operate the new storage control unit, and sends a request to the cluster control unit of the storage node , instructs the rebuilding of the storage controller. In the example of FIG. 10, the "storage controller 22" 219c is reconstructed in the "storage node 2" 210 as a replacement for the "storage controller 0" that has lost its function, and one storage node has three storage controllers. It is placed. In addition, although not shown, a storage control unit that replaces the "storage control unit 21" is also reconstructed.

When selecting a destination to reconstruct the storage control unit, if there is no storage node with free information processing resources, the storage control unit cannot be reconstructed and the redundancy cannot be recovered. By reserving information processing resources for rebuilding the storage control unit in each storage node in advance, it is possible to guarantee that the redundancy of the storage control unit can be recovered reliably in the event of a storage node failure. Processing resources cannot be used, storage node usage efficiency decreases, and system construction costs rise.

11 to 16, a technique for achieving high storage node utilization efficiency and low system construction costs will be described.

FIG. 11 shows control information (management tables) 256, 257, 261, 271 and 272 and programs 250-255, 258, 260 and 270 stored in the memory 212 of the storage node. The actual memory may store programs and management information other than these, but FIG. 11 shows what is necessary for explaining the present invention. For example, programs and cache management tables for processing IO requests from the host are omitted. The program is also stored in the storage device of each storage node, and is loaded into the memory when the storage system is started or when the program is executed. The management tables 256, 257, 261, 271, and 272 may be stored in a storage device in preparation for a power failure or the like, or the memory may be used as a cache for the management tables stored in the storage device.

The failure recovery program 250, the volume evacuation program 251, the volume evacuation destination determination program 252, the storage control unit pair creation program 253, the storage control unit pair deletion program 254, the storage node reduction program 255, the storage control unit pair reconstruction program 258, It is part of the program that constitutes the cluster control unit 216 . The failure recovery program 250, volume evacuation program 251, volume evacuation destination determination program 252, storage node reduction program 255, and storage control unit pair reconstruction program 258 are programs that can be executed when the cluster control unit 216 operates in the master role. is. The storage controller pair creation program 253 and storage controller pair deletion program 254 are programs that can be executed when the cluster controller operates in the worker role.

The storage node management table 256 and storage controller management table 257 are management information stored in the memory of the cluster controller 216 in the master role. The contents of the storage node management table 256 and the storage controller management table 257 are shown in FIGS. 5 and 6, respectively. Copies of these management tables are held in the cluster controllers of one or more worker roles. This is because when a failure occurs in the storage node in which the master role cluster control unit 216 is arranged, the worker role cluster control unit is promoted to the master role and takes over the processing. In the explanation that will be given later, when these management tables are updated, it is assumed that the management tables replicated by communication between cluster control units are also updated at the same time.

The volume migration program 260 is part of the programs that make up the storage control unit 219 . The volume management table 261 is management information stored in the memory of the storage control unit 219, and its contents are shown in FIG. The volume management table 261 is replicated between the storage controllers forming the storage controller pair. This is because the standby mode storage control unit takes over the processing when the active mode storage control unit becomes inoperable due to a storage node failure or the like. In the explanation that will be given later, when updating the volume management table 261, it is assumed that the management table duplicated by communication between storage control units is also updated at the same time.

The physical chunk rearrangement program 270 is part of the programs that make up the data redundancy unit 218. FIG. A logical chunk management table 271 and a physical chunk management table 272 are management information stored in the memory of the data redundancy unit. The contents of the logical chunk management table 271 and the physical chunk management table 272 are shown in FIGS. 8 and 9, respectively. The Logical Chunk Management Table 271 holds only records relating to the storage controller pair in which the data redundancy part of each storage node is located in that storage node. In the example of FIG. 8, since physical chunks are replicated to the master and mirror, one record is replicated between the data redundancy sections of two storage nodes. Since the physical chunk management table 272 is information for managing the addresses of the storage devices within the storage nodes, there is no need to duplicate it.

12 and 13 are diagrams showing an example of processing of the failure recovery program 250, respectively. A process is performed to restore the state in which the redundancy of all storage control unit groups is maintained from the state in which there are storage control unit groups whose redundancy has been lowered due to a storage node failure or the like. Instead of storage node failure, storage node reduction can also be applied by reading failure as reduction. For ease of explanation, the storage controller group will be described below as a storage controller pair consisting of one active mode storage controller and one standby mode storage controller. However, not limited to pairs, even in the case of a storage control unit group composed of three or more storage control units, basically the same processing is performed. If a storage controller pair or group contains multiple active mode storage controllers, the active mode storage controllers surviving the recovery process are essentially the same as the standby mode storage controllers. can handle.

For example, this program 250 performs processing for recovering the redundancy of the volume that has been reduced due to the redundancy reduction of the storage controller pair due to the failure of the storage node. Also, this program is stored in the memory of the cluster control unit 216 and executed by the CPU of the storage node in which the cluster control unit is arranged. This program 250 is started when the cluster controller 216 operating in the master role detects a failure in a storage node, and is executed by the cluster controller. If a failure occurs in the storage node where the master role cluster controller is located, the CPU of the storage node where the master role cluster controller is located is newly selected from the worker role cluster controller. This program will run.

In FIG. 12, when a storage node failure occurs, all storage controller pairs in which one of the storage controller pairs has been lost are deleted, and the released information processing resources are used to create a new storage controller pair. is creating According to the example of FIG. 10, since there are two storage controller pairs 0 and 10 in the failed storage node "0", these two storage controller pairs are deleted and one Create a control pair.

In FIG. 13, of the storage control unit pairs in which one of the pairs has been lost, only the storage control unit pair in which the active mode storage control unit has been lost is deleted, and the released information processing resources are used to perform standby mode storage control. Rebuild a new pair partner for a lost storage controller pair. According to the example of FIG. 10, the storage controller pair 0 to which the active storage controller 219a in the failed storage node 0 belongs is deleted, and the redundancy of one storage controller pair 10 is restored. In other words, by deleting the storage control unit pair 0 (and the storage control unit 1), resources for one storage control unit are released to the storage node 1. FIG. Using this, the storage controller 21 is reconstructed in the storage node 1 and the redundancy of the storage controller pair 10 is recovered.

First, an example of the processing in FIG. 12 will be described. The failure recovery program (1) in FIG. 12 identifies a storage controller pair in which one of the storage controller pairs has been lost due to a storage node failure (S100). This processing is realized by referring to the storage control unit management table 257, searching by the failed storage node ID 2573, and specifying the storage control pair ID 2572. FIG. At this time, processing for inhibiting creation of a new volume in the identified storage control unit pair may be added. This process can be implemented by adding a volume creation inhibition flag column to the storage control unit management table 257 in FIG. 6 and turning this flag ON for the specified storage control unit.

Subsequently, the failure recovery program (1) 250 selects a storage control pair to be processed from the identified storage control pairs (S101). In the case of Embodiment 1 (FIG. 10), one storage node has one active mode storage controller and one standby mode storage controller. One of the remaining storage controllers is operating in active mode and the other is operating in standby mode.

The failure recovery program (1) 250 determines whether or not the operation mode of the remaining storage control unit is standby (S102), and if not standby mode, skips step S103 and proceeds to step S104. If it is in the standby mode, in step S103, the storage controller is instructed to switch to the active mode. Processing related to switching of operation modes is realized as follows. The failure recovery program (1) 250 refers to the storage control unit management table 257 and retrieves the storage control unit IDs that make up the storage control pair selected in step S101 by searching with the storage control pair ID selected in step S101. Narrow down. Further, by referring to the storage node management table 256 and searching with the narrowed down storage controller IDs, the remaining storage controller IDs can be identified by identifying the storage controller IDs corresponding to the storage nodes whose operating status is normal. . By obtaining the operation mode 2574 of the storage control unit management table 257 corresponding to the identified remaining storage control unit ID 2571, the operation mode can be obtained.

When the failure recovery program (1) receives the active mode switching completion from the storage controller, it updates the operation mode of the storage controller in the storage controller management table 257 to "active" (S103). It should be noted that the storage control unit in the standby mode may monitor the active mode of the pair partner and autonomously switch the operation to the active mode when detecting that a failure has occurred. In this case, step S103 is a process of waiting for the standby mode storage control unit to switch to active, and updating the storage control unit management table 257 after the switching is completed. In any case, this step allows the storage controller in standby mode to take over the processing of IO requests to volumes that was being executed by the active mode storage controller that was running on the failed storage node. be

Next, the failure recovery program (1) determines whether or not the processing of steps S101 to S103 has been completed for all storage controller pairs identified in step S100 (S104). If completed, proceed to step S105; if not completed, return to step S101.

Although not shown in the figure, before proceeding to step S105, it is checked whether or not the remaining free storage capacity of each storage node is sufficient to recover from the storage node failure, and if there is insufficient free storage capacity Failure of failure recovery processing due to In this case, if it is determined that the free storage capacity is insufficient, the administrator of the storage system 200 is notified of the lack of free storage capacity via the management terminal 110, and the processing of the program is terminated. After that, the administrator of the storage system 200 takes measures such as adding storage devices and adding storage nodes, and then instructs the master role cluster control unit 216a to re-execute the failure recovery program 250 via the management terminal 110. do.

If the configuration is such that physical chunks are stored in storage nodes other than the storage node where the storage controller is located, or if the physical chunks are tripled, RAID or erasure coding is applied between the storage nodes to distribute the physical chunks. In the case of redundancy, the physical chunks lost due to the storage node failure may reduce the redundancy of the logical chunks associated with the lost storage controller pair and unrelated storage controller pairs. Therefore, before proceeding to step S105, the failure recovery program (1) performs redundancy of logical chunks associated with a pair of storage control units unrelated to the lost storage control unit for the data redundancy units of each storage node. Instruct to confirm the degree of degradation and recover the degree of redundancy. The data redundancy unit that has received the instruction executes redundancy recovery processing according to the redundancy method.

For example, in the case of a redundancy method in which physical chunks are duplicated (duplicated), the data redundancy unit checks from the logical chunk management table 271 whether or not there is a logical chunk storing a physical chunk in the failed storage node. Confirm. If it exists, it is checked whether the storage controller pair ID 2712 associated with the relevant logical chunk is a storage controller pair including the lost storage controller. If the storage control unit pair does not include the lost storage control unit, secure a new physical chunk, copy data from the unlost physical chunk that constitutes the logical chunk to the newly secured physical chunk, Update the mapping between logical chunks and physical chunks. When the confirmation of the reduction in the redundancy of the logical chunk and the restoration of the redundancy are completed in all storage nodes, the process proceeds to step S105.

Next, the failure recovery program (1) selects a storage control pair to be processed from the storage control pairs specified in step S100 (S105). Subsequently, the failure recovery program (1) evacuates all volumes handled by the relevant storage controller pair to a plurality of normal storage controller pairs that are not affected by the storage node failure. Execute the process (S106). In other words, all the volumes that the relevant storage control unit pair was in charge of are distributed to different storage control unit pairs for each volume and saved. Details of this volume saving process will be described later with reference to FIGS. , IO requests from the host device to the volume are processed by the save destination storage control unit pair.

After all volumes have been saved, the failure recovery program (1) instructs the cluster controller of the storage node in which the remaining storage controller is operating to delete the storage controller pair (S107). Details of the deletion processing of the storage control unit pair will be described later with reference to FIG. 16 . By deleting the storage control unit pair, the information processing resources such as CPU and memory allocated to the storage control unit pair and the storage resources such as logical chunks and physical chunks are released. After the deletion is completed, the fault recovery program (1) updates the storage controller management table 257 and deletes the record of the storage controller pair 2572 (S108).

Next, the failure recovery program (1) determines whether or not deletion of all storage control unit pairs identified in step S100 has been completed (S109). If completed, proceed to step S110; if not completed, return to step S105.

When the deletion of the storage controller pair that lost one of the pairs is completed, the failure recovery program (1) sends a message to the cluster controller of the storage node where the remaining storage controller of the storage controller pair that lost one of the pair was located. command to create a storage controller and a storage controller pair (S110). The cluster control unit (storage control unit pair creation program 253) that received the instruction secures information processing resources such as CPU cores and memory, loads the programs that make up the storage control unit from the storage device into the memory, and performs storage control. start the department. New storage controllers and storage controller pairs are created using the information processing resources released by the deletion. After completing the creation of the storage controller and the storage controller pair, the failure recovery program (1) updates the storage controller management table 257 and adds a record.

The storage control unit and storage control unit pair creation processing by the storage control unit pair creation program 253 are the same as when the storage system is constructed, so the details are omitted.

Next, an example of the processing in FIG. 13 will be described. The failure recovery program (2) 250 of FIG. 13 executes step S100 of FIG. 12 to identify the storage controller pair in which one of the storage controller pairs has been lost (S200). At this time, processing for inhibiting creation of a new volume in the identified storage control unit pair may be added. This process can be implemented by adding a volume creation inhibition flag column to the storage control unit management table 257 in FIG. 6 and turning this flag ON for the specified storage control unit.

Subsequently, the failure recovery program (2) executes steps S101 to S104 in FIG. 12 to issue an IO request to the volume that was being executed by the active mode storage controller operating on the failed storage node. Such processing is handed over to the standby-mode storage control units forming the pair (S201).

Although not shown in the figure, before proceeding to step S202, it is checked whether or not the remaining free storage capacity of each storage node is sufficient to recover from the failure of the storage node. Failure of failure recovery processing due to In this case, if it is determined that the free storage capacity is insufficient, the administrator of the storage system 200 is notified of the lack of free storage capacity via the management terminal 110, and the processing of the program is terminated. After that, the administrator of the storage system 200 takes measures such as adding storage devices and adding storage nodes, and then instructs the master role cluster control unit 216a to re-execute the failure recovery program 250 via the management terminal 110. do.

If the configuration is such that physical chunks are stored in storage nodes other than the storage node where the storage control unit is located, or if the physical chunks are tripled, RAID or erasure coding is applied between the storage nodes. If chunk redundancy is used, physical chunks lost due to a storage node failure may reduce the redundancy of logical chunks associated with storage controller pairs unrelated to the lost storage controller. be. Therefore, before proceeding to step S202, the failure recovery program (2) performs redundancy of logical chunks associated with a pair of storage control units unrelated to the lost storage control unit for the data redundancy units of each storage node. Instruct to confirm the degree of degradation and recover the degree of redundancy. The data redundancy unit that has received the instruction executes redundancy recovery processing according to the redundancy method. For example, in the case of a redundancy method in which physical chunks are duplicated (duplicated), the data redundancy unit checks from the logical chunk management table 271 whether or not there is a logical chunk storing a physical chunk in the failed storage node. Confirm. If it exists, it is checked whether the storage controller pair ID 2712 associated with the relevant logical chunk is a storage controller pair including the lost storage controller. If the storage control unit pair does not include the lost storage control unit, secure a new physical chunk, copy data from the unlost physical chunk that constitutes the logical chunk to the newly secured physical chunk, Update the mapping between logical chunks and physical chunks. When the confirmation of the reduction in the redundancy of the logical chunk and the recovery of the redundancy are completed in all storage nodes, the process proceeds to step S202.

Next, the failure recovery program (2) selects a storage control pair to be processed (S202). Similar to FIG. 12, according to the example shown in FIG. 10, there are two storage controller pairs in which one of the pairs has been lost, one of the remaining storage controllers is operating in active mode, and the other is Working in standby mode.

The failure recovery program (2) determines whether or not the operation mode of the remaining storage control unit is standby (S203), and if not standby mode, skips step S204 and proceeds to step S205. If in the standby mode, steps S106 to S108 of FIG. 12 are executed in step S204 to save all volumes handled by the storage control unit pair, delete the storage control unit pair, and extract the storage unit from the storage control unit management table 257. Delete records related to control unit pairs.

Next, the failure recovery program (2) determines whether or not deletion of the storage control unit pair in which the standby mode storage control unit remains among the storage control unit pairs identified in step S200 has been completed ( S205). If completed, proceed to step S206; if not completed, return to step S202.

When the deletion of the storage control unit pair in which the standby mode storage control unit remains is completed, the failure recovery program (2) deletes the cluster control unit (storage control unit pair The reconstruction program 258) is instructed to reconstruct the remaining storage controller pair by the active mode storage controller (S206).

Upon receiving the instruction, the cluster control unit secures information processing resources such as a CPU core and memory, loads a program that configures the storage control unit from the storage device into the memory, and activates the storage control unit. After startup, the volume management table 261 is copied from the active mode storage controller. Reconstruction of the storage controller pair is performed using the information processing resources released by the deletion. After completing the creation of the storage control unit and the storage control unit pair, the failure recovery program (2) updates the storage control unit management table 257 and updates the information of the storage control unit pair. Since this reconstruction process is the same as the redundancy recovery of the storage control unit pair when the present invention is not applied, the details are omitted. In step S200, if processing for inhibiting new volume creation for the specified storage control unit pair is added, processing for canceling the inhibition of new volume creation is added after this step. This process is performed by turning off the added volume creation suppression flag for the storage control unit that is the storage control unit pair identified by the storage control unit pair ID 2572 in the storage control unit management table 257 of FIG. realizable.

In the case of the processing shown in FIG. 12, the two storage control unit pairs that have lost one of the pairs perform volume evacuation in order, but in reality, the two storage control unit pairs may simultaneously perform volume evacuation. In the case of the processing according to FIG. 13, the storage control unit in which the active mode storage control unit remains until the volume evacuation from the storage control unit pair in which the standby mode storage control unit remains and the deletion of the storage control unit pair are completed. It is not possible to rebuild the storage controller that is the pair partner of the pair. Therefore, the processing according to FIG. 12 shortens the time when the redundancy of the volume is low, and the availability is high. On the other hand, in the case of the processing shown in FIG. 12, the newly created storage control unit pair will not be in charge of any volumes immediately after the completion of the failure recovery processing, and the information processing resources assigned to the storage control unit will not be used. It becomes inefficient.

Thus, the two processes in FIG. 12 and FIG. 13 are in a trade-off relationship between availability and utilization efficiency of information processing resources. An actual storage system may have only one of the two processing methods shown in FIGS. 12 and 13, or both may be provided, and the cluster controller selects which processing method according to some criteria. It may be determined whether to execute it, or a method set in advance by the storage system administrator may be executed. The present invention improves the utilization efficiency of information processing resources by eliminating the need for reserved resources for reconstructing the storage control unit. A configuration in which resources are reserved may also be used. In this case, if up to one storage node fails, the storage control unit can be reconstructed using the reserved information processing resource without saving the volume.

In the case of the processing in FIG. 12, the newly created storage control unit pair will not be in charge of even a single volume immediately after the completion of the failure recovery processing. By referring to the CPU usage rate 2577, etc. in the storage control unit management table 257 shown, the storage control unit with a high load and the volume handled by the storage control unit pair are moved to the newly created storage control unit pair, reducing the load on the entire system. You may perform the process which aims at dispersion|distribution of.

By the way, when the free storage capacity of the entire storage system runs short, it is common practice in SDS to solve the storage capacity shortage by adding a new storage node. In this case, the storage capacity shortage is resolved by allocating the physical chunks of the newly added storage node to the logical chunks. On the other hand, some erasure coding has the feature (read locality) that read processing can be performed without accessing physical chunks of other storage nodes. In a storage system to which erasure coding with read locality is applied, when a new storage node is added and physical chunks of the added storage node are assigned to logical chunks in order to solve the shortage of storage capacity, the feature of read locality is added. There is the problem of being lost. Features of erasure coding with read locality by moving volumes to a newly created storage controller pair in a newly added node in the same way that volumes are moved to a newly created storage controller pair after failure recovery processing. You can solve the lack of free storage space without losing the

FIG. 14 is a diagram showing an example of processing of the volume saving program 251. As shown in FIG. This program 251 saves all the volumes handled by the storage control unit pair, one of which has been lost due to a storage node failure, to a normal storage control unit pair whose redundancy has not decreased, as instructed by the failure recovery program 250. process to cause In other words, a process of distributing and saving all the volumes handled by the storage control unit pair in which one of the pairs has been lost to a plurality of normal storage control unit pairs whose redundancy has not decreased is performed. This program 251 is stored in the memory of the cluster control unit 216 and executed by the CPU of the storage node in which the cluster control unit is arranged. This program 251 is started from the failure recovery program 250 executed by the cluster control unit 216 of the master role, and is executed by the cluster control unit. If the cluster controller of the master role fails due to a failure, the cluster controller of another worker role in the cluster becomes the master role and executes.

The volume evacuation program 251 acquires a list of volumes that are in charge of the evacuation source storage control unit pair specified by the failure recovery program 250 (S300). In other words, the information of all the volumes handled by the storage control unit pair in which one of the pairs has been lost is obtained. As described above, the volume management table 261 is information existing in the memory of the storage controller. Therefore, information is actually acquired via the cluster controller of the storage node in which the remaining storage controller of the relevant storage controller pair is arranged. The volume save program 251 receives the volume management table 261 from the remaining storage control unit of the storage control unit pair via the cluster control unit of the storage node where the remaining storage control unit is arranged, and corresponds to the storage control unit pair ID 2614. Get all volume ID2611.

Next, the volume evacuation program 251 selects unevacuated volumes one by one from all the volumes corresponding to the acquired storage control unit pair ID 2614 (S301), and determines the volume evacuation destination storage control unit pair. , volume save destination determination processing is executed for each selected volume (S302). Details of the volume save destination determination process will be described later with reference to FIG.

After the volume evacuation destination is determined, the volume evacuation program 251 instructs the evacuation source storage control unit and the evacuation destination storage control unit to move the volume (S303). The volume migration process itself is the same as the technique and function generally called volume migration, so the explanation is omitted. After completing the evacuation of the volume, the volume evacuation program 251 determines whether or not evacuation of all the volumes identified in step S300 has been completed (S304). When evacuation of all volumes is completed, the volume evacuation program 251 terminates processing. On the other hand, if evacuation of all volumes has not been completed, the volume evacuation program 251 returns to step S301 and executes steps S302 to S304 for another volume.

In the example of FIG. 14, the volumes are evacuated one by one while waiting for the completion of the movement of the volumes. However, in this case, when determining the save destination storage control unit pair, it is necessary to consider the influence of the volumes that are currently being saved in parallel.

Also, when creating volumes, etc., it is possible to rank each volume such as Gold (high rank), Silver (middle rank), Bronze (low rank), etc., and the order of volume migration is determined according to the rank. You can change it. Similarly, the processing speed for volume migration may be changed according to the rank of the volume. For example, the higher the rank, the earlier the volume is migrated and the higher the processing speed is, so that the period during which the redundancy of the volume with the higher rank is reduced can be shortened. These processes add a column indicating the rank of the volume to the volume management table 261 of FIG. You can do this by referencing columns.

In the example of FIG. 14, the save destination storage control unit pair is determined independently for each volume. Alternatively, one save destination storage controller pair may be determined by In a storage system with a deduplication function that removes duplicated data between multiple volumes, even if a single save destination storage control unit pair is determined for multiple volumes with a high degree of data duplication, good.

FIG. 15 is an example of the processing of the volume save destination determination program 252. As shown in FIG. This program 252 performs the process of determining the optimum storage control unit pair as the save destination of the volume instructed by the volume save program 251 . That is, the processing shown in FIG. 15 is executed for each volume for all volumes handled by the storage control unit pair that has lost one of the pairs. This program 252 is stored in the memory of the master role cluster control unit 216 and is executed by the CPU of the storage node in which the cluster control unit is arranged. This program 252 is started from the volume save program executed by the cluster control unit of the master roll, and is executed by the cluster control unit 216 concerned. If the cluster controller of the master role fails due to a failure, the cluster controller of another worker role in the cluster becomes the master role and executes.

The volume save destination determination program 252 refers to the storage control unit management table 257 and obtains a list of storage control unit pairs that are save destination candidates (S400). In other words, all information of the storage controller pair ID 2572 of the storage controller management table 257 is obtained.

From the acquired list of storage controller pairs, select one storage controller pair to be processed (S401), and select one storage controller to be processed from the storage controllers that make up the storage controller pair. Select one by one (S402). The volume save destination determination program 252 determines whether the operating state of the storage node in which the storage control unit is arranged is normal (S403). If not normal, return to step S401 and execute S402 to S407 for another storage controller pair.

If it is normal, when the volume is saved to the storage node where the storage control unit is arranged, it is checked whether the free space of the storage device of the storage node where the storage control unit is arranged is equal to or greater than the threshold. Confirm (S404). If less than the threshold, return to step S401 and execute S402 to S407 for another storage controller pair.

If it is equal to or higher than the threshold, it is determined whether or not the communication band utilization rate of the storage node in which the storage control unit is arranged is equal to or lower than a predetermined threshold (S405). If it is larger than the threshold, the process returns to step S401 and executes S402 to S407 for another storage controller pair.

If it is equal to or less than the threshold, the storage control unit management table is referenced to determine whether the CPU utilization rate of the storage control unit is equal to or less than a predetermined threshold (S406). If it is larger than the threshold, the process returns to step S401 and executes S402 to S407 for another storage controller pair. If it is equal to or less than the threshold, the process proceeds to step S407.

The operating state of the storage node in which the storage control unit is arranged determined by the volume save destination determination program 252 in step S403 is the storage node ID 2573 corresponding to the storage control unit ID 2571 selected in step S402 from the storage control unit management table 257. can be obtained from the operating state 2563 of the storage node management table 256 corresponding to the storage node ID 2573.

The free capacity of the storage node after volume evacuation determined in step S404 is obtained by subtracting the total storage device usage 2568 from the storage device total capacity 2567 corresponding to the storage node ID 2561 in the storage node management table 256, It can be obtained by subtracting the used capacity 2613 corresponding to the volume ID 2611.

The communication band utilization rate of the storage node determined in step S405 is obtained by acquiring the storage node ID 2573 corresponding to the storage control unit ID 2571 selected in step S402 from the storage control unit management table 257, and calculating the storage node management table corresponding to the storage node ID 2573. It can be obtained from the operating state 2563 and communication band utilization rate 2566 of 256.

The CPU utilization rate of the storage control unit determined in step S406 by the volume save destination determination program 252 can be obtained from the CPU utilization rate 2577 of the storage control unit management table 257 corresponding to the storage control unit ID selected in step S402.

The thresholds used in the determinations in steps S404 to S406 may be fixed values for the entire storage system, or may be set for each storage node. If the settings can be made for each storage node, a column for each threshold is added to the storage node management table, and the volume save destination determination program acquires the threshold when executing steps S404 to S406.

If the physical chunk is stored in a storage node other than the storage node where the storage control unit is arranged, step S404 may be skipped across the board and the process may proceed to step S405.

The volume save destination determination program 252 determines whether or not steps S403 to S406 have been completed for all storage control units that make up the storage control unit pair (S407). If completed, the relevant storage control unit pair is determined as the save destination (S408). If there remains a storage control unit that has not been completed, the process returns to step S402 and executes steps S403 to S407.

The determination process in FIG. 15 is an example, and any determination process may be performed according to the structure and characteristics of the actual storage system. For example, the determinations in steps S403 to S406 may be made by executing at least one step according to the characteristics of the storage system required as the save destination. In addition, the distance on the network between the host device that uses the volume and the storage node where the storage control unit of the save destination candidate is arranged, the communication bandwidth utilization rate of the network switch that exists on the communication path through the network, etc. may be determined by In addition, in a storage system with a deduplication function, if there is a restriction that deduplication can be applied only between volumes within the same storage control unit pair, the amount of data reduction by the deduplication function changes depending on the save destination of the volume. In this case, even if you estimate in advance the amount of data that will be reduced by the deduplication function when the volume is evacuated to the storage control unit pair that is the backup destination candidate, and then evacuate to the storage control unit pair that has the largest amount of reduction. good. Also, the IO amount managed by the volume management table 261 (FIG. 7) is used, and the communication band utilization rate in step S405 and the CPU utilization rate threshold in step S406 are changed according to the IO amount for the volume to be saved. may be configured. In addition, information such as the operating frequency of the CPU installed in the storage node, the type and IO performance of each installed storage device, etc. is added to the storage node management table 256 in FIG. By selecting a storage control unit placed on a storage node that is equipped with a CPU and storage device equivalent to or higher than those installed in the , as the save destination, even after saving the volume, the same level as before saving IO performance may be obtained.

FIG. 16 is an example of processing of the storage controller pair deletion program 254 . This program 254 performs the processing of deleting the storage control units instructed by the failure recovery program 250 . This program 254 is stored in the memory 212 of the cluster control unit 216 and executed by the CPU of the storage node in which the cluster control unit is arranged. This program 254 is started from the failure recovery program 250 executed by the master role cluster controller. Then, it is executed by the cluster controller of the storage node in which the remaining storage controller of the pair of storage controllers to be deleted and one of which has been lost is operating.

The storage control unit pair deletion program 254 instructs the remaining storage control unit of the deletion target storage control unit pair operating in the relevant storage node to stop (S500).

When the storage control unit stops, the information processing resources such as CPU cores and memory allocated to the storage control unit are released (S501). After this program 254 is executed, the released information processing resources are used to create a new storage control unit, so they are not actually released. The planned information processing resource may be reused. Next, the storage control unit pair deletion program 254 deletes the logical chunks assigned to the storage control unit pair and the physical chunks associated with the logical chunks for the data redundancy unit 218 operating in the storage node. (S502). The data redundancy unit deletes the instructed logical chunks and physical chunks, and deletes the related records from the logical chunk management table 271 and the physical chunk management table 272. FIG.

As described above, according to the first embodiment, a plurality of volumes processed by the storage control unit are distributed to the normal storage control units without recovering the redundancy of the storage control unit whose redundancy has decreased. By evacuating and deleting the storage control unit itself whose redundancy has been lowered after the completion of the evacuation, the reserved information processing resource for guaranteeing the possibility of recovering the redundancy is unnecessary, and the utilization efficiency of the physical server is improved.

In addition, if a failure occurs in a storage node that configures the storage system, the control information (various management tables) managed by the storage node and data stored in physical chunks can be restored to normal without securing spare resources. It can be handed over to the storage node. Even after the processing is handed over to a normal storage node, by managing the correspondence between the storage controller and the volume and storing data in the storage node where the storage controller is located (ensuring data locality), high responsiveness to IO requests. In other words, when a data read request is issued to a storage device that provides a volume to a host device, there is no need to read data from another storage node.

In addition, since there is no need to secure spare resources, the construction cost of the storage system can be reduced, and the cost reduction required for SDS using virtualization technology can be efficiently met. In addition, in order to achieve the same level of availability as the conventional technology that secures reserve resources, the required number of CPU cores and memory capacity can be reduced by about 2/3. This makes it possible to reduce the storage system construction cost by 20%.

The second embodiment will be described in detail below with reference to FIG.

In Example 1, a recovery method when a storage node failure occurs has been shown. In a second embodiment, a technique applied to storage node reduction will be described. In other words, the process of removing the storage node from the storage system is performed. For ease of explanation, the storage controller group will be described below as a storage controller pair consisting of one active mode storage controller and one standby mode storage controller. However, not limited to pairs, even in the case of a storage control unit group composed of three or more storage control units, basically the same processing is performed. If a storage controller pair or group contains multiple active mode storage controllers, the active mode storage controllers can be treated basically the same as the standby mode storage controllers during the removal process. good.

FIG. 17 is an example of processing of the storage node reduction program 255. In FIG. This program 255 performs processing for removing a storage node specified by the storage system administrator from the storage system. This program 255 is started by the instruction of the administrator of the storage system 200 via the management terminal 110, and is executed by the CPU of the storage node in which the master role cluster controller is arranged. When the cluster control unit 218 of the master role periodically monitors each storage node and detects a sign of a storage node failure, this program 255 is started to prevent a decrease in redundancy due to a storage node failure. can be prevented. Although not shown, if the cluster control unit 218 arranged in the storage node to be removed is the master role, one of the worker role cluster control units arranged in the other storage nodes is assigned to the master role in advance. switch to

This program 255 is executed by the CPU of the storage node in which the cluster control unit that has newly switched to the master role is arranged. The method of selecting the cluster controller to switch to the master role may be the same as when the cluster controller in the master role is lost due to a storage node failure.

Also, the master role cluster control unit arranged in the storage node to be removed may select a new master role cluster control unit by some kind of determination processing. Before the storage node reduction program 255 starts processing, the total free space of the storage devices of all storage nodes constituting the storage system excluding the storage node to be removed and the storage capacity of the storage node to be removed If it is determined that the capacity of the storage device is insufficient during the storage node reduction by comparing with the total usage amount of the device, the processing is stopped. During the reduction process, a process for preventing capacity shortage during storage node reduction may be added.

Also, when a volume creation instruction is received from the storage system administrator during execution of the storage node reduction program 255, it is determined whether there is a possibility that the storage node reduction will cause a shortage of free space, If it is determined that the possibility of occurrence is high, processing such as stopping volume creation and preventing capacity shortage during storage node reduction may be added.

The storage reduction program 255 selects the storage controller pair 217 including the storage controller 219 arranged in the storage node to be removed (S600). This processing is realized by referring to the storage control unit management table 257, searching by the deletion target storage node ID 2573, and specifying the storage control pair ID. At this time, processing for inhibiting creation of a new volume in the identified storage control unit pair may be added. This processing adds a volume creation inhibition flag column to the storage control unit management table 257 of FIG. This can be achieved by setting As in FIGS. 12 and 13, according to the example shown in FIG. 10, there are two storage controller pairs including the storage controller arranged in the storage node to be removed. One of the connected storage controllers is operating in active mode and the other is operating in standby mode.

If the configuration is such that physical chunks are stored in storage nodes other than the storage node where the storage control unit 217 is arranged, or if the physical chunks are tripled, the physical chunks are stored by applying RAID or erasure coding between the storage nodes. are redundant, there is a possibility that the physical chunks that make up the logical chunks associated with the storage controller pair other than the storage controller pair identified in step S600 are saved in the storage node to be removed. be. Therefore, before proceeding to step S601, the storage node reduction program 255 sets the data redundancy units of all storage nodes other than the target storage node to be independent of the storage control unit arranged in the target storage node. It confirms whether or not the physical chunks that make up the logical chunks associated with the storage control unit pair that does not exist are arranged in the storage node to be removed, and instructs the reallocation to another storage node.

The data redundancy unit that has received the instruction confirms the allocation destination of the physical chunk and performs rearrangement according to the redundancy method. For example, in the case of a redundancy method in which physical chunks are duplicated (duplicated), the data redundancy unit checks from the logical chunk management table 271 whether or not there is a logical chunk that stores a physical chunk in the storage node to be removed. Confirm. If it exists, it is checked whether the storage controller pair ID 2712 associated with the relevant logical chunk is a storage controller pair including the storage controller arranged in the storage node to be removed. If the storage controller pair does not include the lost storage controller, a new physical chunk is secured, and data is copied from one of the physical chunks that make up the logical chunk to the newly secured physical chunk. After the copy is completed, the logical chunk management table 271 is updated, and among the physical chunks that make up the logical chunk, the physical chunk saved in the deletion target node is changed to the copy destination physical chunk.

When the physical chunk placement destinations have been confirmed and rearranged in all storage nodes other than those to be removed, the process proceeds to step S601.

The storage node removal program 255 selects a storage controller pair to be processed from the identified storage controller pairs (S601). It is determined whether or not the active-mode storage controller of the relevant storage controller pair is arranged in the removal target storage node (S602).

If so, steps S106 to S108 of FIG. 12 are executed to save all the volumes handled by the storage control unit pair and delete the storage control unit pair (S603). As a result, the information processing resources necessary for placing one storage control unit in the storage node where the standby mode of the storage control unit pair has been placed are released. If the active mode storage controller is not arranged, skip step S603 and proceed to step S604. In the process of S602, the storage control unit management table 257 is referred to, and the operation mode 2574 of the storage control unit obtained by searching with the AND condition of the storage node ID 2571 to be removed and the storage control unit pair ID 2572 selected in step S601 is set. It is realized by acquiring

The storage node reduction program 255 determines whether deletion of all storage control unit pairs in which active mode storage control units are arranged in the reduction target storage node has been completed (S604). If completed, proceed to step S605; if not completed, return to step S601.

Next, the storage node reduction program 255 instructs the cluster control unit 216 of the storage node not targeted for reduction that constitutes the deleted storage control unit pair to switch to the standby mode of the storage control unit pair remaining in the storage node targeted for reduction. Instruct to copy the storage controller (S605). Copying from the standby mode storage control unit without rebuilding from the active mode storage control unit of the relevant storage control unit pair is an obstacle to the IO processing of the volume for which the active mode storage control unit is in charge of processing. This is to minimize the impact.

Upon receipt of the instruction, the cluster control unit 216 secures information processing resources such as a CPU core and memory, loads a program that configures the storage control unit from the storage device into the memory, and activates the storage control unit. The information processing resources to be secured are released by deletion. Copy the volume management table from the standby mode storage controller of the storage controller pair remaining in the removal target storage node after startup.

After the copy of the storage control unit 217 is completed, the storage node reduction program 255 deletes the data redundancy unit of the storage node that constitutes the deleted storage control unit pair and is not targeted for reduction. The rearrangement of the physical chunks constituting the logical chunks associated with the control unit pair is instructed (S606). The data redundancy unit that has received the instruction copies the record associated with the storage control unit pair ID 2712 from the record of the logical chunk management table 271 of the data redundancy unit of the storage control unit to be removed.

After that, the data redundancy unit 218 reserves a new physical chunk and copies the data of the physical chunk saved in the deletion target storage control unit. The logical chunk management table is updated, and among the physical chunks that make up the logical chunks, the physical chunks saved in the storage node to be removed are changed to the copied physical chunks.

When the reallocation of the physical chunks is completed, the storage node reduction program 255 issues an instruction to switch the standby mode storage controller of the relevant storage controller pair to the copied storage controller, and updates the storage controller management table 257. (S607). In step S600, if processing for inhibiting creation of a new volume for the specified storage control unit pair is added, processing for canceling inhibition of creation of new volume is added to this step. This process is performed by turning off the added volume creation suppression flag for the storage control unit that is the storage control unit pair identified by the storage control unit pair ID 2572 in the storage control unit management table 257 of FIG. realizable.

Next, the storage node removal program 255 instructs the cluster control unit and data redundancy unit operating in the storage node to be removed to stop (S608), and retrieves the record related to the storage node from the storage node management table 256. Delete (S609). When the above processing is completed, the storage node to be removed is completely separated from the storage system, and there is no problem even if the storage node is physically removed.

As described above, according to the second embodiment, a plurality of volumes that were in charge of the storage control unit operating in the storage node to be deleted are distributed to the storage control units not to be deleted and saved, and after the backup is completed, the volumes are deleted. By deleting the storage control unit that operates on the storage node to be installed, it is possible to eliminate the need for a reserved spare information resource for reducing the number of storage nodes and improve the utilization efficiency of the physical server.

In addition, even after other storage controllers have taken over the processing of IO requests for volumes that were in charge of the storage controller running on the storage node to be removed, data will not be sent to the storage node where the storage controller is located. By saving (ensuring data locality), it is possible to maintain high responsiveness to IO requests from the host device. In other words, when a data read request is issued to a storage device that provides a volume to a host device, there is no need to read data from another storage node.

In addition, there is no need to secure spare resources for reducing storage nodes, and the storage system can be scaled in, making it possible to improve the scalability required for SDS using virtualization technology. Become.

As described above, in the first embodiment, the volume is saved from the storage controller pair configured using the storage controller of the failed storage node, but in the second embodiment, the storage node associated with the storage controller is reduced. The case of letting

In addition, the present invention may allow the storage node to continue operating without failure or reduction, and the storage control unit pair to be changed in charge of the volume may continue to operate while leaving the volume in charge. In addition, the person in charge of the volume is distributed and moved from one storage control unit pair to a plurality of storage control unit pairs. Volume responsibility may be moved from storage controller pairs to one storage controller pair and from multiple storage controller pairs to multiple storage controller pairs.

100: host device, 200: storage system, 210: storage node, 211: CPU, 212: memory, 213: storage device, 214: communication device, 215:, 216: cluster controller, 217: storage controller pair, 218 : data redundancy unit, 219: storage control unit, 220: volume, 221: logical chunk, 222: physical chunk, 250: failure recovery program, 251: volume evacuation program, 252: volume evacuation destination determination program, 256: storage node Management table, 257: Storage control unit management table, 300: Network.

Claims (6)

In a storage system having multiple storage nodes forming a cluster,
a storage device for storing data;
a cluster control unit that controls the storage system;
a storage control unit provided in each storage node for providing a host device with a storage area in units of volume using the storage device and storing data in the storage device in response to an IO request from the host device; death,
The storage controller constitutes a storage controller group with the storage controllers of other storage nodes in the cluster, and at least one storage controller in the storage controller group is configured as an active mode storage controller from a host device. processing IO requests of the storage controller group and allowing other storage controllers of the storage controller group to take over the processing of the active mode storage controller;
The cluster control unit
When the number of operating storage nodes decreases due to reduction or failure, the identification information of the volumes handled by the first storage control unit group is acquired, and for each volume corresponding to the acquired identification information, a plurality of second A storage control unit group is determined, and a volume managed by the first storage control unit group is moved from the storage control units constituting the first storage control unit group to the storage control units of the second storage control unit group. ,
A storage system that deletes a storage control group that has no volume by moving the volume . When removing a storage node, the volume is moved,
2. The storage system according to claim 1 , wherein the volume of the storage controller group in which the active mode storage controller is arranged in the node to be removed is moved to another storage controller group. When a standby mode storage controller capable of taking over the processing of the active mode storage controller is arranged in the node to be removed, the volume of the storage controller group is moved to another node and the standby mode 2. The storage system according to claim 1 , wherein the storage controller is activated to become a standby mode storage controller of the same storage controller group. When a failure occurs in a storage node, the volume is moved,
the storage controller group in which the active mode storage controller is arranged in the failed node switches the storage controller arranged in another node to active;
2. The method according to claim 1 , wherein at least one storage controller group in which the storage controller is arranged in the failed node is deleted by moving the volume to another storage controller group. storage system. A storage control unit is arranged in the node where the failure occurred, and at least one storage control unit group having the volume is a node where the storage control unit of the storage control unit group to be deleted is arranged and no failure has occurred. 5. The storage system according to claim 4 , wherein the volume is copied and the storage controller is activated to form a storage controller group. In a control method for a storage system having a plurality of storage nodes forming a cluster,
The storage system is
a storage device for storing data;
a cluster control unit that controls the storage system;
a storage control unit provided in each storage node for providing a host device with a storage area in units of volume using the storage device and storing data in the storage device in response to an IO request from the host device; death,
The storage controller constitutes a storage controller group with the storage controllers of other storage nodes in the cluster, and at least one storage controller in the storage controller group is configured as an active mode storage controller from a host device. processing IO requests of the storage controller group and allowing other storage controllers of the storage controller group to take over the processing of the active mode storage controller;
When the number of operating storage nodes decreases due to deletion or failure, the cluster control unit acquires identification information of the volumes that the first storage control unit group is in charge of, and for each volume corresponding to the acquired identification information: , a plurality of second storage controller groups are determined, and from the storage controllers constituting the first storage controller group to the storage controllers of the second storage controller group, the first storage controller group Move the volume in charge,
A control method for a storage system, comprising the steps of: deleting a storage control group in which the volume has disappeared by moving the volume .

Images