JP5404947B2 - Storage device - Google Patents

Description

  Embodiments described herein relate generally to a storage apparatus and a data processing method in which memory nodes having a transfer function are connected to each other, for example, a data packet transfer control system in the storage apparatus.

  As a storage device whose capacity can be easily expanded, a storage device in which memory nodes having a transfer function are connected to each other can be considered. Each memory node performs a predetermined process such as reading or writing when receiving a data packet addressed to its own memory node. On the other hand, when each memory node receives a packet that is not addressed to its own memory node, it transfers the received packet to another appropriate memory node. By repeating appropriate transfer by each memory node, the data packet can reach the target memory node.

  Each memory node has a memory, a controller having a transfer function, and a plurality of ports. Each memory node maintains and manages a routing table (routing table) indicating the transfer destination of the packet, and transfers the packet accordingly. If the routing table is managed, it is possible to construct an arbitrary logical packet transfer network regardless of the physical position.

  However, when adding a new memory node for capacity expansion or removing an existing memory node due to a failure or the like, it is necessary to update the routing table of each memory node, and the procedure is complicated. It is. The routing table maintenance and management costs become enormous, especially when the number of memory nodes becomes large, which imposes restrictions on capacity scalability.

  In addition, in a storage apparatus in which memory nodes having a transfer function are connected to each other, it is generally difficult to simultaneously communicate a plurality of data through the same wiring when writing / reading a plurality of data to / from a plurality of memory nodes. Therefore, there is a tendency to wait for data transfer. Such data transfer waiting increases the time required for data writing / reading.

US Patent Application Publication No. 2009/0216924

  There is provided a storage device and a data processing method capable of efficiently transferring a packet without requiring a memory node to manage a routing table.

A storage apparatus according to an embodiment includes a first memory node including a first input port, a first nonvolatile memory, and a first control unit that controls the first nonvolatile memory; A second input port connected to the first memory node, a second nonvolatile memory, a second control unit for controlling the second nonvolatile memory, and any one of the first input port A second memory node including a plurality of first output ports connected to the input port; a third input port connected to one of the plurality of first output ports; A storage device comprising a plurality of third memory nodes different from the first to second memory nodes, wherein the second memory node is position information of the second memory node in the storage device Memorize the first position information , The first control unit outputs including at least a packet destination address and the source address to the second memory node, said second input port, the output from the first memory node The second control unit receives a packet, and the second control unit includes at least the plurality of third memory nodes based on information including at least the transmission destination address, the transmission source address, and the first position information in the packet. and is configured to determine an output destination of said packet from said first memory node.

It is a figure which shows the structure of the storage apparatus of 1st Embodiment. It is a figure which shows the structure of the memory node in 1st Embodiment. It is a figure which shows the example of arrangement | positioning of the memory node in 1st Embodiment. It is a figure which shows the transfer algorithm 1 in the storage apparatus of 1st Embodiment. It is a figure which shows the transfer process of the packet by the transfer algorithm 1 in 1st Embodiment. It is a figure which shows the transfer process of the packet by the transfer algorithm 1 in 1st Embodiment. It is a flowchart of the transfer process of the packet by the transfer algorithm 1 in 1st Embodiment. It is a figure which shows the specific example of the transfer process of the packet by the transfer algorithm 1 in 1st Embodiment. It is a figure which shows the structure of the storage system containing the storage apparatus of 1st Embodiment. It is a figure which shows the write-in operation | movement in the storage system of 1st Embodiment. It is a figure which shows the read-out operation | movement in the storage system of 1st Embodiment. It is a figure which shows the automatic address acquisition system of the memory node in the storage apparatus of 1st Embodiment. It is a figure which shows the transfer algorithm 2 in the storage apparatus of 2nd Embodiment. It is a figure which shows the transfer process of the packet by the transfer algorithm 2 in 2nd Embodiment. It is a figure which shows the transfer process of the packet by the transfer algorithm 2 in 2nd Embodiment. It is a flowchart of the transfer process of the packet by the transfer algorithm 2 in 2nd Embodiment. It is a figure which shows the specific example of the transfer process of the packet by the transfer algorithm 2 in 2nd Embodiment. It is a figure which shows the specific example of the transfer process of the packet by the transfer algorithm 1. FIG. It is a figure which shows the specific example of the transfer process of the packet by the transfer algorithm 2. FIG. It is a figure which shows the transfer algorithm 3 in the storage apparatus of 3rd Embodiment. It is a figure which shows the transfer process of the packet by the transfer algorithm 3 in 3rd Embodiment. It is a figure which shows the transfer process of the packet by the transfer algorithm 3 in 3rd Embodiment. It is a flowchart of the transfer process of the packet by the transfer algorithm 3 in 3rd Embodiment. It is a figure which shows the specific example of the transfer process of the packet by the transfer algorithm 3 in 3rd Embodiment. It is a figure which shows the transfer algorithm 4 in the storage apparatus of 3rd Embodiment. It is a figure which shows the transfer algorithm 5 in the storage apparatus of 3rd Embodiment. It is a figure which shows the structure of the storage system of 4th Embodiment. It is a figure which shows the bypass transfer incidence and the traffic congestion incidence in the storage system of 4th Embodiment. It is a figure which shows the structure of the storage apparatus of 5th Embodiment. It is a figure which shows the structure of the storage system of 6th Embodiment. It is a figure which shows the other structural example of the storage system of 6th Embodiment. It is a figure which shows the structure of the storage system of 7th Embodiment. It is a figure which shows the comparative example with respect to the storage system of 7th Embodiment. It is a figure which shows the address information recorded on the header part of the packet in 7th Embodiment. It is a figure which shows write-in operation | movement in the storage system of 7th Embodiment. It is a figure which shows the other structural example of the storage system of 7th Embodiment. It is a figure which shows the structure of the storage system of 8th Embodiment. It is a figure which shows the comparative example with respect to the storage system of 8th Embodiment. It is a figure which shows the address information recorded on the header part of the packet in 8th Embodiment. It is a figure which shows write-in operation | movement in the storage system of 8th Embodiment. It is a figure which shows the other structural example of the storage system of 8th Embodiment. It is a figure which shows the other structural example of the storage system of 8th Embodiment. It is a figure which shows the structure of the storage system of 9th Embodiment. FIG. 10 is a diagram illustrating a read operation in which a transfer standby occurs in the storage system. FIG. 10 is a diagram illustrating a read operation in which a transfer standby occurs in the storage system. FIG. 10 is a diagram illustrating a read operation in which a transfer standby occurs in the storage system. It is a figure which shows the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the other example of the read-out operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 9th Embodiment. It is a figure which shows the structure of the storage system of 10th Embodiment. FIG. 10 is a diagram illustrating a write operation in which a transfer standby occurs in a storage system. FIG. 10 is a diagram illustrating a write operation in which a transfer standby occurs in a storage system. FIG. 10 is a diagram illustrating a write operation in which a transfer standby occurs in a storage system. FIG. 10 is a diagram illustrating a write operation in which a transfer standby occurs in a storage system. FIG. 10 is a diagram illustrating a write operation in which a transfer standby occurs in a storage system. FIG. 10 is a diagram illustrating a write operation in which a transfer standby occurs in a storage system. It is a figure which shows the write-in operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 10th Embodiment. It is a figure which shows the write-in operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 10th Embodiment. It is a figure which shows the write-in operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 10th Embodiment. It is a figure which shows the write-in operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 10th Embodiment. It is a figure which shows the write-in operation | movement which avoids generation | occurrence | production of transfer waiting in the storage system of 10th Embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. In the following description, components having the same function and configuration are denoted by the same reference numerals, and redundant description will be given only when necessary.

[First Embodiment]
The first embodiment includes a storage apparatus in which a plurality of memory nodes having a transfer function are connected to each other, and has a transfer method in which each memory node efficiently transfers a data packet.

[1] Configuration of Storage Device FIG. 1 is a diagram showing a configuration of a storage device according to the first embodiment, and shows an example of a physical arrangement method of a plurality of memory nodes and an address allocation method corresponding thereto.

  As illustrated, the storage apparatus 10 includes a plurality of memory nodes 11 having a data transfer function. Each memory node 11 is placed at a lattice point of a square lattice. The logical address of the memory node located at the lattice coordinate (x, y) that is the coordinate of the lattice point coincides with the position coordinate and is (x, y). That is, the logical address of the memory node 11 matches the physical address (lattice coordinate (x, y)).

  Each memory node 11 has four input ports 12 and four output ports 13. Each memory node is connected to four adjacent memory nodes via the input port 12 and the output port 13. Specifically, the input port 12 and the output port 13 of each other are connected to two adjacent memory nodes facing each other.

  FIG. 2 shows the configuration of each memory node 11. The memory node 11 includes an input port 12, an input port buffer 12A, an output port 13, an output port buffer 13A, a selector 14, a packet controller 15, a memory 16, a memory controller 17, an MPU 18, and a local bus 19.

  A packet input to the input port 12 is temporarily stored in the input port buffer 12A. The selector 14 receives a packet from the input port buffer 12 </ b> A and receives a control signal from the packet controller 15. The selector 14 selects one of the input packets according to the control signal and outputs the selected packet to the output port buffer 13A. The output port buffer 13A temporarily stores the packet output from the selector 14 and outputs it to the output port 13. The packet controller 15 controls the output of the selector 14. The packet is a unit of transfer data including a header part including at least a transmission destination address and a transmission source address, and a data part.

  The memory 16 has a plurality of memory cells that store data. The memory 16 is composed of, for example, a NAND flash memory. The memory controller 17 controls writing, reading and erasing operations to the memory 16. The MPU 18 performs necessary arithmetic processing in the memory node. The local bus 19 interconnects the input port buffer 12A, the packet controller 15, the memory controller 17, and the MPU 18, and performs signal transmission among them.

  The packet received by the memory node 11 is stored in the input port buffer 12A via the input port 12. The packet controller 15 determines whether or not the received packet is addressed to the own node based on two pieces of information, that is, the destination (destination) address included in the packet and the address of the memory node itself (hereinafter referred to as the own node). .

  If it is addressed to the own node, the packet controller 15 performs writing, reading, or other predetermined processing to the memory 16 of the own node. If not addressed to the own node, an adjacent memory node to which the packet is to be transferred is determined based on the two information of the packet transmission destination address and the address of the own node, and the selector 14 sends the packet to the corresponding output port buffer 13A. Is output.

  In the storage device shown in FIG. 1, memory nodes are arranged at lattice points of a square lattice, but the present embodiment is not limited to this example. This will be described with reference to FIG.

The configuration shown in FIG. 3A corresponds to the example shown in FIG. More generally, in the present embodiment, each memory node is arranged at a lattice point. A lattice refers to a point on the plane where both x and y coordinates are integers in geometry. Here, it includes the case where the unit vectors in the x direction and the y direction, the lengths of _ex and e _y are different, that is, the case where the repetition period in the x direction and the y direction are different. An example of this is shown in FIG.

  In addition, the case where the unit vectors in the x direction and the y direction are not orthogonal, that is, the case where the x axis and the y axis are not orthogonal is included. An example of this is shown in FIG. Also, the number of memory node interconnections is not necessarily four. An example in which the number of interconnections is 6 is shown in FIG.

  In this embodiment, regardless of the definition of the lattice, in any case, the logical address of the memory node located at the lattice coordinate (x, y) coincides with the position coordinate and is (x, y). That is, the logical address of the memory node matches the physical address (lattice coordinate (x, y)).

  More generally, the present embodiment includes a case where a set of two or more memory nodes is arranged at each lattice point. An example in which two memory nodes are arranged at each lattice point is shown in FIG. In this example, two memory nodes are arranged at a lattice point (1, 0) or the like. These are (1, 0, 0) and (1, 0, 1). That is, the coordinates of one memory node are represented by a set of three integers (x, y, z). Even in this case, the logical address of the memory node located at the node coordinates (x, y, z) coincides with the position coordinates and is (x, y, z). That is, the logical address of the memory node matches the physical address (lattice coordinates (x, y, z)). Furthermore, it includes the case where the plane on which the lattice points are arranged is bent or folded three-dimensionally.

  Adjacent memory nodes refer to memory nodes having the following positional relationship in FIGS. 3 (a) to 3 (e). 3A to 3C, for example, when the own node exists at the coordinates (1, 1), the memory nodes adjacent to the own node are coordinates (0, 1), (1, 2), ( 2, 1) and (1, 0). 3D, for example, when the own node exists at the coordinates (1, 1), the memory nodes adjacent to the own node are coordinates (0, 1), (0, 2), (1, 2), Six memory nodes existing at (2, 1), (2, 0), and (1, 0). Further, in FIG. 3E, for example, when the own node exists at the coordinates (1, 0, 1), the memory nodes adjacent to the own node are the coordinates (0, 1, 0), (1, 1, 0). ), Three memory nodes existing at (1, 0, 0).

[2] Storage device transfer algorithm 1
In the transfer algorithm 1, the memory node of the transfer destination is determined based on two pieces of information of the transmission destination address included in the packet and the address of the own node.

  FIG. 4 is a diagram showing a transfer algorithm 1 in the storage apparatus of the first embodiment. Referring to FIG. 4, an example of a method for determining a transfer destination adjacent node based on two pieces of information, that is, a packet transmission destination address and an address of the own node will be described. This is referred to as transfer algorithm 1.

  The memory node that has received the packet transfers the packet to an adjacent memory node where the distance between the packet destination node (to) and its own node (PP: Present position) is the smallest.

  With reference to FIGS. 5A to 5D, a specific example of a packet transfer process according to the transfer algorithm 1 will be described.

As shown in FIG. 5A, the address of the destination node (to) is (x _to , y _to ), the address of its own node (PP) is (x _pp , y _pp ), and dx = x _to −x _pp , Let dy = y _to −y _pp . As shown in FIG. 5B, the direction indicating the transfer destination is N (North) when y increases, E (East) when x increases, S (South) when y decreases, and x decreases. The direction to perform is W (West).

  If a symbol representing the absolute value of a is | a |, | dx |> | dy | proceeds in the x direction, and if | dx | <| dy |, proceeds in the y direction. When proceeding in the x direction, transfer to E if dx> 0, and to W if dx <0. Similarly, when proceeding in the y direction, transfer to N if dy> 0, and to S if dy <0.

  FIG. 5C shows a flow of the transfer algorithm 1 in the storage apparatus. The transfer algorithm 1 is stored in the packet controller 15 and executed by the packet controller 15.

First, the packet controller 15 calculates dx = x _to −x _pp and dy = y _to −y _pp (step S1). Subsequently, the packet controller 15 determines whether dx is 0 (step S2). When dx is 0, it is determined whether y _to > y _pp is satisfied (step S3). When y _to > y _pp holds, the packet is transferred to N (step S4). On the other hand, when y _to > y _pp does not hold, the packet is transferred to S (step S5).

Next, when dx is not 0 in step S2, the packet controller 15 determines whether dy is 0 (step S6). When dy is 0, it is determined whether x _to > x _pp is satisfied (step S7). When x _to > x _pp holds, the packet is transferred to E (step S8). On the other hand, when x _to > x _pp does not hold, the packet is transferred to W (step S9).

  Next, when dy is not 0 in step S6, that is, when dx and dy are not 0, the packet controller 15 determines whether dx> 0 and dy> 0 are satisfied (step S10). When dx> 0 and dy> 0 hold, it is determined whether dx> dy holds (step S11). When dx> dy holds, the packet is transferred to E (step S12). On the other hand, when dx> dy does not hold, the packet is transferred to N (step S13).

  Next, when dx> 0 and dy> 0 do not hold in step S10, the packet controller 15 determines whether dx <0 and dy> 0 hold (step S14). When dx <0 and dy> 0 hold, it is determined whether (−1) · dx> dy holds (step S15). (-1) When dx> dy holds, the packet is transferred to W (step S16). On the other hand, when (−1) · dx> dy does not hold, the packet is transferred to N (step S17).

  Next, when dx <0 and dy> 0 do not hold in step S14, the packet controller 15 determines whether dx <0 and dy <0 hold (step S18). When dx <0 and dy <0 hold, it is determined whether dx> dy holds (step S19). When dx> dy holds, the packet is transferred to S (step S20). On the other hand, when dx> dy does not hold, the packet is transferred to W (step S21).

  Next, when dx <0 and dy <0 do not hold in step S18, the packet controller 15 determines whether dx> (− 1) · dy holds (step S22). When dx> (− 1) · dy holds, the packet is transferred to E (step S23). On the other hand, when dx> (− 1) · dy does not hold, the packet is transferred to S (step S24).

  Through the above processing, the packet input to the memory node is transferred to an adjacent memory node where the distance between the transmission destination node (to) and the own node (PP) is minimized.

  FIG. 5D shows a specific example of the packet transfer process. In the transfer algorithm 1, until | dx | = | dy |, if | dx |> | dy |, proceed in the x direction, and if | dx | <| dy |, continue in the y direction, and | dx | = | After reaching dy |, the process proceeds alternately in the x and y directions.

  For example, in case 1, since dx = 0 and dy> 0, the process proceeds to N until the destination node (to) is reached. In case 4, since dx> 0 and dy> 0 and dx <dy, the process proceeds continuously to N until dx = dy, and after dx = dy, the process proceeds alternately to E and N.

[3] Storage System Including Storage Device FIG. 6 is a diagram showing a configuration of a storage system including the storage device of the first embodiment.

  The storage system 20 is a system for a client to use a storage apparatus, and has the following configuration.

  The storage device 10 is connected to a client via a gateway server. When the communication standard in the storage device 10 and the communication standard of the gateway servers 21A and 21B are different, the adapters 22A and 22B may be installed between the two.

  More specifically, the memory nodes (1, 4) arranged on the outer periphery of the storage apparatus 10 are connected to the client 31A via the adapter 22A and the gateway server 21A. Similarly, the memory node (1, 1) is connected to the clients 31B1 and 31B2 via the adapter 22B and the gateway server 21B. “Memory node (x, y)” represents a memory node at address (x, y). The same applies thereafter.

  The gateway servers 21A and 21B are composed of computers, for example, and have addresses based on the same rules as the memory node 11 inside the storage apparatus 10. In FIG. 6, the gateway server 21A has an address (0, 4), and the gateway server 21B has an address (0, 1).

  Next, a write operation in the storage system will be described.

  FIG. 7 shows a procedure for a client to write a file to the storage apparatus 10. Here, a case where the client 31A writes a file to the storage apparatus 10 will be described.

  The client 31A transmits the file and the file ID to the gateway server 21A (see (1)). The file ID is an identifier that can uniquely identify a file. As the file ID, a full path file name in a predetermined file system such as \\ strage_system \ home \ cliantA \ file1.txt can be used.

  Next, the gateway server 21A divides the file into data packets of a prescribed size and assigns a packet ID to each packet. Subsequently, the file ID and the packet ID of the divided packet are written in the file table (see (2)). The packet ID is an identifier that can uniquely identify the packet. As the packet ID, “file ID + consecutive number” can be assigned such as... \ File1.txt ~ 1, ... \ file1.txt ~ 2.

  Next, the gateway server 21A determines an address of a memory node (hereinafter referred to as a write node address) to which the packet is written based on the packet ID information (see (3)). At this time, a node determination method called consistent hashing (see Document 1) used in a large-scale distributed file system may be used. Consistent hashing is characterized in that the write address is determined using both the hash value of the node address and the hash value of the packet ID.

[Reference 1]: Fujio Maruyama / Kazuyuki Shudo, “Cloud technology to grab beyond the cloud world”, ASCII Media Works, Inc., November 6, 2009, p.88, ISBN978-4-04-868064 -6
Next, the gateway server 21A transmits the write packet to the storage device 10 using the write address as the transmission destination address and the address of the gateway server 21A as the transmission source address (see (4) and (5)).

  The packet transmitted to the memory node (1, 4) reaches the memory node at the destination address by repeating appropriate transfer in the storage apparatus 10 according to the transfer algorithm 1 (see (6)). The target memory node writes the received packet in the memory 16 of its own node (see (7)). Thereafter, a write completion packet is returned to the gateway server 21A (see (8)).

  Next, a read operation in the storage system will be described.

  FIG. 8 shows a procedure for a client to read a file from the storage apparatus 10. Here, a case where the client 31A reads a file from the storage apparatus 10 will be described.

  The client 31A transmits a read request (file ID) to the gateway server 21A (see (1)).

  The gateway server 21A acquires a packet ID corresponding to the file ID from the file table (see (2)). Subsequently, the gateway server 21A determines an address of a memory node to be read (hereinafter referred to as a read node address) based on the packet ID information (see (3)). At this time, a node determination method called consistent hashing used in a large-scale distributed file system may be used.

  Next, the gateway server 21A transmits the read packet to the storage device 10 using the read node address as the transmission destination address and the gateway server 21A as the transmission source address (see (4) and (5)).

  The packet transmitted to the memory node (1, 4) reaches the memory node at the destination address by repeating appropriate transfer in the storage apparatus 10 according to the transfer algorithm 1 (see (6)). The target memory node reads the target data from the memory 16 of its own node according to the read packet (see (7)). Thereafter, the read data is returned to the gateway server 21A as a data packet (see (8)).

  In the system shown in FIG. 6, randomization processing (Rand processing) of write data, addition of error-correcting code (ECC) to write data, and error detection and correction of read data using ECC are performed. The function (ECC function) may be performed by the gateway server. In this case, the memory controller of each memory node does not need to have a randomization process and an ECC function. Further, there is no need to perform ECC processing during communication between memory nodes. For this reason, the cost per memory node can be reduced.

[4] Extensibility of storage apparatus A method for adding a new memory node to the storage apparatus will be described.

  FIG. 9 shows an automatic address acquisition method when a new memory node is added to the storage apparatus.

  In the storage apparatus 10 of this embodiment, since the logical address of the memory node located at the lattice coordinate (x, y) coincides with the position coordinate to be (x, y), the added memory node addresses the adjacent node. Can be used to easily determine the address of the own node.

  For example, when the memory node to be added inquires of an adjacent node in the W direction about the address, if the address of the adjacent node is (x, y), the address of the additional node is (x + 1, y). Similarly, when an inquiry is made to the adjacent nodes (x, y) in N, E, and S, the addresses of the own nodes are (x, y-1), (x-1, y), (x, y + 1) and Become.

  In the case shown in FIG. 9, a situation is shown in which a new memory node is added in the E direction of the memory node (4, 4). The memory node to be added transmits an address_request packet in which to and from are none to the adjacent nodes (4, 4) in the W direction. The memory node (4, 4) that has received the address_request packet sets an address_answer packet with from as its own node address and to as (5, 4) in the output port buffer of the memory node in the E direction. The additional node can determine that the address of its own node is (5, 4) by receiving the address_answer packet.

  As another method, a manual address setting method is also conceivable. In the manual address setting method, the operator of the storage apparatus 10 determines the address of a memory node to be added in advance, presets it, and then adds a new memory node.

  The automatic address acquisition method described above needs to have a function for the packet controller of each memory node to respond to the Address_request packet. For this reason, the cost per packet controller tends to be expensive. On the other hand, since it is not necessary to preset the address of the memory node to be added, the procedure for adding the memory node is simplified and the operation cost can be reduced.

  As described above, according to the first embodiment, since the logical address of the memory node matches the physical position (physical address), it is not necessary for each memory node to manage the routing table. A data transfer method can be realized. For this reason, the expandability of the storage apparatus is excellent, that is, it is possible to easily add a memory node to the storage apparatus.

[Second Embodiment]
A storage apparatus according to the second embodiment will be described. The second embodiment has a transfer algorithm 2 that is different from the transfer algorithm 1 of the first embodiment. In the transfer algorithm 2, the memory node of the transfer destination is determined based on the three information of the transmission destination address and the transmission source address included in the packet and the address of the own node. Since this storage device has the same configuration as that shown in FIGS. 1 and 2, description thereof is omitted.

[1] Storage device transfer algorithm 2
FIG. 10 is a diagram illustrating a transfer algorithm 2 in the storage apparatus according to the second embodiment. Referring to FIG. 10, an example of a method for determining a transfer destination adjacent node based on three pieces of information of a packet transmission destination address, a transmission source address, and an address of the own node will be described. This is referred to as transfer algorithm 2.

  As shown in FIG. 10, it is determined in which of the two areas separated by a straight line connecting the transmission destination node (to) and the transmission source node (from) the own node (PP) exists and assigned to that area. Transfer the packet to adjacent memory nodes in the specified direction.

  A specific example of a packet transfer process according to transfer algorithm 2 will be described with reference to FIGS. 11A to 11D.

As shown in FIG. 11A, the address of the destination node (to) is (x _to , y _to ), the address of the source node (from) is (x _from , y _from ), and the address of its own node (PP) is ( x _pp , y _pp ). Further, dx = x _to −x _from , dy = y _to −y _from , Dx = x _pp −x _from , and Dy = y _pp −y _from . As shown in FIG. 11B, N indicates a direction in which y increases, E indicates a direction in which x increases, S indicates a direction in which y decreases, and W indicates a direction in which x decreases.

  Based on the equation of a straight line connecting the destination node (to) and the source node (from), y = (dx / dy) · x, and comparing which of Dy and (dy / dx) · Dx is larger, It is determined in which of the two areas separated by the straight line the own node (PP) exists.

  An example of assigning the transfer direction will be described by taking the case of dx> 0 and dy> 0 as an example. When Dy is compared with (dy / dx) · Dx, an area where Dy is larger than (dy / dx) · Dx is an A area, and the other area is a B area. In the transfer, E is assigned to the first priority direction of the A area, and N is assigned to the first priority direction of the B area. In the transfer algorithm 2, transfer is performed so that the packet proceeds along a straight line connecting the transmission destination node (to) and the transmission source node (from).

  FIG. 11C shows a flow of transfer algorithm 2 in the storage apparatus. The transfer algorithm 2 is stored in the packet controller 15 and is executed by the packet controller 15.

First, the packet controller 15 calculates dx = x _to −x _from , dy = y _to −y _from , Dx = x _pp −x _from , and Dy = y _pp −y _from (step S31). Subsequently, the packet controller 15 determines whether dx is 0 (step S32). When dx is 0, it is determined whether or not y _to > y _pp is satisfied (step S33). When y _to > y _pp holds, the packet is transferred to N (step S34). On the other hand, when y _to > y _pp does not hold, the packet is transferred to S (step S35).

Next, when dx is not 0 in step S32, the packet controller 15 determines whether dy is 0 (step S36). When dy is 0, it is determined whether x _to > x _pp is satisfied (step S37). When x _to > x _pp holds, the packet is transferred to E (step S38). On the other hand, when x _to > x _pp does not hold, the packet is transferred to W (step S39).

  Next, when dy is not 0 in step S36, that is, when dx and dy are not 0, the packet controller 15 determines whether or not Dy · dx> dy · Dx is satisfied (step S40). When Dy · dx> dy · Dx holds, it is determined whether dx> 0 and dy> 0 hold (step S41). When dx> 0 and dy> 0 hold, the packet is transferred to E (step S42). On the other hand, when dx> 0 and dy> 0 do not hold, it is determined whether dx <0 and dy> 0 hold (step S43). When dx <0 and dy> 0 hold, the packet is transferred to N (step S44).

  Next, when dx <0 and dy> 0 do not hold in step S43, it is determined whether dx <0 and dy <0 hold (step S45). When dx <0 and dy <0 hold, the packet is transferred to W (step S46). When dx <0 and dy <0 do not hold, the packet is transferred to S (step S47).

  Next, when Dy · dx> dy · Dx does not hold in step S40, it is determined whether dx> 0 and dy> 0 hold (step S48). When dx> 0 and dy> 0 hold, the packet is transferred to N (step S49). When dx> 0 and dy> 0 do not hold, it is determined whether dx <0 and dy> 0 hold (step S50). When dx <0 and dy> 0 hold, the packet is transferred to W (step S51). When dx <0 and dy> 0 do not hold, it is determined whether dx <0 and dy <0 hold (step S52). When dx <0 and dy <0 hold, the packet is transferred to S (step S53). When dx <0 and dy <0 do not hold, the packet is transferred to E (step S54).

  Through the above processing, the packet input to the memory node is transferred to an adjacent memory node so that the packet advances along a straight line connecting the transmission destination node (to) and the transmission source node (from).

  FIG. 11D shows a specific example of the packet transfer process. In the transfer algorithm 2, transfer is performed so that the packet proceeds along a straight line connecting the transmission destination node (to) and the transmission source node (from). For example, in case 3, dx <0 and dy> 0, and the process proceeds to W or N so as not to be as far as possible from the straight line connecting the transmission destination node (to) and the transmission source node (from). In Case 4, dx> 0 and dy> 0, and the process proceeds to N or E so as not to be as far as possible from the straight line connecting the transmission destination node (to) and the transmission source node (from).

  With reference to FIGS. 12A and 12B, the superiority of the transfer algorithm 2 over the transfer algorithm 1 is shown.

  FIG. 12A and FIG. 12B illustrate the packet transfer process when packets addressed to the destination node (to) are transmitted from the two source nodes (from1) and (from2), respectively. FIG. 12A shows a transfer process by the transfer algorithm 1, and FIG. 12B shows a transfer process by the transfer algorithm 2.

  In the transfer algorithm 1 shown in FIG. 12A, a confluence waiting jam (indicated by jam) occurs at a point where two packets join. On the other hand, no congestion has occurred in the transfer algorithm 2 shown in FIG. 12B. Therefore, the transfer algorithm 2 can reduce the probability of occurrence of traffic congestion than the transfer algorithm 1.

  As described above, according to the second embodiment, it is possible to reduce the probability of occurrence of traffic congestion as compared with the first embodiment, and it is possible to maintain the response speed even when many clients are connected simultaneously. Can be provided.

  Further, as in the first embodiment, since the logical address and the physical address of the memory node coincide with each other, an efficient data transfer method that does not require each memory node to manage the routing table can be realized. For this reason, the expandability of the storage apparatus is excellent, that is, it is possible to easily add a memory node to the storage apparatus. Other configurations and effects are the same as those of the first embodiment described above.

[Third Embodiment]
A storage apparatus according to the third embodiment will be described. The third embodiment has a transfer algorithm 3 different from the transfer algorithms 1 and 2 of the first and second embodiments. In the transfer algorithm 3, a memory node as a transfer destination is determined based on four pieces of information including a transmission destination address and a transmission source address included in the packet, an address of the own node, and output port occupation information of the own node. Since this storage device has the same configuration as that shown in FIGS. 1 and 2, description thereof is omitted.

[1] Storage device transfer algorithm 3
FIG. 13 is a diagram showing a transfer algorithm 3 in the storage apparatus of the third embodiment. Referring to FIG. 13, an example of a method for determining a transfer destination adjacent node based on four pieces of information, that is, a transmission destination address and a transmission source address of a packet, an address of the own node, and output port occupation information of the own node is shown. . This is referred to as transfer algorithm 3.

  As shown in FIG. 13, it is determined in which of two areas separated by a straight line connecting the transmission destination node (to) and the transmission source node (from) the own node (PP) exists and assigned to that area. Of the two directions, the packet is transferred to the adjacent node in the direction determined by the occupation information of the output port of the own node. A first priority direction and a second priority direction are assigned to each area. The second priority direction is selected when the output port buffer in the first priority direction is occupied by another packet.

  A specific example of a packet transfer process according to transfer algorithm 3 will be described with reference to FIGS. 13 and 14A to 14D.

As shown in FIG. 14A, the address of the destination node (to) is (x _to , y _to ), the address of the source node (from) is (x _from , y _from ), and the address (PP) of its own node is ( x _pp , y _pp ). Further, dx = x _to −x _from , dy = y _to −y _from , Dx = x _pp −x _from , and Dy = y _pp −y _from .

  FIG. 13 illustrates a case where dx> 0 and dy> 0. When Dy is compared with (dy / dx) · Dx, an area where Dy is larger than (dy / dx) · Dx is an A area, and the other area is a B area. In the transfer, E is assigned to the first priority direction of the A area, and N is assigned to the first priority direction of the B area. Further, N is assigned to the second priority direction (bypass direction) of the A area, and E is assigned to the second priority direction of the B area. For example, as shown in the figure, when a memory node that transfers a packet belongs to the B region, if the output port buffer in the N direction, which is the first priority direction, is occupied by another packet, the memory node Forward the packet in the E direction, which is the second priority direction.

  FIG. 14C shows a flow of the transfer algorithm 3 in the storage apparatus. The transfer algorithm 3 is stored in the packet controller 15 and is executed by the packet controller 15. As shown in FIG. 14B, an output port buffer for outputting a packet to N is OPBN, an output port buffer for outputting a packet to E is OPBE, and an output port buffer for outputting a packet to W is provided. An output port buffer for outputting a packet to OPBW and S is OPBS.

  Information about whether these output port buffers OPBN, OPBE, OPBW, and OPBS are free (can store a packet) or is occupied by a packet (output port occupation information) is stored in the packet controller 15 as follows. The The packet controller 15 has buffer occupancy flag bits corresponding to the total number of output port buffers and input port buffers. The packet controller 15 sets the buffer occupancy flag bit corresponding to the buffer to “1” when a packet is written to the output port buffer, and when the packet is output from the output port buffer, the buffer occupancy flag corresponding to the buffer. Set the bit to “0”. The packet controller 15 can determine whether or not the corresponding output port buffer is empty (whether or not it is occupied by the packet) by evaluating the buffer occupancy flag bit.

First, the packet controller 15 calculates dx = x _to −x _from , dy = y _to −y _from , Dx = x _pp −x _from , and Dy = y _pp −y _from (step S61). Subsequently, the packet controller 15 determines whether dx is 0 (step S62). When dx is 0, it is determined whether or not y _to > y _pp is satisfied (step S63). When y _to > y _pp holds, the packet is transferred to N (step S64). On the other hand, when y _to > y _pp does not hold, the packet is transferred to S (step S65).

Next, when dx is not 0 in step S62, the packet controller 15 determines whether dy is 0 (step S66). When dy is 0, it is determined whether x _to > x _pp is satisfied (step S67). When x _to > x _pp holds, the packet is transferred to E (step S68). On the other hand, when x _to > x _pp does not hold, the packet is transferred to W (step S69).

  Next, when dy is not 0 in step S66, that is, when dx and dy are not 0, the packet controller 15 determines whether or not Dy · dx> dy · Dx is satisfied (step S70). When Dy · dx> dy · Dx holds, it is determined whether dx> 0 and dy> 0 hold (step S71). When dx> 0 and dy> 0 hold, it is determined whether or not the output port buffer OPBE to be output to E is empty, that is, is not occupied by another packet (step S72). When the output port buffer OPBE is empty, the packet is transferred to E (step S73). On the other hand, when the output port buffer OPBE is not empty, the packet is transferred to N (step S74).

  Next, when dx> 0 and dy> 0 do not hold in step S71, it is determined whether dx <0 and dy> 0 hold (step S75). When dx <0 and dy> 0 hold, it is determined whether or not the output port buffer OPBN to be output to N is empty (step S76). When the output port buffer OPBN is empty, the packet is transferred to N (step S77). On the other hand, when the output port buffer OPBN is not empty, the packet is transferred to W (step S78).

  Next, when dx <0 and dy> 0 do not hold in step S75, it is determined whether dx <0 and dy <0 hold (step S79). When dx <0 and dy <0 hold, it is determined whether or not the output port buffer OPBW to be output to W is empty (step S80). When the output port buffer OPBW is empty, the packet is transferred to W (step S81). On the other hand, when the output port buffer OPBW is not empty, the packet is transferred to S (step S82).

  Next, when dx <0 and dy <0 are not satisfied in step S79, it is determined whether or not the output port buffer OPBS output to S is empty (step S83). When the output port buffer OPBS is empty, the packet is transferred to S (step S84). On the other hand, when the output port buffer OPBS is not empty, the packet is transferred to E (step S85).

  Next, when Dy · dx> dy · Dx does not hold in step S70, it is determined whether dx> 0 and dy> 0 hold (step S86). When dx> 0 and dy> 0 hold, it is determined whether or not the output port buffer OPBN that outputs to N is empty (step S87). When the output port buffer OPBN is empty, the packet is transferred to N (step S88). On the other hand, when the output port buffer OPBN is not empty, the packet is transferred to E (step S89).

  Next, when dx> 0 and dy> 0 do not hold in step S86, it is determined whether dx <0 and dy> 0 hold (step S90). When dx <0 and dy> 0 hold, it is determined whether or not the output port buffer OPBW to be output to W is empty (step S91). When the output port buffer OPBW is empty, the packet is transferred to W (step S92). On the other hand, when the output port buffer OPBW is not empty, the packet is transferred to N (step S93).

  Next, when dx <0 and dy> 0 do not hold in step S90, it is determined whether dx <0 and dy <0 hold (step S94). When dx <0 and dy <0 hold, it is determined whether or not the output port buffer OPBS to be output to S is empty (step S95). When the output port buffer OPBS is empty, the packet is transferred to S (step S96). On the other hand, when the output port buffer OPBS is not empty, the packet is transferred to W (step S97).

  Next, when dx <0 and dy <0 do not hold in step S94, it is determined whether or not the output port buffer OPBE to be output to E is empty (step S98). When the output port buffer OPBE is empty, the packet is transferred to E (step S99). On the other hand, when the output port buffer OPBE is not empty, the packet is transferred to S (step S100).

  By the above processing, if there is a traffic jam in the first priority direction, the destination node (to) and the source node (from) while avoiding the traffic jam by forwarding the packet in the second priority direction A packet is transferred to an adjacent memory node along a straight line connecting the two.

  FIG. 14D shows a specific example of the packet transfer process. Case 1 is an example in which no traffic jam occurs, and packets are transferred along a straight line connecting the transmission destination node (to) and the transmission source node (from). Cases 2 and 3 show examples of avoiding traffic jams. In Case 2, when trying to transfer a packet to N, there is a traffic jam in the N direction. For this reason, as shown in Case 3, the packet is transferred to E to avoid congestion. Cases 4 and 5 show other examples for avoiding traffic jams. In Case 4, there is a traffic jam in the E direction when trying to transfer a packet to E. For this reason, as shown in Case 5, the packet is transferred to N to avoid congestion.

  In the transfer algorithm 3, when a traffic jam occurs in the first priority direction, the traffic jam can be avoided by transferring the packet in the second priority direction. At this time, the number of transfers until the packet arrives at the transmission destination node does not change as compared with the case where no traffic jam occurs.

[2] Storage device transfer algorithm 4
FIG. 15 is a diagram showing the transfer algorithm 4 in the storage apparatus. Referring to FIG. 15, another method of determining a transfer destination adjacent node based on four pieces of information, that is, a transmission destination address and a transmission source address of a packet, an address of the own node, and output port occupation information of the own node. An example is shown. This is referred to as transfer algorithm 4.

  As shown in FIG. 15, four regions defined by a straight line connecting the transmission destination node (to) and the transmission source node (from) and a straight line orthogonal to the position of the transmission destination node (to) are defined. Of these four areas, it is determined in which area the own node (PP) exists, and among the two directions of the first priority and the second priority assigned to the area where the own node (PP) exists, The packet is forwarded to the adjacent node in the direction determined by the occupation information of the output port of the own node (PP).

  The superiority of algorithm 4 compared to algorithm 3 is as follows. For algorithm 3, it is necessary to provide a restriction that prohibits packets from being transferred outside the area defined by the rectangle whose diagonal line is the line connecting the destination node (to) and the source node (from). . On the other hand, it is not necessary to provide a restriction for algorithm 4.

[3] Storage device transfer algorithm 5
FIG. 16 is a diagram showing the transfer algorithm 5 in the storage apparatus. Referring to FIG. 16, another method for determining a transfer destination adjacent node based on four pieces of information, that is, a transmission destination address and a transmission source address of a packet, an address of the own node, and output port occupation information of the own node. Two examples are shown. This is referred to as transfer algorithm 5.

  As shown in FIG. 16, a straight line connecting the transmission destination node (to) and the transmission source node (from), a straight line orthogonal to the position of the transmission destination node (to), and x at the position of the transmission destination node (to). Eight regions defined by four straight lines of two straight lines extending in the direction and y direction are defined. Of these 8 areas, it is determined in which area the own node (PP) exists, and among the two directions of the first priority and the second priority assigned to the area where the own node (PP) exists, The packet is forwarded to the adjacent node in the direction determined by the occupation information of the output port of the own node (PP). The two straight lines extending in the x direction and the y direction at the position of the transmission destination node (to) include, for example, two straight lines passing through the transmission destination node (to) and along the arrangement direction of the memory nodes.

  The superiority of algorithm 5 compared to algorithm 4 is as follows. In Algorithm 4, when a packet is transferred in the second priority direction (bypass direction) outside the area defined by a rectangle whose diagonal line is a straight line connecting the transmission destination node (to) and the transmission source node (from), The number of transfers until the packet reaches the destination node (to) increases by at least one. On the other hand, in the case of algorithm 5, there is no increase in the number of transfers.

  As described above, according to the third embodiment, it is possible to reduce the occurrence probability of traffic congestion as compared with the first and second embodiments, and to maintain the response speed even when many clients are connected simultaneously. It is possible to provide a storage device that can be used.

  Further, as in the first embodiment, since the logical address and the physical address of the memory node coincide with each other, an efficient data transfer method that does not require each memory node to manage the routing table can be realized. For this reason, the expandability of the storage apparatus is excellent, that is, it is possible to easily add a memory node to the storage apparatus. Other configurations and effects are the same as those of the first embodiment described above.

[Fourth Embodiment]
In the fourth embodiment, in a storage system including a storage apparatus having a plurality of memory nodes having a transfer function and a control computer connected to the storage apparatus, no congestion occurs at the time of packet transfer, that is, no congestion occurs. The operating conditions will be described.

[1] Configuration of Storage System FIG. 17 is a diagram showing the configuration of the storage system of the fourth embodiment.

  As illustrated, the storage system includes a storage device 10 including a plurality of memory nodes 11 and a plurality of control computers 41 connected to the storage device 10. As shown in FIG. 1, the storage apparatus 10 has a configuration in which a plurality of memory nodes having a data transfer function are connected to each other. A control computer 41 is connected to the memory node 11 arranged on the outer periphery of the storage apparatus 10.

[2] Storage System Congestion-Free Operation Conditions With reference to FIG. 17, a framework for packet transfer simulation in the storage system will be described. The process until the request packet transmitted from the control computer 41 connected to the storage apparatus 10 reaches the memory node of the destination address, and the data packet returned from the target memory node arrives at the original control computer 41. Simulate the process up to. As a packet transfer algorithm, the routing function-added routing algorithm (transfer algorithm 3) described with reference to FIG. 13 is used.

The number of memory nodes of the storage apparatus 10 is N _node = Nx × Ny, and the number of control computers connected to the storage apparatus 10 is Nc. The time for one packet transfer between the memory nodes is defined as a unit step time. Let Rr be the probability that the control computer 41 transmits a request packet per unit step time. Rr is 1 at the maximum. _Assume that the bypass transfer occurrence rate is R _bypass and the traffic jam occurrence rate is R _jam .

The bypass transfer occurrence rate R _bypass indicates a probability that traffic congestion is avoided by bypass transfer when the memory node attempts to transfer one packet. On the other hand, when the memory node tries to transfer one packet, the jam occurrence rate R _jam is not only the output port buffer in the first priority direction but also the output port buffer in the second priority direction is occupied by another packet. Therefore, it indicates the probability that the packet transfer could not be performed and the traffic jam occurred and the packet stayed. Also, _let R _{load be} the load coefficient. The load coefficient R _load is an average of the number of packets existing in one memory node.

FIG. 18 shows the relationship between the bypass transfer occurrence rate R _bypass and the load factor R _{load and} the relationship between the traffic jam occurrence rate R _jam and the load factor R _load . Many plots show the results when the number of memory nodes is N _node , the number of control computers is Nc, and the probability Rr is variously changed.

The traffic jam occurrence rate R _jam is almost zero when the load coefficient R _load is smaller than 0.2, and it can be seen that the traffic congestion free system operating condition is the load coefficient R _load <0.2. The congestion occurrence rate R _jam starts to increase when the load coefficient R _load becomes 0.2 or more. However, even if the load coefficient R _load = 2, the traffic jam occurrence rate R _jam is about 0.05, which _poses no practical problem. On the other hand, the jam occurrence rate R _jam increases rapidly when the load coefficient R _load is larger than 2. This is because when the load coefficient R _load is greater than 2, a hang-up phenomenon occurs in which traffic congestion occurs in a chained manner.

The bypass transfer occurrence rate R _bypass rises one digit earlier than the traffic jam occurrence rate R _{jam with} respect to the increase of the load coefficient R _load . In other words, the bypass transfer algorithm has an effect of improving the congestion occurrence requirement by about one digit.

Next, based on the simulation results, the relationship between the number of control computers and the number of memory nodes is determined so that congestion free or occurrence of traffic congestion is not a practical problem. Let the average packet stay step time be _Savg . Average packet staying step time S _avg, the data control computer 41 is returned from the step time S _request and the memory node of interest from sending the request packet to the request packet reaches the memory node of the destination address This is the sum of the step times S _dat until the packet arrives at the original control computer. The step times S _request and S _dat are when no traffic jam occurs (Nx / 2 + Ny / 2). Therefore, the average packet stay step time S _avg is (Nx + Ny), which is approximately 2√N _node .

On the other hand, the total number of request packets that the control computer 41 transmits per step time is Nc × Rr. Therefore, the total number of packets existing in the storage system is Nc × Rr × 2√N _node on average. Therefore, R _load ˜Nc × Rr × 2√N _node / N _node = Nc × Rr × 2 / √N _node . Here, when the congestion occurrence requirement is R _load ^limit , a preferable system operation condition is R _load <R _load ^limit . Accordingly, the maximum number of control computers Nc ^{max at which} no congestion or no occurrence of traffic congestion is a practical problem is Nc ^max <R _load ^limit × √N _node / (Rr × 2). Since Rr is 1 at maximum, the stricter condition is Nc ^max <R _load ^limit × √N _node / 2.

As described above, the operation condition for free of occurrence of traffic congestion at the time of packet transfer is R _load ^limit = 0.2, and the operation condition for which traffic congestion does not cause a practical problem is R _load ^limit = 2. Therefore, the preferred maximum number of control computers Nc ^max is Nc ^max <√N _node , more preferably Nc ^max <0.1 × √N _node . Therefore, the relationship Nc <√N _node is established. Other configurations and effects are the same as those of the first embodiment described above.

[Fifth Embodiment]
The fifth embodiment includes a storage device in which a plurality of memory nodes having a transfer function shown in FIG. 1 are connected to each other, and each memory node has a distributed processing function.

[1] Configuration of Storage Device FIG. 19 is a diagram showing a configuration of a storage device according to the fifth embodiment.

  As illustrated, the storage apparatus includes the storage apparatus 10 including the plurality of memory nodes 11 illustrated in FIG. 1, and each memory node 11 includes an address converter 42 and has a distributed processing function.

[2] Distributed Processing Function of Storage Device Here, with reference to FIG. 19, the distributed processing function will be described by taking an AND search process for finding a URL including both the apple and orange keywords as an example.

  Here, it is assumed that the memory nodes (1, 3) and (4, 3) hold apple and orange transposed files, respectively. The transposed file is an index file created for each keyword. For example, the transposed file corresponding to the keyword apple stores a list of all URLs including apple.

  The gateway server transmits an AND search command to the memory node that manages the transposed files of apple and orange. The memory node that manages the transposed file maps the contents of the transposed file to key-value type data. Here, key and value are the URL and the keyword of the transposed file, respectively.

  Each memory node converts the key into an address by the address converter 42 for each record of the key-value type data held by itself, and transmits a packet including the value at the address. The address converter 42 may calculate an address from the key itself according to a certain rule, or may inquire a server having a function of converting the key into an address to obtain an address corresponding to the key.

  For example, the address converter 42 calculates an address from the key according to the following addressing rule.

Address = hash (key) mod N
Here, hash () is a cryptographic hash function, N is the number of memory nodes in the storage device 10, mod N denotes the remainder operation modulo N.

  Further, for example, the address converter 42 inquires of another server about the address corresponding to the key, and the other server converts the key into the address by consistent hashing and returns the reply to the original address converter 42.

  In the example shown in FIG. 19, it is assumed that both the apple and orange transposition files include URL1, and URL1 is converted to the address (2, 1). In this case, the memory node (2, 1) plays a role of determining whether URL1 satisfies the AND search expression. Two packets whose values are apple and orange arrive at the memory node (2, 1). For this reason, the memory node (2, 1) knows that URL1 satisfies the AND search condition.

  The memory node (2, 1) notifies the gateway server that issued the query to that effect. A large number of memory nodes make the same determination for all the URLs described in the transposed file, so that the gateway server that issued the query can obtain a list of URLs that satisfy the AND search condition.

When performing the above AND search operation with a single memory node, it is necessary to perform the operation indicated by the following code.

Here, the numbers of URLs stored in the transposed files of apple and orange are N _apple and N _orange , respectively, and == indicates a matching operation. The memory node needs to repeat the matching operation only N _apple × N _orange / 2 times. In the storage apparatus of the present embodiment, this coincidence operation can be carried out by being distributed among a large number of memory nodes.

  As described above, according to the fifth embodiment, not only the storage function described in the first embodiment but also a distributed processing (distributed computing) function can be provided. Other configurations and effects are the same as those of the first embodiment described above.

[Sixth Embodiment]
In the sixth embodiment, a storage apparatus including a plurality of memory nodes having a transfer function, and an input / output port (hereinafter, referred to as an adjacent node) in which a part of the memory nodes of the storage apparatus is connected to an adjacent memory node (hereinafter referred to as an adjacent node). There are input / output ports (hereinafter referred to as non-adjacent ports) other than adjacent ports, and the some memory nodes are connected to a control computer or non-adjacent memory nodes (hereinafter referred to as non-adjacent nodes) by the non-adjacent ports. Have a configuration.

[1] Configuration of Storage System FIG. 20 is a diagram showing the configuration of the storage system of the sixth embodiment.

  As illustrated, the storage system includes a storage device 10 including a plurality of memory nodes 11 and a plurality of gateway servers 21A and 21B connected to the storage device 10. As shown in FIG. 1, the storage apparatus 10 has a configuration in which a plurality of memory nodes having a data transfer function are connected to each other. A gateway server 21A is connected to the memory nodes (1, 4) arranged on the outer periphery of the storage apparatus 10 via an adapter 22A. The memory nodes (3, 5) arranged in the central part of the storage apparatus 10 have input / output ports (non-adjacent ports) other than the input / output ports connected to the adjacent nodes, and the adapter 22B is connected to this non-adjacent port. The gateway server 21B is connected via

[2] Packet Transfer of Storage System The effect of introducing a memory node having an input / output port (non-adjacent port) other than the input / output port connected to the adjacent node will be described with reference to FIG. Here, the number of memory nodes in the x direction of the storage apparatus 10 is Nx, and the number of memory nodes in the y direction is Ny.

  When the connection destination of the gateway server is limited to the memory nodes arranged on the outer periphery of the storage apparatus 10, the average number of packet transfers from the gateway server to the target memory node is Nx / 2 + Ny / 2. .

  On the other hand, as shown in FIG. 20, the number of input / output ports of the memory node (3, 5) located in the central part of the storage apparatus 10 is increased, and the memory node (3 , 5) is connected to the gateway server 21B. Then, the average number of packet transfers from the gateway server 21B to the target memory node is Nx / 4 + Ny / 4.

  As described above, when the number of input / output ports of some memory nodes in the storage apparatus 10 is increased and the memory node and the gateway server are connected by the added input / output ports, the average transfer until the packet reaches the target memory node. The number of times can be reduced.

  Further, as shown in FIG. 21, a plurality of memory nodes connected to the gateway server by the added input / output port may be arranged in the storage apparatus 10. That is, the memory node (4, 7) has an input / output port (non-adjacent port) other than the input / output port connected to the adjacent node, and this non-adjacent port is connected to the gateway server 21C via the adapter 22C. Yes.

  Furthermore, the plurality of gateway servers 21A, 21B, 21C, and 21D connected to the memory nodes in the storage apparatus 10 may be connected by a wiring 23 that is different from the wiring that connects the servers and the memory nodes. Further, the adapter 22E may be connected between the gateway servers, for example, between the gateway servers 21A and 21B. Thus, packet transfer can be performed between the gateway servers and between the gateway server and the memory node using the wiring 23. For example, the packet may be transferred in the order of gateway server 21A → gateway server 21B → gateway server 21C → memory node (4, 7). Alternatively, the packet may be transferred in the order of gateway server 21A → gateway server 21D → memory node (5, 4).

  As described above, according to the sixth embodiment, when packet transfer is performed between the gateway server and the memory node in the storage apparatus, the transfer time can be shortened. Other configurations and effects are the same as those of the first embodiment described above.

[Seventh Embodiment]
In the seventh embodiment, a storage device having a plurality of memory nodes having a transfer function, and a part of the storage device have input / output ports (non-adjacent ports) other than input / output ports connected to adjacent nodes. And a part of the memory nodes is connected to a control computer or a memory node that is not adjacent (non-adjacent node) by the non-adjacent port.

[1] Configuration of Storage System FIG. 22 is a diagram showing the configuration of the storage system of the seventh embodiment.

  As illustrated, the storage system includes a storage apparatus 10 including a plurality of memory nodes 11 and a gateway server 21A connected to the storage apparatus 10. As shown in FIG. 1, the storage apparatus 10 has a configuration in which a plurality of memory nodes having a data transfer function are connected to each other. A gateway server 21A is connected to the memory nodes (7, 1) arranged on the outer periphery of the storage apparatus 10 via an adapter 22A.

  Memory nodes (3, 3), (3, 8), (8, 3), (8, 8) arranged inside the storage apparatus 10 are input / output ports other than the input / output ports connected to the adjacent nodes. (Non-adjacent port). Non-adjacent ports are connected by wiring 24. For example, between memory nodes (3, 3) and (3, 8), between memory nodes (3, 8) and (8, 8), between memory nodes (8, 8) and (8, 3), and memory nodes (8,3) and (3,3) are connected to each other by a wiring 24.

  A memory node connected to a non-adjacent node has an additional address determined by a relative physical position of memory nodes connected to the non-adjacent node, in addition to an address determined by a physical position. When a memory node connected to a non-adjacent node receives a packet destined for a memory node other than the self-node, the packet controller of the self-node will send an additional address of the self-node or a non-adjacent node other than the self-node. A packet output port is determined based on information including at least one of the additional addresses of the connected memory nodes.

[2] Packet Transfer in Storage System First, the influence of the connection between non-adjacent memory nodes (non-adjacent nodes) on the number of packet transfers in the storage apparatus 10 will be described. As described above, FIG. 22 shows a storage apparatus in which some non-adjacent nodes are interconnected by non-adjacent ports, and FIG. 22 shows a storage apparatus in which the number of input / output ports of all memory nodes is the same and there is no connection between non-adjacent nodes. It shows in FIG.

  In these storage apparatuses, consider a case in which a data packet is sent from the gateway server 21A at the address (7, 0) to the memory node (7, 8).

  When there is no connection between non-adjacent nodes (FIG. 23), the shortest data transfer path is gateway server 21A (7,0) → memory node (7,1) → (7,2) → (7,3) → (7, 4) → (7, 5) → (7, 6) → (7, 7) → (7, 8). Therefore, the total number of packet transfers (between the gateway server and the memory node and between the memory node and the memory node) when performing packet transfer from the gateway server 21A to the memory node (7, 8) is eight.

  On the other hand, when non-adjacent nodes are interconnected (FIG. 22), gateway server 21A → memory node (7, 1) → (7, 2) → (7, 3) → (8, 3) → (8 , 8) → (7, 8) is the shortest path, and the packet can be sent from the gateway server 21A to the destination memory node (7, 8) in six transfers.

  In this way, some non-adjacent memory nodes are connected by an additional input / output port, and packet transfer is performed using connection wiring between these input / output ports, so that packet transfer until reaching the destination node The number of times can be reduced and the transfer time can be shortened.

  However, as shown in FIG. 22, in a storage apparatus in which non-adjacent nodes are connected, the physical distance between the memory node and the memory node or between the memory node and the gateway server 21A and the distance in data transfer between them. Note that does not necessarily match.

  As described above, when a data packet is sent from the gateway server 21A at the address (7, 0) to the memory node (7, 8), the storage device (FIG. 23) in which there is no connection between non-adjacent nodes. The minimum total packet transfer count (8 times) is equal to the difference between the addresses (gateway server 21A (7, 0), memory node (7, 8)) determined by the physical location of the packet source and destination.

  On the other hand, as described above, in the storage apparatus of FIG. 22 in which non-adjacent nodes are connected, packet communication between the gateway server 21A (7, 0) and the destination memory node (7, 8) having the same physical address is performed. , 6 transfers can be performed. In this way, in a storage apparatus in which non-adjacent nodes are connected, the shortest physical distance and the shortest distance in data transfer are different, so the shortest transfer path of a packet is determined only by an address determined by the physical position. It is not possible.

  Therefore, in storage devices connected between non-adjacent nodes, addresses reflecting the relative physical positions of memory nodes connected to these non-adjacent nodes (hereinafter referred to as subaddresses) are connected to non-adjacent nodes. If the data transfer route / packet transfer destination is determined from both the sub-address and the address (hereinafter referred to as the main address) that is given to all memory nodes and reflects the physical location, it is given to all the memory nodes according to the algorithm described above. Packet transfer can be performed efficiently. In FIG. 22, the main address is displayed in parentheses and the sub address is displayed in square brackets. The memory nodes connected to the non-adjacent nodes indicate, for example, the memory nodes (3, 3), (3, 8), (8, 3), (8, 8) shown in FIG.

  However, as described above, in order to perform packet transfer efficiently, all the necessary address information is written in the header part of the packet, and the source address and destination address of the packet used for determining the transfer destination are set. It is necessary to rewrite as appropriate. These will be described next.

  FIG. 24 shows an example of address information of a header portion of a desired packet when packet transfer is performed in a storage system having a memory node connected to a non-adjacent node.

  Here, the final transmission destination node and the transmission source node mean a destination node that finally delivers the packet and a node that first creates and transmits the packet. On the other hand, the temporary transmission destination node address and the temporary transmission source node address are addresses used for determining the packet transfer destination in each memory node, and are rewritten in the packet communication process. The rewrite rule at this time will be described later.

  The type of temporary transmission destination / transmission source address is used to distinguish whether the address is a main address or a sub address. Further, the first relay node and the second relay node are memory nodes connected to non-adjacent nodes that should be routed at the time of packet transmission, and the memory node connected to the non-adjacent node closest to the transmission source node is the first relay node. The memory node connected to the non-adjacent node closest to the node and the final destination node is the second relay node.

  As shown in FIG. 22, in a system including a storage apparatus in which a non-adjacent memory node and a part of memory nodes are connected, when performing packet transfer via connection between non-adjacent nodes, the memory node that received the packet It is preferable to perform packet transfer destination determination, packet header correction, and the like in FIG.

(A) When a memory node that does not have an additional input / output port (non-adjacent port) receives a packet, the packet controller examines address information recorded in the header of the packet,
(i) If the address of the final destination node matches the address of its own node, packet transfer is not performed.

    (ii) If the address of the final destination node is different from the address of its own node, determine the transfer destination by referring to the temporary source node, temporary destination node, and the main address of its own node, and Send the packet.

(B) When the memory node connected to the non-adjacent node receives the packet, the packet controller examines the address information recorded in the header part of the packet,
(i) If the address of the own node matches the address of the final destination node, packet transfer is not performed.

(ii) The address of its own node is different from the address of the final destination node,
(1) If it matches the address of the first relay node, the temporary transmission destination and temporary transmission source addresses in the header of the packet are rewritten to the subaddress of the second relay node and the subaddress of the own node, respectively. Further, the transfer destination is determined with reference to the subaddress of the temporary transmission source node, temporary transmission destination node, and own node, and the packet is transmitted to the memory node connected to another non-adjacent node.

      (2) If it matches the address of the second relay node, the temporary transmission destination address is changed to the main address of the final transmission destination node, and the temporary transmission source address is changed to the main address of the own node. Further, the transfer destination is determined with reference to the main address of the temporary transmission source node, temporary transmission destination node, and own node, and the packet is transferred to another adjacent memory node.

      (3) When the temporary transmission destination address and the temporary transmission source address written in the header part of the packet are subaddresses, unlike the addresses of the first relay node and the second relay node, the temporary transmission source node, The transfer destination is determined with reference to the subaddress of the temporary transmission destination node and its own node, and the packet is transferred to the memory node connected to another non-adjacent node.

      (4) Unlike the addresses of the first relay node and the second relay node, and when the temporary transmission destination address and the temporary transmission source address are main addresses, the temporary transmission source node, the temporary transmission destination node, and the own node The transfer destination is determined with reference to the main address of the packet, and the packet is transferred to another adjacent memory node.

  Next, as an example of performing packet transmission using a connection between non-adjacent nodes together, a procedure for a client to write a file to the storage device via the gateway server 21A in the system shown in FIG. 22 will be described.

  FIG. 25 is a diagram showing a write operation in the storage system shown in FIG.

  First, the client transmits a file and a file ID to the gateway server 21A (see (1)). The file ID is an identifier that can uniquely identify a file.

  Next, the gateway server 21A divides the file into data of a prescribed size and assigns a divided data ID to each divided data. Further, the file ID and the divided data ID are written in the file table. The divided data ID is an identifier that can uniquely identify the divided data (see (2)).

  Next, the gateway server 21A determines an address of a write destination memory node (hereinafter referred to as a write node) to write the divided data based on the information of the divided data ID (in FIG. 22, the memory node of the main address (7, 8)). ).

Next, in the following procedure, when a packet is transferred from the gateway server 21A to the writing node, a route that minimizes the communication time (number of packet transfers) is obtained:
1. Each of a memory node (first relay node) connected to a non-adjacent node closest to the gateway server 21A and a memory node (second relay node) connected to a non-adjacent node closest to the write node Look up the address. When writing to (7, 8) in FIG. 22, the memory nodes connected to the closest non-adjacent nodes of the gateway server 21A and the writing node are the memories of the main addresses (8, 3) and (8, 8). It is a node.

  2. Find the shortest path that includes connections between non-adjacent nodes and the shortest path that does not. When writing to the memory node (7, 8) in FIG. 22, the former is the gateway server 21A → memory node (7,1) → (7,2) → (7,3) → (8,3) ([2 , 1]) → (8,8) ([2,2]) → (7,8), the latter being the gateway server 21A → memory node (7,1) → (7,2) → (7,8). 3) → (7,4) → (7,5) → (7,6) → (7,7) → (7,8).

  3. The route between memory nodes connected only to adjacent nodes is the main address, and the route between memory nodes connected to non-adjacent nodes is based on sub-addresses when packets are sent on each route. Calculate the number of transfers that occur. When writing to the memory node (7, 8) in FIG. 22, the number of transfers of the shortest path through the memory node connected to the non-adjacent node is 6, and the shortest path not through the memory node connected to the non-adjacent node Then, the number of transfers is eight.

  4. Determine the route with the smaller number of packet transfers as the default route (see (3)). Hereinafter, a case will be described in which a route including a connection between non-adjacent nodes has a smaller number of packet transfers than a route not including the connection.

  The gateway server 21A creates a write packet in which a header portion composed of address data, a write command, etc. is added to the write data. The temporary transmission destination of the address information recorded in the packet header is the main address of the first relay node ((8, 3) in FIG. 22), and the temporary transmission source is the main address of the packet transmission source (in FIG. 22). (7, 0)).

  Transfer the created write packet from the gateway server 21A to the memory node connected to the gateway server 21A (see (4)).

  In accordance with the algorithm described above, write packets are transferred between the memory nodes until the temporary destination memory node (the first relay node of the main address (8, 3) in FIG. 22) is reached (each memory node is temporarily transmitted). The transfer destination is determined by referring to the destination, the temporary transmission source, and the main address of the own node) (see (5)).

  The first relay node that has received the write packet reads the header of the packet, and since the final destination node of the packet is another memory node and the first relay node is its own node, 2 The sub-address of the relay node ([2, 2] in FIG. 22) and the temporary transmission source are changed to the sub-address of the own node ([2, 1] in FIG. 22) (see (6)). Further, the packet is transferred to the memory node connected to the adjacent non-adjacent node (see (7)).

  In accordance with the algorithm described above, write packets are transferred between memory nodes connected to non-adjacent nodes until reaching the second relay node which is a temporary transmission destination memory node (each memory node is a temporary transmission destination, temporary transmission The forwarding destination is determined by referring to the subaddress of the original node and the own node).

  The second relay node that has received the write packet reads the header part of the packet, and since the final transmission destination is another memory node and the second relay node is its own node, the temporary transmission destination is the final transmission destination node. The main address (write node (7, 8) in FIG. 22) and the temporary transmission source are changed to the main address of the own node ((8, 8) in FIG. 22) (see (8)). Further, the write packet is transferred to the adjacent memory node.

  With reference to the main addresses of the temporary transmission source node, temporary transmission destination node, and own node, the transfer is repeated between the memory nodes by the algorithm described above, and the write node (the main address (7, The write packet arrives at 8)) (see (9)).

  The writing node that has received the packet does not transfer the packet because the final transmission destination is its own node, and writes address information, write data, and the like of the header portion to the memory 16 of its own node (see (10)). After that, a write completion report packet is created with the destination and relay destination reversed. In this packet, the source node is the write node (7, 8), the final destination node is (7, 0) of the gateway server 21A, and the first relay node is the main address (8, 8) (subaddress [2, 2). 2]) and the second relay node are nodes of the main address (8, 3) (sub-address [2, 1]). Thereafter, the write completion report packet is returned to the gateway server 21A in the same procedure as described above (see (11)). After all the divided data has been written, the gateway server 21A reports the end of file writing to the client.

  In the above-described example, a client is connected to a storage apparatus having an input / output port (non-adjacent port) other than an input / output port connected to an adjacent node and having memory nodes in which non-adjacent memory nodes are interconnected by non-adjacent ports. Explained the procedure for writing files, but in order to read / erase data written to the memory node and to check the free space of the memory, etc., a read command / erase command and a free space return command are sent from the gateway server to the storage device. Also in the case of sending to the memory node, the packet transfer time can be shortened by performing communication through the connection between the non-adjacent nodes. Furthermore, even when the read data or inquiry results are sent from the memory node to the gateway server, the packet transfer time can be shortened by performing communication via the connection between the non-adjacent nodes.

  As described above, it is possible to reduce the average number of packet transfers by performing data transmission through connections between non-adjacent nodes. Note that the memory nodes connected to the non-adjacent nodes by the additional input / output ports (non-adjacent ports) are preferably arranged uniformly distributed in the storage apparatus.

  For example, in the storage apparatus shown in FIG. 22 composed of 10 × 10 memory nodes, the area is divided into four areas including 5 × 5 memory nodes (the boundary is indicated by a dotted line) and positioned at the center of each area. The memory nodes (3, 3), (8, 3), (3, 8), and (8, 8) are connected to each other by extension ports (non-adjacent ports). In this case, there is a memory node having one additional input / output port (non-adjacent port) in any 5 × 5 memory node block.

  In this way, the storage device is divided into multiple areas where the number of included memory nodes is the same, and the central memory node is interconnected as a memory node with an additional input / output port. Memory nodes with ports can be uniformly distributed in the storage apparatus.

  For example, when a storage apparatus composed of nodes of al × bm is divided into a × b areas of l × m, and memory nodes located at the center of each area are connected by an additional input / output port, (cl + d , Em + f) (a ≧ c ≧ 0, b ≧ e ≧ 0, l ≧ d ≧ 0, m ≧ f ≧ 0) with an additional input / output port located at the center of each region. The memory node address is represented by (cl + round (l / 2, 0), em + round (m / 2, 0)). (Round (A, 0) is a function that rounds off the decimal point of A).

  In the storage apparatus, a problem caused by connection between some memory nodes that are not adjacent to each other is that packet communication congestion is likely to occur as the number of packets that are communicated via the connection increases.

  In contrast, by increasing the packet communication speed between non-adjacent memory nodes as compared to between adjacent memory nodes, this packet transfer congestion can be alleviated. However, in this case, the gateway server calculates the total packet communication time for the shortest route through the connection between non-adjacent nodes and the shortest route through the connection between non-adjacent nodes, which is performed when determining the default route for packet transfer. It is necessary to pay attention to. The total packet communication time is equal to the product of the communication time between memory nodes and the number of packet transfers. Therefore, when the data communication speeds between adjacent memory nodes and non-adjacent memory nodes are equal, as described above, the data communication times can be compared with the number of transfers of each path. However, if the packet transmission rates are different, the total packet communication time cannot be compared only by the number of transfers.

  By referring to the product of the reciprocal of the communication speed (speed = distance / time, so this is proportional to the communication time) and the number of transfers, the path via the connection between non-adjacent nodes and the connection between non-adjacent nodes It is possible to accurately compare the total packet communication time of routes that do not pass through. However, when the connection between adjacent nodes and the connection between non-adjacent nodes are mixed in the packet communication path as in the example described above, this calculation becomes complicated.

  In a storage apparatus in which the packet communication speed is different between adjacent nodes and non-adjacent nodes, one method is to consider the address step to be given inversely proportional to the packet communication speed and regard the address difference as the number of transfers.

  FIG. 26 shows an example of address assignment when the packet communication speed is 10 times higher between non-adjacent nodes than between adjacent nodes. In FIG. 26, the memory nodes connected to the non-adjacent nodes reflect sub-addresses so that they differ one by one between a certain memory node and a non-adjacent node connected to the memory node while reflecting their physical positions. Is given (indicated by square brackets). Further, all the memory nodes shown in FIG. 26 are given 10 different main addresses between adjacent memory nodes (indicated by parentheses) while reflecting the physical positional relationship with each other.

  In such a storage device, when packet transmission is performed between memory nodes connected to non-adjacent nodes, the sub-address difference of each memory node is calculated, and when packet transmission is performed between adjacent memory nodes, the difference between main addresses is calculated. And consider them as the number of transfers. If the rule for calculating the number of transfers is determined in this way, the packet communication time can be estimated and compared only by the number of transfers even when the connection speed is different between adjacent nodes and non-adjacent nodes.

  As described above, according to the seventh embodiment, when packet transfer is performed between the gateway server and the memory node in the storage device, the transfer time can be shortened. Other configurations and effects are the same as those of the first embodiment described above.

[Eighth Embodiment]
In the eighth embodiment, a storage system will be described in which a switching relay is added between the gateway server and the storage apparatus in order to reduce the average number of packet transfers.

[1] Configuration of Storage System FIG. 27 is a diagram showing the configuration of the storage system of the eighth embodiment.

  As illustrated, the storage system includes a storage device 10 including a plurality of memory nodes 11, a switching relay 81 connected to the storage device 10, and a gateway server 21A connected to the switching relay 81.

  As shown in FIG. 1, the storage apparatus 10 has a configuration in which a plurality of memory nodes having a data transfer function are connected to each other. The switching relay 81 includes memory nodes (1, 1), (1, 2), (1, 3), (1, 4), (1, 1) arranged on one end side (the left end in FIG. 27) of the storage device 10. 5), (1,6), (1,7), (1,8), and (1,9) are all connected. The gateway server 21A is connected to the switching relay 81 via the adapter 22A.

  The switching relay 81 transfers the received packet to the designated destination node according to the address information recorded in the header part of the packet. An address different from that of the memory node in the storage apparatus is given to the switching relay 81 (indicated by square brackets in FIG. 27).

[2] Packet Transfer of Storage System A procedure for transferring packets in the storage system of the eighth embodiment will be described.

  The packet sent out from the gateway server 21A enters the switching relay 81 through the adapter 22A. Packets entering the switching relay 81 are connected to the switching relay 81 and are connected to the memory nodes (1, 1), (1, 2), (1, 3), (1, 4), (1, 5), ( 1, 6), (1, 7), (1, 8), (1, 9), and then transferred to the memory node of the destination address.

  On the contrary, the packet transmitted from the memory node in the storage apparatus 10 is sent to any one of the memory nodes (1, 1) to (1, 9), via the switching relay 81 and the adapter 22A. It is transferred to the gateway server 21A.

  In the system shown in FIG. 27, a packet is sent from the gateway server 21A to the memory node (5, 9) via the switching relay 81 and the memory node (1, 9) as follows. The shortest path is switching relay 81 → memory node (1,9) → (2,9) → (3,9) → (4,9) → (5,9), and the number of transfers in the storage apparatus 10 is four times. It is.

  On the other hand, as shown in FIG. 28, in a system in which the gateway server 21A is connected to the memory node (1, 4) of the storage apparatus 10 without passing through the switching relay 81, the gateway server 21A to the memory node (5, 9 ) To send a packet: Gateway server 21A → memory node (1,4) → (1,5) → (1,6) → (1,7) → (1,8) → (1,9) → (2,9) → (3 , 9) → (4, 9) → (5, 9) is one of the shortest paths, and the packet transfer within the storage apparatus 10 is required at least nine times.

  As described above, when the switching relay 81 is introduced between the gateway server 21A and the storage device 10, the number of packet transfers can be reduced and the transfer time can be reduced.

  The case of transmitting a packet from the gateway server 21A to the storage apparatus 10 has been described. However, even when data stored in a memory node in the storage apparatus 10 is transmitted to the gateway server 21A, the gateway server 21A and the storage apparatus 10 In general, the system (FIG. 27) in which the switching relay 81 exists between the gateway server 21A and the storage apparatus 10 is directly connected to the gateway server 21A via the adapter (FIG. 28). And transfer time is short.

  However, the number of packet transfers in a system in which a switching relay is introduced depends on the memory node that relays when transmitting and receiving packets between the switching relay and the storage device. For example, as described above, when the gateway server 21A performs packet transfer with the memory node (5, 9) in the system shown in FIG. 27, if the memory node to be relayed is the memory node (1, 9). The number of transfers in the storage apparatus 10 is at least four. However, when another memory node is relayed, transfer is required five times or more.

  Thus, in order to minimize the number of packet transfers, it is important to select the memory node closest to the transmission destination node as the relay memory node among the memory nodes directly connected to the switching relay. The procedure for calculating the address of the relay memory node that will minimize the packet transfer time will be described later.

  In a system in which a gateway server and a storage device are connected without using a switching relay, when packet transfer is performed between the gateway server and a memory node in the storage device, the transmission source and the transmission destination are the gateway server, the memory Any of the nodes, which is not changed while packet forwarding is taking place.

  On the other hand, when performing packet transfer in a system in which a gateway server and a storage device are connected via a switching relay, packet transfer is first performed with the relay memory node as the destination. After the packet reaches the relay memory node, packet transfer is performed with the memory node (when the packet is transmitted from the gateway server to the memory node) or the switching relay (when the packet is transmitted from the memory node to the gateway server) as the destination. . That is, the packet transmission destination needs to be changed before and after the packet reaches the relay memory node.

  FIG. 29 shows an example of the address information recorded in the header part of the packet in the system using the switching relay. Here, the final destination node and the source node mean a destination memory node that finally delivers the packet and a memory node that first creates and transmits the packet. On the other hand, the temporary transmission destination node address and the temporary transmission source node address are addresses used for determining the packet transfer destination in each memory node, and are rewritten in the relay memory node connected to the switching relay (the rewriting rules will be described later). ).

  The address type is for distinguishing whether the address is an address of a memory node in the storage apparatus 10 or an address of the switching relay 81. Further, the relay node address is an address of a memory node connected to the switching relay 81 that should be passed at the time of packet transfer.

  As described above, in the memory node connected to the switching relay, the address information recorded in the header part of the packet is rewritten. When the address information of the packet is as shown in FIG. 29, the rewrite rule is, for example, as follows.

  1. If the final destination node is a memory node other than its own node and the source is a switching relay, the temporary destination is changed to the final destination node and the temporary source is changed to its own node.

  2. If the final destination node is a switching relay, change the temporary destination to the switching relay and the temporary source to its own node.

  3. If the final destination is a memory node other than its own node and the source is also a memory node, neither the temporary destination nor the temporary source is changed.

  For example, in the system shown in FIG. 27, the procedure for the client to write a file to the storage apparatus 10 is as follows.

  FIG. 30 is a diagram showing a write operation in the storage system shown in FIG.

  First, the client transmits a file and a file ID to the gateway server 21A (see (1)). The file ID is an identifier that can uniquely identify a file.

  Next, the gateway server 21A divides the file into data of a prescribed size and assigns a divided data ID to each divided data. Further, the file ID and the divided data ID are written in the file table. The divided data ID is an identifier that can uniquely identify the divided data (see (2)).

  Next, the gateway server 21A determines the address of the memory node (write node) to which the divided data is written based on the information of the divided data ID (address (5, 9) in FIG. 27). Further, of the memory nodes connected to the switching relay 81, the memory node closest to the writing node is used as a relay memory node, and the address is calculated (in FIG. 27, the address (1, 9)) (see (3)).

  Next, the gateway server 21A creates a write packet in which the header portion including the address information described above is added to the write data. Here, the temporary transmission destination address is the relay node address (address (1, 9) in FIG. 27), and the temporary transmission source node address is the address of the switching relay 81 ([1] in FIG. 27). Thereafter, the packet is transmitted to the switching relay 81 (see (4)).

  The switching relay 81 that has received the write packet transmits this write packet to the designated temporary transmission destination memory node (relay memory node (1, 9)) (see (5)).

  The relay memory node that has received the write packet from the switching relay 81 reads the header part of the packet, and the final transmission destination node is another memory node in the storage apparatus 10, so that the write with the header part rewritten as follows Create a packet. In the header section, the temporary transmission source address is changed to its own node address, and the temporary transmission destination node address is changed to the address of the writing node which is the final transmission destination ((5, 9) in FIG. 27) (see (6)).

  Thereafter, the write packet is transferred to the adjacent memory node. By repeating appropriate transfer in the storage device 10, the write packet reaches the write node (memory node (5, 9) in FIG. 27) (see (7)).

  In the write destination memory node, the write data of the received packet, the address of the packet transmission source, the relay node, and the like are written in the memory 16 of the own node (see (8)). Thereafter, a write completion report packet is created and sent back to the gateway server 21A through the reverse route (see (9)). In the write completion report packet, the final transmission destination node of the address information in the header section is a switching relay, the transmission source is the write node, the relay node is the same memory node as the packet transfer from the gateway server 21A to the write node, The transmission destination node is a relay node, and the temporary transmission source node is a writing node. In the relay memory node, the address information of the header part is rewritten in the relay memory node with the temporary transmission destination as the switching relay and the temporary transmission source as the relay node (see (10)).

  After all the divided data has been written, the gateway server 21A reports the end of file writing to the client.

  It should be noted that not only when the client writes a file to the storage apparatus 10, but also for reading / erasing the written data and checking the free capacity of the memory, a read / erase command, a free capacity reply command, etc. Even when the packet is sent to the memory node, the packet can be transferred via the memory node connected to the switching relay 81 in accordance with the procedure described above. In addition, when a command execution completion report such as writing or erasing, data read from a memory node, or the like is transmitted from the memory node to the gateway server 21A, the memory node connected to the switching relay 81 in the same manner as described above The packet can be transferred via Thereby, the transfer time can be shortened.

  The storage system in which only the leftmost memory node is connected to the switching relay 81 as shown in FIG. 27 has been described so far. However, as shown in FIG. Even in a storage system connected to the relays 81, 82, 83, and 84, the average number of packet transfer times is smaller and the transfer time can be shortened than a system that does not introduce a switching relay. In the storage system shown in FIG. 31, a switching relay 81 is connected to the memory node arranged at the left end of the storage device, the switching relay 82 is connected to the memory node arranged at the upper end, the switching relay 83 is connected to the memory node arranged at the right end, Switching relays 84 are respectively connected to the memory nodes arranged at the lower end. All the switching relays are connected to the gateway server 21A through the adapter 22A.

  Further, as shown in FIG. 32, a storage device in which a plurality of memory nodes having input / output ports (non-adjacent ports) other than the input / output ports connected to adjacent nodes are connected to the switching relay 81 by non-adjacent ports. Even in the storage system including the switching system, the average number of packet transfer times is smaller and the transfer time can be shortened than in a system that does not introduce a switching relay. In the storage system shown in FIG. 32, the switching relay 81 is connected to the memory nodes (3, 2), (3, 7), (8, 2), (8, 7) in the storage apparatus 10.

  As described above, when packet transfer is performed via a memory node connected to a switching relay, it is important to select a memory node having a minimum packet transfer time (number of transfers) as a relay memory node. . Hereinafter, address calculation of the relay memory node will be described.

  First, as shown in FIG. 27, when all the leftmost memory nodes perform data transfer between the storage apparatus 10 connected to the switching relay 81 and the gateway server 21A, the addresses (x, y in the storage apparatus 10). The memory node connected to the switching relay 81 that minimizes the packet transfer distance (number of transfers) is the memory node at the address (1, y).

  In the storage device having memory nodes arranged at the corners of (a0, b0) and having c0 horizontal and d0 vertical memory nodes, all the memory nodes on the outer periphery are connected to the switching relay. When the memory node at the address (x, y) in the device 10 performs packet transfer with the gateway server 21A, the address of the relay memory node that minimizes the transfer time (transfer count) is as follows. Here, as shown in FIG. 31, the address of the memory node connected to the switching relay is represented by (a0, y), (a0 + c0, y), (x, b0), (x, b0 + d0). Is an arbitrary integer value of a0 to a0 + c0, and y is an arbitrary integer value of b0 to b0 + d0).

min (x−a0, a0 + c0−x) ≦ min (y−b0, b0 + d0−y) and min (x−a0, a0 + c0−x) = (x−a0, y) when x−a0;
min (x−a0, a0 + c0−x) ≦ min (y−b0, b0 + d0−y) and min (x−a0, a0 + c0−x) = a0 + c0−x (a0 + c0−x, y);
min (x−a0, a0 + c0−x) ≧ min (y−b0, b0 + d0−y) and min (y−b0, b0 + d0−y) = y−b0 (x, y−b0);
When min (x−a0, a0 + c0−x) ≧ min (y−b0, b0 + d0−y) and min (y−b0, b0 + d0−y) = b0 + d0−y, (x, b0 + d0−y).

  Note that min (x, y) is a function that gives the smaller of the two arguments x and y.

  On the other hand, as shown in FIG. 32, when the address of the memory node connected to the switching relay 81 is given by (am + b, cn + d) (m and n are integers) (in FIG. 32, a = 5, b = 3, c = 5, d = 2, m = 0 or 1, n = 0 or 1), packet transfer time (number of transfers) is minimized for any memory node (address (x, y)), switching The memory node (relay memory node) connected to the relay 81 is given by (a (round (x / a, 0) + b, c (round (y / c, 0) + d)) where round (u , 0) is a function for rounding the numerical value u to the decimal point.

  As described above, according to the eighth embodiment, the number of packet transfers between memory nodes in the storage apparatus can be reduced. Other configurations and effects are the same as those of the first embodiment described above.

[Ninth Embodiment]
The ninth embodiment includes a data processing procedure for selecting, as data storage destinations, memory nodes having different data transfer times with the gateway server in data processing for storing a plurality of data in a plurality of memory nodes.

[1] Configuration of Storage System FIG. 33A is a diagram showing the configuration of the storage system of the ninth embodiment.

  As illustrated, the storage system includes a storage apparatus 10 including a plurality of memory nodes 11 and a plurality of gateway servers 21A connected to the storage apparatus 10. As shown in FIG. 1, the storage apparatus 10 has a configuration in which a plurality of memory nodes having a data transfer function are connected to each other. A gateway server 21A is connected to the memory nodes (1, 4) arranged on the outer periphery of the storage apparatus 10 via an adapter 22A.

[2] Data processing method of storage system A data processing procedure in the storage system of the ninth embodiment will be described.

  33A to 33D and FIGS. 34A to 34E are storage systems in which the storage apparatus 10 is connected to the gateway server 21A via the adapter 22A. The storage device is composed of a plurality of memory nodes in which adjacent memory nodes 11 are interconnected.

  Each memory node 11 receives the transmitted packet if it is addressed to itself, and transfers the packet to an adjacent memory node if another memory node is the destination. With this data transfer function, packet communication can be performed between the gateway server 21A and the designated memory node.

  However, the number of transfers required for packet transfer differs depending on the memory node. For example, the minimum number of transfers required for packet communication with the gateway server 21A is 0 for the memory node at the address (1, 4), and the memory at the addresses (1, 5), (2, 4), and (1, 3). The node is once, and the memory nodes at the addresses (1,6), (2,5), (3,4), (2,3), (1,2) are twice. In FIGS. 33A to 33D and FIGS. 34A to 34E, memory nodes having the same minimum number of transfers are represented by the same hatching.

  Here, the packet transfer time in the memory node of the storage apparatus 10 and the packet transmission time between the memory nodes are constant regardless of the memory node. In this case, the total packet transfer time is the same between any memory nodes. Note that the packet transfer time is the time from when a packet is received at the input port to when it is determined whether the packet is addressed to the own node from the address recorded in the header of the packet and when the packet is output from the output port. The total packet transfer time is the time from when a memory node receives a packet until the packet is transmitted to an adjacent node until the packet reaches the adjacent node.

  Now, one file is divided into three, and the three data with ID = 1, ID = 2, ID = 3 and ID are stored in three memory nodes according to the order of the divided data. . Then, consider data processing in which three data are read from the memory node and transmitted to the gateway server.

  The three memory nodes that store data are read for each of the memory nodes that have the same minimum transfer count required for packet communication with the gateway server and the memory nodes that have different minimum transfer counts. Compare the transfer process of stored data in the storage device. It is assumed that the read command is performed all at once, and the transmission of the read data is started at the same time, so that the transfer of the three data proceeds at exactly the same timing.

  Three data stored in the memory nodes (1, 1), (2, 2), (3, 5) having a minimum transfer count of 3 required for packet communication with the gateway server 21A are received from each memory node. FIG. 33A to FIG. 33D show an example of a process in which the data is read and transferred to the gateway server 21A.

  FIG. 33A shows the state of the storage apparatus 10 immediately after data is read from the memory in each memory node (before transfer), and FIG. 33B shows that a fixed time has passed from FIG. 33A and all three data are stored. It shows the state transferred to the adjacent node of the memory node that has been set. As shown in FIG. 33B, the three data stored in the memory nodes (1, 1), (2, 2), (3, 5) in FIG. 33A are stored in the memory nodes (1, 2), (2, 3) and (2, 5), respectively.

  FIG. 33C and FIG. 33D show a state where data transfer is performed twice and three times, respectively. As shown in FIG. 33C, the three data stored in the memory nodes (1, 2), (2, 3), (2, 5) in FIG. 33B are stored in the memory nodes (1, 3), (2, 4) and (1, 5), respectively. Furthermore, as shown in FIG. 33D, the three data stored in the memory nodes (1, 3), (2, 4), and (1, 5) in FIG. 33C are stored in the memory nodes (1, 4), respectively. Has been transferred. 33A to 33D are processes in which data is transferred from the stored memory node to the gateway server 21A, so that the minimum number of transfers required for packet communication with the gateway server 21A with time elapses. Data is transferred to few memory nodes.

  In the storage apparatus 10 shown in FIGS. 33A to 33C, the three data exist in different memory nodes. However, in the storage apparatus 10 shown in FIG. ) Located on the memory node. This represents a state in which three pieces of data have reached the memory node at the address (1, 4) at the same time.

  However, if the storage capacity size of the temporary storage memory (input port buffer) of the memory node is not large enough to store a plurality of data, the memory node (1, 4) can accept only one data at a time. Can not. The temporary storage memory is a memory for storing data that is not addressed to the own node until the data is transferred.

  In this case, until transfer of one received data to the adapter 22A is completed, other data cannot be transferred to the memory node at the address (1, 4), and it is necessary to wait at an adjacent node.

  Even when the size of the memory for temporary storage of the memory node is sufficiently large and three data can be temporarily stored at the same time, it is normal to transmit three data from the memory node at the address (1, 4) to the gateway server 21A at a time. Since this is not possible, the other data must wait at the memory node at the address (1, 4) until one data is transmitted to the gateway server 21A. When such data transfer waiting occurs, the time required to transfer all data increases.

  As can be seen from FIGS. 33A to 33D, the number of memory nodes having the same minimum number of transfers required for packet communication with the gateway server 21A decreases as the number of transfers decreases. Therefore, as shown in FIG. 33A, when a plurality of data is stored in different memory nodes having the same minimum number of transfers required for packet communication with the gateway server 21A and read out at the same time, and the transfer starts, the data passes over time. The closer to the gateway server 21A, the smaller the number of transfer destination memory nodes, and the higher the probability that a plurality of data will be transferred to the same memory node. In this way, if a plurality of data is stored in the memory nodes having the same minimum number of transfers required for packet communication with the gateway server 21A, useless waiting time is likely to occur during transfer after reading.

  With reference to FIGS. 34A to 34E, 35A, and 35B, a data processing procedure for reducing the waiting time at the time of such transfer will be described.

  34A to 34E are diagrams illustrating an example of a process in which data read from the memory node is transferred to the gateway server 21A. 34A to 34E show the memory node (1, 5) having the minimum transfer count required for packet communication with the gateway server 21A of 1 and the memory node having the minimum transfer count of 2 in the memory node (1, 5). The process of storing data of ID = 1 in (1, 2) and storing data of ID = 3 in the memory node (4, 4) which is three times, and simultaneously reading these data and transferring them to the gateway server 21A. Show.

  Also in this case, data is transferred to a memory node having a minimum minimum number of transfers required for packet communication with the gateway server 21A. However, in the case of FIGS. 34A to 34E, the transfer of a plurality of data to the same memory node as shown in FIG. 33D does not occur in all the processes. This is because data is stored in memory nodes having different minimum transfer counts necessary for packet communication with the gateway server 21A. In this case as well, data transfer is performed to a memory node with a minimum minimum transfer count, but since all data transfer starts from a memory node with a different minimum transfer count, each data has a minimum transfer count at any time. Located in a different memory node.

  In this way, by storing data in memory nodes having different minimum transfer counts necessary for packet communication with the gateway server 21A, simultaneous transfer of a plurality of data to the same memory node at the time of transfer of read data is avoided. Transfer time can be shortened.

  Of course, when data stored in memory nodes with different minimum number of transfers required for packet communication with the gateway server is read out and transferred to the gateway server, the memory nodes storing the data are in the order closest to the gateway server. Data arrives at the gateway server. In FIG. 34A, ID = 2 data is stored in the memory node closest to the gateway server 21A, ID = 1 data is stored in the memory node closest to the server 21A, and ID = 3 data is stored in the memory node farthest from the server 21A. Since the data is stored, the data read / transferred from these memory nodes is the gateway in the order of data of ID = 2, data of ID = 1, and data of ID = 3 as shown in FIGS. 34C to 34E. The server 21A is reached.

  As described above, these data are obtained by dividing one file into three, and ID = 1, ID = 2, ID = 3 and ID are assigned in accordance with the order of the divided data. For this reason, in order to reconstruct a file from the divided data, it is necessary to replace the data that has reached the gateway server 21A in the order of ID.

  On the other hand, in FIG. 35A, ID = 1 data is transferred to the memory node having the minimum transfer count of 1 from the gateway server 21A, ID = 2 data is transferred to the memory node having the minimum transfer count of 2 times, and ID = 3 is transferred to the memory node. It is stored in a memory node with 3 transfers. Therefore, when these data are read from the memory node and transferred to the gateway server 21A, as shown in FIG. 35B, the data of ID = 1, the data of ID = 2, the data of ID = 3 are sequentially sent to the gateway server 21A. To reach. Therefore, it is not necessary to rearrange data when reconstructing a file.

  As described above, when storing a plurality of data whose data reading order is meaningful in a memory node having a different minimum transfer count required for packet communication with the gateway server, a memory node close to the gateway server according to the order. The memory nodes are stored in order from the remote node. Thereby, the rearrangement work of the data transferred to the gateway server 21A can be omitted.

  Now, in the storage system in which a single memory node of a storage device composed of all equivalent memory nodes and a gateway server are connected via an adapter, the procedure for storing multiple data in the storage device has been described. As shown in FIG. 36A, a gateway server also includes a storage device having a plurality of memory nodes having input / output ports (non-adjacent ports) other than input / output ports connected to adjacent nodes and interconnected by non-adjacent ports. By storing data in memory nodes having different minimum transfer counts necessary for packet communication with 21B, it is possible to avoid unnecessary waiting time when transferring stored data.

  Further, as shown in FIGS. 36B to 36D, even when the storage apparatus 10 and the gateway server 21A are connected via a switching relay, the memory nodes having different minimum transfer counts necessary for packet communication with the gateway server 21A are different. By saving the data, it is possible to avoid useless waiting time when transferring the saved data. In FIG. 36A to FIG. 36D, an example in which data of ID = 1, ID = 2, and ID = 3 is stored in the memory nodes having the minimum transfer count of 1, 2, and 3 required for packet communication, respectively. Show.

As described above, there is a method (consistent hashing) for determining the address of the memory node that stores the packet data from the hash value of the address of the memory node and the hash value of the packet ID. For example, an ID space taking an integer value of 0 to 2 ¹⁶⁰ −1 is considered, and for each of the memory node and the packet, the former is an address and the latter is a packet ID to calculate the cryptographic hash function SHA-1. The calculation result is any integer value between 0 and 2 ¹⁶⁰ −1), so that they are assigned to any ID in the ID space. Next, for each packet, the ID space is traced clockwise from the ID of the hash value, and the first memory node encountered is determined as the memory node that stores the packet.

  This method not only can distribute and store packets in memory nodes with high uniformity, but also has a feature that the number of packets that need to be changed is small even if the number of memory nodes is increased or decreased (that is, the scalability of the memory nodes is high). . However, when a memory node that stores a packet is determined by this method, the packet can be stored in a memory node having the same number of transfers required for communication with the gateway server (as described above, this causes a packet collision when reading data). cause).

  In order to uniformly distribute and store N packets in memory nodes having different packet transfer counts in communication with the gateway server, for example, the hash value mod N of the packet ID is calculated (mod is the remainder of the division) Return function), determine the number of transfers (any of 0 to (N-1)) of the memory node storing the packet (if there is a packet with the same number of transfers as a result of the calculation, adjust it to a different value) To do). Furthermore, for each packet, a memory node that stores the packet may be determined by the above-described consistent hashing from among the memory nodes having the determined number of transfers.

  In the procedure for determining the memory node of the packet storage destination as described above, it is necessary to grasp the address of the memory node having the determined number of transfers. The address of a memory node having a certain number of transfers is expressed as follows.

  As shown in FIG. 33A to FIG. 35B, the gateway server 21A has a memory node (1, a0) located at the end of the storage device (here, all memory node addresses are positive integer values) via the adapter 22A. In the system connected to the memory node, when performing packet communication with the memory node (1, a0), the address of the memory node whose minimum transfer count is n is (1 + b, a0 + (nb)) (n ≧ b ≧ 0) and (1 + c, a0− (nc)) (a0−1 ≧ nc ≧ 0).

  As shown in FIG. 36A and FIG. 36D, in the storage device (memory node address is a positive integer value) in which the memory nodes are not adjacent to each other or the memory node and the switching relay are connected by an increase port (non-adjacent port). When a packet communication is performed with a node (a0, b0) having an increased port, the address of the memory node having the minimum transfer count of n is (a0 + c, b0 + (nc)) (n ≧ c ≧ 0), (a0 + d, b0− (n−d)) (n ≧ d ≧ 0, b0−1 ≧ nd), (a0−e, b0 + (n−e)) (min (a0−1, n) ≧ e ≧ 0), (a0−f, b0− (n−f)) (a0−1 ≧ f ≧ 0, b0−1 ≧ n−f ≧ 0).

  As shown in FIG. 36B, in the system in which the memory node at the left end (address (1, y)) of the storage apparatus (addresses of all memory nodes are positive integer values) is connected to the switching relay, the address (1 , A0), when performing packet communication with the relay memory node, the address of the memory node whose minimum transfer count is n is represented by (1, a0 + n).

  Note that a system including a storage device in which memory nodes having the same data transfer time are interconnected has been described. However, as shown in FIG. 36E, a storage system in which memory nodes having the same data transfer time are connected in a tree shape, The present embodiment can be applied to any system including a storage apparatus having memory nodes having different data communication times with a server, such as a storage system including memory nodes having different data transfer times as shown in 36F.

  The system shown in FIG. 36E is a storage system in which memory nodes having the same data transfer time are connected in a tree shape, and the memory nodes having the same communication time with the gateway server 21A are indicated by the same hatching. In the system shown in FIG. 36F, the data communication speeds between the memory nodes and between the memory nodes and the switching relays are the same, and addresses (a, l), (a, m), (a, n), (b, l ), (B, m), (b, n) between the memory nodes and between the memory nodes and the switching relay 81, the addresses (α, δ), (β, γ), (β , ε) and the data communication speed between the memory nodes and the switching relay 81 are doubled, and the communication speed between the address (A, B) and the switching relay 81 is quadrupled. Note that memory nodes having the same data transfer time with the switching relay 81 are indicated by the same hatching. However, it is assumed that the data transfer time in the memory node is sufficiently shorter than the data transfer time between the memory nodes and between the switching relay and the memory node.

  As described above, according to the ninth embodiment, it is possible to reduce the transfer time when data stored in a plurality of memory nodes is read from each memory node and then transmitted to the gateway server. Thereby, high-speed data reading becomes possible. Other configurations and effects are the same as those of the first embodiment described above.

[Tenth embodiment]
According to the tenth embodiment, in data processing for transmitting a plurality of data from a gateway server to a plurality of memory nodes, data destined for a memory node with a short data communication time from data destined for a memory node with a long data communication time In this order, a data processing procedure for performing data transfer is provided.

[1] Configuration of Storage System FIG. 37A is a diagram showing the configuration of the storage system of the tenth embodiment.

  As illustrated, the storage system includes a storage apparatus 10 including a plurality of memory nodes 11 and a plurality of gateway servers 21A connected to the storage apparatus 10. As shown in FIG. 1, the storage apparatus 10 has a configuration in which a plurality of memory nodes having a data transfer function are connected to each other. A gateway server 21A is connected to the memory nodes (1, 4) arranged on the outer periphery of the storage apparatus 10 via an adapter 22A.

[2] Data processing method of storage system A data processing procedure in the storage system of the tenth embodiment will be described.

  37A to 37G and FIGS. 38A to 38E are storage systems in which the storage apparatus is connected to the gateway server 21A via the adapter 22A, as in the ninth embodiment. The storage device is composed of a plurality of memory nodes in which adjacent memory nodes 11 are interconnected.

  Here, the gateway server 21A transmits three data with ID = 1, ID = 2, and ID = 3 to the memory nodes with addresses (1, 5), (1, 2), and (4, 4), respectively. Consider the relationship between the data transmission order from the gateway server 21A and the time required for all data transmission. Note that only one packet can be stored in the temporary storage memory (input port buffer) of the memory node, and only one packet at a time can be transmitted from the gateway server 21A to the memory node at the address (1, 4). To do.

  In FIGS. 37A to 37G, the order of data destined for a memory node far from the memory node close to the gateway server 21A (ie, data with ID = 1, data with ID = 2, and data with ID = 3). The packet transfer process when a packet is transmitted from the gateway server 21A to the storage device is shown in the order of

  In this case, as a matter of course, as shown in FIGS. 37B to 37C, the transmission of data (ID = 1) addressed to the memory node with the address (1, 5) closest to the gateway server 21A is completed earliest.

  However, the data (ID = 3) destined for the memory node with the address (4, 4) farthest from the gateway server 21A has two data (ID = 1, ID = 2) as shown in FIG. 37D. Until it is transferred from the memory node at the address (1, 4) to the adjacent memory node, it cannot be transmitted to the address (1, 4), and it is necessary to wait at the gateway server 21A until then. Even if the standby is completed and the data is transmitted to the memory node at the address (1, 4), as shown in FIGS. 37D to 37G, at least three transfers are required for the data to reach the address (4, 4). is necessary.

  On the other hand, in FIG. 38A to FIG. 38E, the order of data destined for the memory node farthest from the gateway server 21A to the data destined for the nearest memory node (namely, data with ID = 3, data with ID = 2, ID = 1 shows the packet transfer process in the case where packets are transmitted from the gateway server 21A to the storage device.

  Also in this case, as shown in FIGS. 38B to 38D, ID = 1 and ID = 2 data are transferred from the memory node at address (1, 4) to the adjacent node until ID = 1. Data cannot be transmitted to the memory node at the address (1, 4), and it is necessary to wait at the gateway server 21A.

  However, the memory node of the address (1, 5) that is the destination of the data (ID = 1) transmitted later is close to the gateway server 21A and reaches the destination memory node with a small number of transfers. On the other hand, the data (ID = 3) transmitted from the gateway server 21A earlier is away from the gateway server 21A, but transfer is started while other data is waiting on the gateway server 21A. Therefore, the destination is reached that much earlier. As a result, as shown in FIG. 38E, the three data reach the destination memory node at the same time.

  As can be seen from the comparison between FIGS. 37A to 37G and FIGS. 38A to 38E, the data is transmitted from the gateway server 21A in the order of the data in which the destination memory node is away from the gateway server 21A and the data closer to the gateway server 21A. As a result, the time required for all data transmission can be minimized.

Although a system including a storage device in which memory nodes having the same data transfer time are interconnected has been described, a storage system (FIG. 36E) in which memory nodes having the same data transfer time are connected in a tree shape, or a data transfer time In the case where a plurality of data is transmitted to a storage apparatus having memory nodes having different data communication times, such as a storage system (FIG. 36F) including different memory nodes, the present embodiment can be similarly applied. As described above, according to the tenth embodiment, when a plurality of data is sent to a plurality of memory nodes having different communication times with the gateway server, the required communication time can be minimized. Other configurations and effects are the same as those of the first embodiment described above.

  As described above, according to the first to tenth embodiments, it is possible to provide a storage apparatus and a data processing method capable of efficiently transferring packets without the memory node having to manage the routing table. Can do.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 10 ... Storage apparatus, 11 ... Memory node, 12 ... Input port, 12A ... Input port buffer, 13 ... Output port, 13A ... Output port buffer, 14 ... Selector, 15 ... Packet controller, 16 ... Memory, 17 ... Memory controller, 18 ... MPU, 19 ... local bus, 20 ... storage system, 21A, 21B, 21C, 21D ... gateway server, 22A, 22B, 22C, 22D, 22E ... adapter, 31A, 31B1, 31B2 ... client, 41 ... control computer, 42 ... Address converter, 23, 24 ... Wiring, 81, 82, 83, 84 ... Switching relay.

Claims (16)

A first memory node including: a first input port; a first nonvolatile memory; and a first controller that controls the first nonvolatile memory;
A second input port connected to said first memory node, and a second control unit for controlling the second non-volatile memory, said second non-volatile memory, any one of the first A second memory node including a plurality of first output ports connected to the input ports of
Includes a third input port connected to one of said plurality of first output ports, and a plurality of third memory node different from the first to the second memory node,
A storage device comprising:
The second memory node stores first position information that is position information of the second memory node in the storage device;
The first control unit outputs a packet including at least a transmission destination address and a transmission source address to the second memory node,
The second input port receives the packet output from the first memory node;
The second control unit, based on information including at least the transmission destination address, the transmission source address, and the first location information in the packet, at least the plurality of third memory nodes and the first A storage apparatus configured to determine an output destination of the packet from among memory nodes . A first memory node including: a first nonvolatile memory; and a first control unit that controls the first nonvolatile memory;
A first input port connected to the first memory node, a second non-volatile memory, a second control unit for controlling the second non-volatile memory, and a plurality of first output ports; A second memory node comprising:
A plurality of third memory nodes including a second input port connected to one of the plurality of first output ports and different from the first to second memory nodes;
One or more fourth memory nodes different from the first to third memory nodes;
A storage device comprising:
The first control unit outputs a packet including at least a transmission destination address and a transmission source address to the second memory node,
The first input port receives the packet output from the first memory node;
The second control unit is configured such that, among the first memory node, the plurality of third memory nodes, and the one or more fourth memory nodes, position information in the storage device matches the source address. Of the two areas separated by a first straight line connecting the fifth memory node and the sixth memory node whose position information in the storage device matches the destination address. A second region that is different from the first region, when the two memory nodes belong to, outputting the packet to a seventh memory node that exists in a first direction among the plurality of third memory nodes; Storage device that outputs the packet to an eighth memory node existing in a second direction among the plurality of third memory nodes when the second memory node belongs to the second memory node . The second control unit records information included in the packet in the second non-volatile memory when position information of the second memory node in the storage device matches the transmission destination address. The storage apparatus according to claim 2. The second control unit is used for outputting the destination address, the source address, the first location information, and a packet different from the packets at the plurality of first output ports in the packet. The storage apparatus according to claim 1, wherein an output destination of the packet is determined from at least the plurality of third memory nodes and the first memory node, based on information including at least occupancy information indicating whether the packet has been processed. . The second control unit, based on information including at least occupancy information indicating whether the packet is used to output a packet different from the packet at the plurality of first output ports, The storage apparatus according to claim 2, wherein the storage apparatus outputs the data to any one of the third memory nodes. Said second control unit, when the second memory node in said first region belongs, the output port connected to the seventh memory node of the plurality of first output ports the packet if it is used for the output of another packet and outputs the packet to the memory node of the eighth, if the second memory node to the second region belongs to the plurality of first If the output port is the packet to be connected to the eighth memory node of the output ports being used for the output of another packet, claim 2 for outputting the packet to the seventh memory node The storage device described in 1. The second control unit is a region that is perpendicular to the first straight line in the first region and delimited by a second straight line that passes through the sixth memory node, and a third region of the region close to the memory node, and the other was a fourth region in said second region, among the areas separated by the second linear, the area close to the fifth memory node 5 When the second area is the sixth area and the second memory node belongs to the third area, the packet is output to the seventh memory node, and the fourth area is When the second memory node belongs, the packet is output to a memory node existing in a direction opposite to the second direction among the plurality of third memory nodes,
When the second memory node belongs to the fifth area, the packet is output to the eighth memory node, and when the second memory node belongs to the sixth area, the plurality of third nodes The storage apparatus according to claim 6, wherein the packet is output to a memory node existing in a direction opposite to the first direction among the memory nodes. One or more fourth memory nodes different from the first to third memory nodes,
The second control unit is configured such that, among the first memory node, the plurality of third memory nodes, and the one or more fourth memory nodes, position information in the storage device matches the destination address. The storage apparatus according to claim 1, wherein the packet is output to a memory node having the shortest transfer path to the memory node.   The storage apparatus according to claim 1, wherein the second memory node includes one or more input ports connectable to a computer in addition to the first input port. One or more fourth memory nodes different from the first to third memory nodes,
The first non-adjacent memory node not adjacent to the second memory node among the plurality of third memory nodes and the first non-adjacent memory node among the one or more fourth memory nodes are connected. One or more second non-adjacent memory nodes that are not adjacent to the first non-adjacent memory node, and the second memory node has an additional address determined by the relative physical location of the memory nodes,
The second control unit includes information including at least one of an additional address of the second memory node, an additional address of the first non-adjacent memory node, and an additional address of the second non-adjacent memory node. 2. The storage apparatus according to claim 1, wherein an output destination of the packet is determined from at least the plurality of third memory nodes and the first memory node based on. One or more fourth memory nodes different from the first to third memory nodes;
A relay connected to any one or more of the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes;
Further comprising
Among the memory nodes connected to the relay, the relay includes the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes. 2. The storage apparatus according to claim 1, wherein the packet is transmitted to a memory node having a shortest transfer path to a memory node whose position information in the storage apparatus matches the destination address. A relay connected to any one or more of the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes;
Further comprising
The storage device according to claim 2, wherein the relay transmits the packet to a memory node having a shortest transfer path to the sixth memory node among the memory nodes connected to the relay. One or more fourth memory nodes different from the first to third memory nodes,
Memory having at least one or more input ports not connected to other memory nodes among the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes. The total number of computers connectable to a node is Nc, and the total number of the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes is Nnode. 2, the storage apparatus according to claim 1, wherein Nc <√Nnode is satisfied. Memory having at least one or more input ports not connected to other memory nodes among the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes. The total number of computers connectable to a node is Nc, and the total number of the first memory node, the second memory node, the plurality of third memory nodes, and the one or more fourth memory nodes is Nnode. 3, the storage device according to claim 2, wherein Nc <√Nnode is satisfied. 15. The storage apparatus according to claim 13, wherein a relational expression of Nc <0.1 × √Nnode is satisfied. The second memory node further includes an address converter that converts a key included in the packet into an address;
It said second control unit, holds the record of the key-value type data, the destination address and converted the address by the address converter, according to claim 1 and outputs, including the value in the packet or 2. The storage device according to 2 .

Images