JP2736237B2 - Remote memory access controller - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a remote memory access control device in a distributed shared memory type parallel computer capable of accessing a remotely mounted memory, and in particular, to fine-tune the access destination by using a list vector. The present invention relates to a remote memory access control device capable of controlling and accessing at high speed.

[0002]

2. Description of the Related Art In a conventional distributed shared memory type parallel computer, when a read request or a write request issued from a microprocessor is received and one access means access to a remote memory, one remote memory access is performed. Was generating packets.

For this reason, when performing processing for accessing the remote memory at various addresses one word at a time, (i)
A series of operations in which the microprocessor reads the access destination address itself from the memory, or calculates the address based on the data read from the memory, and (ii) accesses the remote memory using the result as an address. Must be sequentially processed in a microprocessor program.

Further, when remote memory on a large number of remote processors is accessed little by little, a large amount of remote memory address information passes through a data cache in the microprocessor.・ Look aside buffer (Trnslation Look Acid Bu)
(hereinafter, abbreviated as TLB), there is a possibility that a remote memory address that overflows a limited number of entries one after another out of the page.

Conventionally, whether or not the writing to the remote memory has been completed has been confirmed by using a response packet. Therefore, the completion of the writing is transmitted only later until all the responses are returned. And there is a disadvantage that the inter-processor connection network is congested by the response packet.

[0006]

As described above, in the conventional distributed shared memory type parallel computer, the packet output from the remote memory access control device to the connection network can be generated only intermittently, and the It increases the processing time of the processor, hinders the overlap of communication with other arithmetic processing, and prevents the cache and TLB from passing remote memory address information through the cache and TLB.
There is a problem that the hit rate is lowered.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and eliminates the occurrence of an interrupt due to a cache or TLB mishit, and enables a remote memory access packet to be continuously generated. It is an object to provide a memory access control device.

Another object of the present invention is to provide a remote memory access control device for preventing a connection network from being congested by a response packet to a write process to a remote memory.

[0009]

According to a first aspect of the present invention, there is provided a parallel computer including a plurality of processors and a distributed shared memory which is physically distributed and mounted and which can be logically accessed as a shared memory from the plurality of processors. A remote memory access control device for controlling remote memory access of said corresponding processor to said distributed shared memory, comprising: pointer specifying means for specifying an address of packet generation information; and packet number specification for specifying the number of packets to be generated. Means, an access request detecting means for detecting a remote memory access request from the corresponding processor, and, when the remote memory access request is detected, based on the address indicated by the pointer designating means, Packet of a predetermined format from the local memory of the processor Local memory access means for reading application information; and packets for remote memory access to the distributed shared memory for the number of packets specified by the packet number designation means based on the read packet generation information. And a packet generating means for generating.

According to a second aspect, in the remote memory access control device according to the first aspect, a storage format of the packet generation information stored in the local memory is arranged.

According to a third invention, in the remote memory access control device according to the first invention, a first buffer means for temporarily storing the packet generation information and a second buffer means for temporarily storing write data. Wherein the local memory access means fetches information from the local memory to the first and second buffer means when the remote memory access request is a remote memory write request. When the remote memory access request is a remote memory write request, the packet generation means starts the remote memory based on the contents of the set of the corresponding packet generation information and the write data. Generating a write packet, said local memory access means and said packet And forming means, characterized in that so as to pipeline manner are simultaneously operated.

According to a fourth aspect of the present invention, in the remote memory access control device according to the third aspect, the local memory access means accesses a local memory having a high continuous access speed. The first and second write data are sequentially accessed from the local memory for a plurality of packets.
In the buffer means.

A fifth aspect of the present invention is the remote memory access control device according to the first aspect, further comprising a burst length designating unit for designating the number of data words of the information to be carried by the packet, and designated by the burst length designating unit. A remote memory burst access packet for reading and writing the number of words to be written is generated.

According to a sixth aspect of the present invention, in the remote memory access control device according to the first aspect, the packet generation information has the number of data words of information to be carried by the packet in addition to the memory address. Each time, a remote memory burst access packet for reading and writing a desired number of words of continuous data is generated.

According to a seventh aspect of the present invention, in the remote memory access control device according to the first aspect, the packet generation means does not directly control the program in the non-privileged mode without shifting to the privileged mode. A possible protection information adding mechanism is provided, and a memory access protection mechanism for controlling permission / non-permission of a memory access based on protection information in the packet when a remote memory access packet from another processor is received is provided. It is characterized by the following.

According to an eighth aspect of the present invention, there is provided a parallel computer including a plurality of vector processors and a distributed shared memory which is physically distributed and logically accessible as a shared memory from the plurality of vector processors. A remote memory access control device for controlling remote memory access to the distributed shared memory, wherein packet generation is started by a vector instruction fetched by a vector processor, and is loaded from a local memory to a vector register by a vector loading means of the vector processor. A packet generator is provided which generates packets for remote memory access to the distributed shared memory for the number of packets specified by the vector length designator based on the packet generation information.

According to a ninth aspect, in the remote memory access control device according to the eighth aspect, the packet generation information is stored in at least one of the vector registers, and the write data or vector operation loaded from the local memory is stored. Storing the write data obtained as an output from the device in another vector register of the vector register, combining the corresponding packet generation information and the write data, and generating a remote memory write packet by the fetched vector instruction. Features.

According to a tenth aspect, in the remote memory access control device according to the eighth aspect, a storage format of the packet generation information stored in the local memory is arranged.

According to an eleventh aspect, in the remote memory access control device according to the eighth aspect, a vector instruction capable of designating the number of data words of information to be carried by the packet is provided,
A remote memory burst access packet for reading and writing a predetermined number of words of continuous data in one packet is generated.

According to a twelfth aspect, in the remote memory access control device according to the eighth aspect, the packet generation information has the number of data words of information to be carried by the packet in addition to the memory address. Each time, a remote memory burst access packet for reading and writing a desired number of words of continuous data is generated.

According to a thirteenth aspect, in the remote memory access control device according to the eighth aspect, the packet generation means does not require direct control from a program in the non-privileged mode without shifting to the privileged mode. In addition to providing a possible protection information addition mechanism, when a remote memory access packet from another processor is received,
A memory access protection mechanism for controlling permission / rejection of memory access based on protection information in the packet.

According to a fourteenth aspect, in the remote memory access control device according to the seventh or thirteenth aspect, a translated address obtained by performing a translation by a system call on the transmission side as an address designating the remote memory is stored in the local memory. It is characterized in that the prepared address is reused in the prepared and actual remote memory access.

According to a fifteenth aspect, in a parallel computer including a plurality of processors and a distributed shared memory that is physically distributed and mounted and logically accessible as a shared memory from the plurality of processors, A remote memory access control device for controlling remote memory access to a distributed shared memory, comprising: an access request detecting means for detecting a remote write request from the corresponding processor; and a corresponding packet in response to the remote write request. Means for generating and transmitting to a predetermined processor, means for receiving a packet corresponding to the remote write request from a processor other than the corresponding processor, and according to the content of the packet corresponding to the remote write request, Means for writing data to the corresponding local memory; Characterized by comprising a counter means for counting the number of times of completion of the processing for the packet corresponding to the remote write request.

[0024]

In the remote memory access control device according to the first invention, part or all of the packet generation information is stored in the local memory based on a predetermined format corresponding to at least one packet. The address is set in the pointer specifying means (packet generation information pointer specifying register), and the number of packets to be generated is set in the packet number specifying means (packet number specifying register). When ready to that state, the corresponding (external) processor instructs the generation of a remote memory access packet.

Upon receiving the instruction, the remote memory access control device reads out one packet of packet generation information on the local memory from the address set in the packet generation information pointer designation register, and uses this to read the remote memory access packet. Is generated, the value of the packet generation information pointer designation register is advanced by the number of data bytes of the packet generation information for one packet, and the value of the packet number designation register is decremented by one.

Further, the above-described series of operations for packet generation is repeated until the value of the packet number designation register becomes zero. In this way, with one activation from the processor,
Since multiple remote memory access packets are generated,
After activating the remote memory access control device, the processor can perform another process without being busy for the process of the remote memory access. In addition, since the packet generation information does not pass through the cache of the processor, the hit rate of the cache of the processor is also improved.

In the remote memory access control device according to the second invention, since the storage format of the packet generation information stored in the local memory is arranged, the packet generation information is successively transferred from the local memory to the remote memory access control device. When fetching data into a device, it is easy to generate an address.For example, there is no need to access the memory to obtain an address as compared with a case where access is performed using a linked list. However, the bandwidth of the local memory can be effectively used for accessing the memory of the processor.

The remote memory access control device according to the third invention comprises a buffer means for temporarily storing packet generation information on the transmission side and a buffer means for temporarily storing write data. . In this way, the packet generation information and the write data for a plurality of packets can exist in the remote memory access control device at the same time, and after the packet generation information for the plurality of packets is read, the write data for the plurality of packets is read out. The pipeline operation is performed such that the packet generation means generates the first packet and the memory access means reads out the packet generation information for another packet of the fourth packet. By doing so, the packet generation means can operate simultaneously with the memory access means. Therefore, a remote memory write packet can be generated at short intervals.

The remote memory write packet is a combination of a buffer means for temporarily storing packet generation information and a buffer means for temporarily storing write data, unless one of them becomes empty. A remote memory write packet can be generated without a break.

In the remote memory access control device according to the fourth aspect of the present invention, since the local memory access means for accessing the local memory for continuous access at high speed is provided,
Loading packet generation information and write data for a plurality of packets into the buffer means for temporarily storing packet generation information and the buffer means for temporarily storing write data from the local memory. Therefore, a plurality of packets can be fetched at high speed into the buffer means while using a high effective memory bandwidth by continuous access.

The remote memory access control device according to the fifth aspect of the present invention includes a burst length specifying means (burst length specifying register) for specifying the number of data words to be carried by the packet. Before starting the remote memory access from the processor, the number of data words to be carried by the packet is set in the burst length designation register. The packet generation means generates a remote memory burst access packet for reading and writing the number of words specified by the register.

In this way, block-shaped data specified by the burst length is carried for one packet header including a remote memory address and the like, so that remote memory access is always performed one word at a time. The connection network and the memory bandwidth are effectively used.

In the remote memory access control device according to the sixth aspect of the present invention, the packet generation information stored in the local memory includes the number of data words to be carried by the packet in addition to the memory address.

When generating the remote memory access packet, the packet generation means refers to the number of data words (burst length) to be carried by the packet in the packet generation information, and if reading, includes the burst length in the header. A remote memory read request packet is generated, and in the case of writing, a remote memory write packet is generated which includes data read from the local memory by a burst length in a data portion. Since the burst length is included in the packet generation information for each packet, it is possible to access with a different burst length depending on the access destination.

In the remote memory access control device according to the seventh aspect, the packet generation means includes a protection information adding mechanism that cannot directly control the program in the non-privileged mode without shifting to the privileged mode. .

Since the protection information can be rewritten from the privileged mode, when the processor performs a context switch and switches to another user's process, the operating system can assign a process ID or the like to allocate resources such as the processor and the remote memory access controller. At the time of remote memory access in the non-privileged mode, the packet generation means embeds this information in the header of the packet. In other words, even if the information for packet generation on the local memory is rewritten in the non-privileged mode, erroneous writing to the remote memory of another person is eliminated by checking the protection information in the packet header with the memory access protection mechanism on the receiving side. .

In the remote memory access control device according to the eighth invention, the distributed shared memory type parallel computer is realized as a part of the function of the vector processor. First,
Part or all of the packet generation information for generating a packet for accessing the remote memory is stored in the local memory in a predetermined format corresponding to at least one packet.

Next, loading of packet generation information from the local memory into the vector register is started by the vector load instruction fetched by the vector processor.
Next, the vector processor activates the vector remote memory access instruction fetched by the vector processor with the information corresponding to the number of packets to be generated set in the vector length specifying means (vector length specifying register).

In this way, the memory access means and the packet generation means operate in a pipeline simultaneously, and remote memory access packets are generated one after another by one vector remote memory access instruction. After notifying the vector processor of the permission to start the remote memory access, the main processor can perform another processing without being busy for the processing of the remote memory access.
Further, since the packet generation information does not pass through the cache of the main processor, the hit rate of the cache of the main processor is also improved.

In the remote memory access control device according to the ninth aspect, the write data is stored in the vector register. Therefore, if there is a vector operation unit, the result from the operation unit is held in the vector register, and if the result is specified as the source operand of the vector remote memory write instruction, the packet generation means does not pass through the memory without passing through another vector. Remote memory write packets are generated one after another in combination with the packet generation information loaded into the register from the local memory.

In the remote memory access control device according to the tenth aspect, the storage format of the packet generation information stored in the local memory is an array. Therefore, packet generation information can be easily and quickly loaded into the vector register by the vector load instruction.

The remote memory access control device according to the eleventh aspect of the present invention includes a vector instruction capable of designating the number of data words (burst length) to be carried by a packet. The burst length can be specified by, for example, a general-purpose scalar register or a dedicated register. The vector remote memory burst access instruction is started with the value set therein.

Then, the packet generating means generates a remote memory burst access packet for reading and writing the number of words specified by the register. In this manner, block-shaped data specified by the burst length is carried for one packet header including a remote memory address and the like, so that the connection network and the remote memory access are always performed one word at a time. The memory bandwidth will be used effectively.

In the remote memory access control device according to the twelfth aspect of the present invention, the number of data words (burst length) to be carried by the packet other than the memory address is used as packet generation information held in the local memory and loaded into the vector register. Have.

When generating a remote memory access packet, the packet generation means refers to the burst length in the packet generation information and, if reading, generates a remote memory read request packet including the burst length in the header. ,
In the case of writing, a remote memory write packet including the data in the vector register by the burst length in the data portion is generated. Since the burst length is included in the packet generation information for each packet, it is possible to access with a different burst length depending on the access destination.

In the remote memory access control device according to the thirteenth aspect, the packet generation means has a protection information adding mechanism that cannot directly control the program in the non-privileged mode without shifting to the privileged mode. .

Since the protection information can be rewritten from the privileged mode, when the processor performs a context switch and switches to another user's process, the operating system will assign a process ID or the like to which resources such as the processor and the remote memory access control device will be allocated. At the time of remote memory access in the non-privileged mode, the packet generation means embeds this information in the header of the packet.

In other words, even if the information for generating a packet on the local memory is rewritten in the non-privileged mode, if the information is erroneously written to the remote memory of another person, the protection information in the packet header is checked by the memory access protection mechanism on the receiving side. Will be eliminated by

In the remote memory access device according to the fourteenth aspect, the packet generating means uses a protection information adding mechanism that cannot be directly controlled from the program in the non-privileged mode without shifting to the privileged mode.

Since the protection information can be rewritten from the privileged mode, when the processor performs a context switch and switches to another user's process, the operating system specifies a process ID to allocate resources such as the processor and the remote memory access controller from now on. At the time of remote memory access in the non-privileged mode, the packet generation means embeds this information in the header of the packet.

As for the address for designating the remote memory, the translated address obtained by performing the translation by the system call on the transmission side is prepared in the local memory, so that the translated address on the local memory may be rewritten in the non-privileged mode. . However, erroneous writing to someone else's remote memory is eliminated by checking the protection information in the packet header with the receiving memory access protection mechanism.

In the actual remote memory access, by reusing the address after the above-mentioned conversion, the address conversion does not have to be performed every time the communication is executed. Therefore, it is possible to eliminate the occurrence of TLB and system calls on the transmitting side.

In the case of a remote memory access, the destination processor number will form part of the address of the remote memory, but these are usually different processors requiring different TLB entries, so In the method in which the address translation by the TLB is required one by one, if there are many destination processors, TLB mishits will occur repeatedly.
In the case of the present invention, even when communication is repeated many times, the conversion is performed only once in advance, and thus there is no such problem.

The remote memory access control device according to the fifteenth aspect includes a counter for counting the number of times of completion of the processing for the received remote write request. Therefore, when performing a process in which the number of times of writing from the remote node is known by the program to be written, without returning a response packet to the remote writing,
The value of the above counter, which is counted on the receiving side, is checked on the receiving side, and after confirming that the processing of the predetermined number of write packets has been completed, barrier synchronization is performed as a whole, and the side that has transmitted the remote write packet has Since the completion of writing is immediately transmitted to the processor, the speed is high and the load on the inter-processor connection network is reduced.

[0055]

Embodiments of the present invention will be described below with reference to the drawings. (Embodiment 1) First, the main configuration of this embodiment and its operation and effect will be described.

<Main Structure of First Embodiment> In this embodiment, a plurality of processors (2) are physically distributed and mounted and logically arranged.
In a parallel computer having a distributed shared memory (3) that can be accessed as a shared memory from a remote computer, part or all of the packet generation information for generating a packet for accessing a remote memory is divided into a part corresponding to a desired number of packets. According to a predetermined format, the address of the packet generation information is stored in the local memory (3) outside the remote memory access control device (1) and at hand, and stored in the remote memory access control device (1). Packet generation information pointer specification register (104)
And a packet number designation register (105) for designating the number of packets to be generated, and generates a desired number of remote memory access packets in response to a single instruction from the external processor (2).

<Main Functions and Effects of First Embodiment> In the remote memory access control device of the present embodiment, a part or all of the packet generation information is based on a predetermined format corresponding to a desired number of packets. Local memory (3)
The address is set in the packet generation information pointer designation register (104), and the number of packets to be generated is set in the packet number designation register (105). When preparation is completed up to that state, the external processor (2) instructs generation of a remote memory access packet.

Upon receiving the instruction, the remote memory access control device (1) reads one packet of packet generation information on the local memory (3) from the address set in the packet generation information pointer designation register (104). , A remote memory access packet is generated, and a packet generation information pointer designation register (10
The value of 4) is advanced by the number of data bytes of the packet generation information for one packet, and the packet number designation register (10
Decrement the value of 5) by one.

The above-described series of operations for packet generation is repeated until the value of the packet number designation register becomes zero. In this way, since a single activation from the external processor (2) generates a plurality of remote memory access packets, the external processor (2) activates the remote memory access controller (1) after activating the remote memory access controller (1). Another process can be performed without being busy for the memory access process. Further, since the packet generation information does not pass through the cache (31) of the external processor (2), the hit rate of the cache (31) of the external processor (2) is also improved.

Hereinafter, this embodiment will be described more specifically. <Specific Description of First Embodiment> FIGS. 1 and 2 are diagrams showing an element processor (PE) of a parallel computer provided with a remote memory access control device of the present embodiment.

The parallel computer is realized by interconnecting a plurality of element processors by an interconnection network between processors. In this embodiment, the interconnection scheme of the element processors is arbitrary.

As shown in the figure, in each element processor, a remote memory access control device 1, a microprocessor 2, and a local memory 3 are connected by an external bus 4. The remote memory access control device 1 is directly connected to the inter-processor connection network. In this embodiment, the local memory 3 of each element processor is used as a distributed shared memory.

The microprocessor 2 includes, in addition to the CPU 33, a cache memory 31 and a TLB 32 for address conversion as in a general microprocessor in recent years.
Shall be provided. The cache memory 31 stores the past memory access data and contributes to the performance improvement by not accessing the local memory 3 or the remote memory (not shown) for the memory access hitting it. The TLB 32 caches the page table on the local memory 3 so that address conversion is performed without overhead while the TLB 32 is hit. Therefore, they are effective when there is access locality.

Incidentally, in the conventional distributed shared memory type parallel computer, communication between processors is realized in the form of memory access, and a mechanism that exerts an effect when there is access locality is used. Since inter-processor communication, which is a process that is inexpensive, is realized, an increase in cache misses due to the passage of address information used for remote memory access through the cache,
The use of a different entry for each remote processor used for remote memory access causes a problem such as an increase in TLB misses when the number of access destination processors is large.

In the present embodiment, in order to solve such a problem, the remote memory access control device 1 is activated for each access by monitoring the memory access from the microprocessor 2, 2 operates in an independent manner except at the time of startup, thereby realizing inter-processor communication, that is, remote memory access.

As shown in FIGS. 1 and 2, the remote memory access control device 1 includes a packet generation unit 100, a bus interface unit 101, a memory address generation unit 102, a control unit TLB 103, a packet generation information pointer designation register 104, Packet number designation register 105, packet data part pointer designation register 106, burst length designation register 107, process ID designation register 10
8. Packet generation information buffer 109, transmission packet data buffer 110, transmission packet buffer 11
1. Received packet buffer 112, packet receiving unit 11
3, a response waiting read request number counter 114, a response waiting write request number counter 115, a reception / writing process completion number counter 116, an inter-processor network interface 117, a reply packet buffer 121, and a status register 122.

The remote memory access control device 1 is connected to an external bus 4 by a bus interface unit 101, and through these components, the internal components of the remote memory access control device 1 and the microprocessor 2 and the local memory 3 are connected. The information is transferred between them.

Control device TLB 103, packet generation information pointer designation register 104, packet number designation register 105, packet data part pointer designation register 1
06, the burst length designation register 107, and the process ID designation register 108 are accessed by being mapped in the memory space of the microprocessor 2.

Here, preferably, the packet generation unit (1)
00), a protection information adding mechanism that cannot be directly controlled from the program in the non-privileged mode without shifting to the privileged mode is provided, and the remote memory access packet receiving side is provided with a protection information adding mechanism based on the protection information in the packet. A memory access protection mechanism for controlling permission / denial of memory access may be provided. Since the protection information can be rewritten from the privileged mode, when the processor (2) is context-switched and switches to another user's process, the operating system allocates resources such as an external processor and a remote memory access controller from the process ID. Is set, and at the time of remote memory access in the non-privileged mode, the packet generator (100) embeds this information in the header of the packet. That is, even if the information for packet generation on the local memory (3) is rewritten in the non-privileged mode, if the information is erroneously written to the remote memory of another person, the protection information in the packet header is checked by the memory access protection mechanism on the receiving side. This has the advantage of being eliminated.

In this embodiment, the above mechanism is adopted, and the TLB 103 in the control device and the process ID designation register 108 are mapped to an area which can be accessed only in the privilege mode of the microprocessor 2.

In the TLB 103 in the control device, in addition to a set of a physical address and a logical address used for address conversion at the time of memory access generated by the memory address generation unit 102, a process I assigned to each entry of the TLB 32
Hold D. The entity is located in an area of the local memory 3 which can be accessed only in the privilege mode, and the TLB 32
The information is cached. Here, a process is a unit for allocating a virtual address space, and has the same process ID across a plurality of processors.
There are multiple threads that use.

The process ID designation register 108 holds the process ID assigned to the remote memory access control device 1, and the header of the packet generated by the packet generator 100 includes the process ID in the process ID designation register 108. The ID is embedded. in this way,
TLB103 in control device and process ID designation register 1
08 is a mechanism for holding protection information and preventing unauthorized memory access, so that direct access is not made from the user mode.

On the other hand, if processing can be performed in user mode without using the OS during communication, a large overhead associated with a context switch due to traps can be avoided. It is desirable that the register 105, the packet data pointer designation register 106, and the burst length designation register 107 be mapped to a memory space directly accessible from the user mode.

Here, it is preferable that the storage format of the packet generation information stored in the local memory (3) be an array. In such a case, when the packet generation information is successively fetched from the local memory 3 to the remote memory access control device 1, it is easy to generate an address, for example, when accessing using a linked list. This eliminates the need for memory access to obtain an address, and makes it possible to effectively utilize the bandwidth of the local memory for reading packet generation information and remote write data, and for accessing memory of an external processor.

In the present embodiment, the storage format of the packet generation information stored in the local memory 3 is an array. When the inter-processor communication is performed by the remote memory access control device of the present embodiment, the packet number n is set in the packet number designation register 105, and the packet generation information necessary to generate the header information of n packets. Is prepared in the local memory 3 set in the packet generation information pointer designation register 104. In this embodiment, since the storage format of the packet generation information is arranged, the packet generation information is prepared in a logically continuous area on the local memory 3.

The header information requires a real address for identifying a remote node serving as routing information in an inter-processor interconnection network and an address in a local memory in the remote node. In the case of remote reading or remote writing with an acknowledgment, furthermore, The real address of the node to which the read data or acknowledge is to be returned and the address in the node are required.

When performing remote writing, data to be written on the local memory 3 is prepared. Here, it is assumed that each transmission packet has data of burst length L words, and that n transmission packets are generated by one activation.

Here, preferably, a burst length designation register (10) for designating the number of data words to be carried by the packet.
7) and a burst length designation register (10) before remote memory access is started from the external processor (2).
In 7), set the number of data words that the packet should carry,
The packet generation unit (100) may generate a remote memory burst access packet for reading and writing the number of words specified by the burst length specification register (107). In a configuration in which the burst length designation register 107 is not provided and L cannot be set, only one word can have a data portion, so that the transmission efficiency of the packet on the interconnection network between processors is reduced. In this case, block data specified by the burst length is carried for one packet header including a remote memory address, etc., so that remote memory access is always performed one word at a time. The memory bandwidth will be used effectively.

In this embodiment, the burst length designation register 1
07, the burst length designation register 10
L is set to 7. Data to be written is stored in a logically continuous area as an L × n two-dimensional array.
The head address of this two-dimensional array is set in the packet data part pointer designation register 106.

In this embodiment, the remote memory access is started by accessing the start address from the microprocessor 2 in the user mode.
Activation of the remote memory access from the user mode at the time of communication execution is realized by requesting the OS only once in advance so that the activation address in the user space is mapped to the actual physical activation address.

In the present embodiment, it is assumed that there are three types of starting addresses for remote memory access: starting for remote reading, starting for remote writing without acknowledgment, and starting for remote writing with acknowledgment.

When the bus interface unit 101 detects an access to the start address, the value of the packet number designation register 105 is copied to the remaining packet number counter in the memory address generation unit 102, and the memory address generation unit 102 Generation information pointer designation register 104, packet data portion pointer designation register 106 for writing, and burst length designation register 107
Based on the above information, generation of an access request for the local memory 3 is started.

The packet generation information pointer is specified by a logical address, and accesses the local memory 3 specified by the physical memory address obtained by converting the logical address by the TLB 103 in the control device. Is loaded with packet generation information. At this time, in the present embodiment, since the storage format of the packet generation information stored in the local memory 3 is arrayed, the packet generation information is stored in a logically continuous area, so that the linked list is traced. Even if the memory is not accessed again for generating the address as in the case, the address can be easily and quickly generated by the simple counter.

Similarly, the packet data section pointer is designated by a logical address, and when remote memory writing is started, the logical address is stored in the TLB in the control device.
The local memory 3 specified by the physical memory address converted by 103 is accessed, and the transmission packet data section is loaded into the transmission packet data section buffer 110.

At this time, the data to be written is L × n
Is stored in a logically continuous area as a two-dimensional array of addresses, so a simple counter makes it easy and fast, without having to access the memory again to generate addresses as in the case of following a linked list. It is possible to generate an address.

Here, preferably, a buffer (10) for temporarily storing packet generation information on the transmitting side.
9), a buffer (110) for temporarily storing write data is provided, and a local memory access unit for taking in the buffer from the local memory (3) and a set of corresponding packet generation information and write data are provided. The packet generation unit (100) that generates a remote memory write packet from the packet may operate in a pipelined manner. With this configuration, the packet generation information and the write data for a plurality of packets can exist in the remote memory access control device (1) at the same time. Read the write data of
For example, while the packet generation unit (100) generates the first packet, the memory access unit reads out the packet generation information for the fourth packet.
By performing the pipeline operation, the packet generation unit (100) can operate simultaneously with the memory access unit. Therefore, a remote memory write packet can be generated at short intervals. The remote memory write packet is a combination of a buffer for temporarily storing packet generation information and a buffer for temporarily storing write data unless the buffer becomes empty. A write packet can be generated.

Preferably, in the mechanism enabling the pipeline operation as described above, the local memory access unit makes continuous access to the local memory at high speed, and stores packet generation information and write data in the local memory. A plurality of packets may be fetched into the buffer by continuous access from the memory. In this case, the buffer (109) for temporarily storing the packet generation information and the buffer (110) for temporarily storing the write data respectively store the packet generation information and the write data for a plurality of packets. Since it allows loading from local memory (3) at once,
It is possible to capture a plurality of packets into the buffer at high speed while using a high effective memory bandwidth by continuous access.

FIG. 3 shows an arrangement of the storage format of the packet generation information stored in the local memory 3 in which the access to the local memory 3 and the generation of the packet can be performed in a pipeline manner. It is a figure showing a pipeline operation. One clock for address generation of array, three for memory access of one word
It is shown that it takes two clocks to generate a packet of one word of the data part with one word of the clock and the header part.

In this embodiment, the packet generator 100
It operates in parallel with the above address generation operation and memory access operation. In this example, the stage that is the bottleneck in the pipeline is the stage of memory access, and since two memory accesses are required to create one packet, packets are generated at a rate of one packet every six clocks. doing.

FIG. 4 shows an arrangement of the storage format of the packet generation information stored in the local memory 3 so that access to the local memory 3 and packet generation can be performed in a pipelined manner.
FIG. 7 is a diagram showing a pipeline operation at the time of packet transmission when accessing the. One clock for generating the address of the array, two clocks for the delay of random memory access, and thereafter, the memory read transfer rate for continuous access is one clock per word, and one word for the header part and one data part for the header part
It is shown that it takes two clocks to generate a word packet.

When a DRAM having a nibble mode or a page mode or a synchronous DRAM is used as a memory, the memory bandwidth is improved by performing continuous access. By accessing the header (packet generation information) for a plurality of packets and the data portion for a plurality of packets in a block shape, the memory bandwidth is improved. For example, if four packets are operated as one block, four packets can be generated in 12 clocks. , It is possible to generate packets at twice the rate as in the case of FIG.

Here, the memory address generation unit 102
If the load request is issued too much to be received by the buffers 109 and 110, the external bus 4 accessing the local memory 3 is occupied while waiting for a long time until data can be received. Since the memory access is delayed, it is desirable to generate a read request to the local memory 3 in consideration of the remaining capacity of the packet generation information buffer 109 and the transmission packet data buffer 110.

In this embodiment, when remote reading is started, the value of the packet number designation register 105 is copied to the remaining packet number counter in the packet generation unit 100, and the packet generation unit 100 109, the number of generated packets reaches the value at the time when remote reading of the packet number designation register 105 is started (the remaining packet number counter is 0).
), A packet is generated while incorporating the value on the process ID designation register 108 and the value on the burst length designation register 107 into the packet header.

FIG. 5 is a diagram showing an example of a remote read request packet. As shown in (a), a code indicating a remote read request is written in the packet type field of the first word of the header, and a process ID, a burst length, a destination physical PE number obtained from packet generation information, and a destination The remaining fields are configured by the logical memory addresses in the nation PE. (B)
In the second word, the own PE number for response packet generation and the logical memory address in the response data write destination PE obtained from the packet generation information are written in the second word.

The packet generator 100 converts the above-mentioned packet into the transmission packet buffer 111 (if necessary).
Via the network interface 117 between processors
, Decrements the remaining generated packet number counter in the packet generating unit 100, and increments the response waiting read request number counter 114. When the remaining generated packet number counter becomes 0, the packet generator 100 sets a bit in the status register 122 to notify the completion of packet generation.

On the other hand, when remote writing is started,
The value of the packet number designation register 105 is copied to the remaining generated packet number counter in the packet generation unit 100, and the packet generation unit 100
9 and the transmission packet data buffer 110 both have data, and the number of generated packets reaches the value at the time when remote reading of the packet number designation register 105 is started (the remaining generated packet number counter becomes 0). Until,
A packet is generated by adding the transmission packet data portion while incorporating the value on the process ID designation register 108 and the value on the burst length designation register 107 into the packet header.

FIG. 6 is a diagram showing an example of a remote write request packet header with an acknowledgment. As shown in (a), a code indicating a remote write request with an acknowledgment is written in the packet type field of the first word of the header, and the destination physical PE obtained from the process ID, burst length, and packet generation information. The remaining fields are constituted by the numbers and the logical memory addresses in the destination PE. As shown in (b), the own PE number is written in the second word for generating the response packet.

FIG. 7 is a diagram showing an example of a remote write request packet header without acknowledgment. A code indicating a remote write request without acknowledgment is written in the packet type field, and the remaining fields are represented by a process ID, a burst length, a destination physical PE number obtained from packet generation information, and a logical memory address in the destination PE. Be composed.

The packet generation unit 100 passes the packet to which the packet header as described above is added to the inter-processor connection network interface 117, and the packet generation unit 100
Is decremented.
Here, in the case of remote writing with an acknowledgment, the response waiting write request number counter 115 is further incremented. When the remaining generated packet number counter becomes 0, the packet generator 100
A bit for notifying the completion of packet generation is set.

As described above, in the present embodiment, the stage for loading data into the buffer on the local memory 3 and the stage for generating packets operate simultaneously in a pipeline manner. Therefore, it is possible to generate packets at a higher rate than in a method in which local memory access and packet generation are performed sequentially.

Next, reception will be described. The packet receiving unit 113 receives various packets from the inter-processor connection network via the reception packet buffer 112, and performs an operation corresponding to each packet. For example, three of the received packets are the remote read request packet, the remote write request packet with acknowledgment, and the remote write request packet without acknowledgment. In addition, there are two types of remote read response packets and remote write response packets.

Hereinafter, each of these five types will be described. When the packet receiving unit 113 receives the remote read request packet, the process ID, the burst length, and the read start address (virtual address) present in the packet header are passed to the memory address generating unit 102.
The process ID, the address of the node to which the response should be returned (source physical PE number), and the logical memory address in the response data write destination PE are passed to the packet generation unit 100.

The memory address generation unit 102 checks the access right based on the process ID using the TLB 103 in the control device, and then generates the physical address of the read start address, and transmits the data from the local memory 3 through the bus interface unit 101. Is read out and stored in the reply packet data part buffer 121.

The packet generator 100 generates a header from the process ID received from the packet receiver 113, the source physical PE number, and the logical memory address in the response data write destination PE, while the data is input to the reply packet data buffer 121. Then, a packet is added together with the header, and the processor
Pass to 7.

When the packet receiving section 113 receives the remote write request packet without acknowledgment, the memory address generating section 102 stores the process ID, burst length, write start address (virtual address) and write data existing in the packet header. hand over.

The memory address generation unit 102 uses the TLB 103 in the control device to check the access right based on the process ID and generates the physical address of the write start address, and writes the write data to the local memory 3 via the bus interface unit 101. .

Here, preferably, a counter for counting the number of times of completion of the processing for the received remote write request may be provided. When performing a process whose number of times of writing from the remote node is known by the program to be written, the value of the counter counted by the receiving side is returned to the receiving side without returning a response packet to the remote writing. Checking that the processing of the write packet of a predetermined number of times is completed, and then performing barrier synchronization as a whole, so that the completion of the write is immediately transmitted to the side that transmitted the remote write packet, so that the speed is high. The burden on the inter-processor connection network is also reduced.

In the present embodiment, the reception write processing completion number counter 116 is provided. When this counter is in the operating state, the writing of 3 to the local memory accompanying the remote write request received by the bus interface unit 101 is completed. Each time the operation is completed, the reception write processing completion number counter 116 is decremented. The counter 116 is initialized by setting the number of remote writes expected from the microprocessor 2 in advance in a state where the reception / write processing is temporarily stopped, releasing the suspension of the reception / write processing, and completing the reception / write processing. The counter 116 is set to the operation state and waits for the arrival of the remote write request packet. When the reception / write processing is completed the set number of times, the reception / write processing completion number counter 116 becomes 0.
And the fact that the specified number of times has been reached is reflected in the status register 122. The microprocessor 2 recognizes that its own PE has received all the data to be received by polling this value or receiving an interrupt, and when all the PEs are in such a state, the microprocessor 2 proceeds to the next phase. If it is necessary to proceed, by applying barrier synchronization, it becomes possible to synchronize write completion at high speed without a response packet related to write.

When the packet receiver 113 receives the remote write request packet with acknowledgment, the memory address generator 102 stores the process ID, burst length, write start address (virtual address) and write data existing in the packet header. On the other hand, the process ID and the address of the node to return the response (source physical PE number) are passed to the packet generation unit 100.

The memory address generation unit 102 uses the TLB 103 in the control device to check the access right based on the process ID and generate the physical address of the write start address, and writes the write data to the local memory 3 via the bus interface unit 101. When the writing is completed, the packet generation unit 100 is instructed to transmit an acknowledge packet.

The packet generating section 100 generates a remote write acknowledge packet from the process ID and the source physical PE number received from the packet receiving section 113, and transfers the packet to the inter-processor connection network interface section 117.

When the packet receiving unit 113 receives the remote read response packet, it performs almost the same operation as the remote write request packet without acknowledgment, and decrements the response wait read request number counter 114 instead of the response wait write request number counter 115. I do. When the response waiting read request counter 114 becomes 0, a bit indicating the completion of the remote read operation of the status register 122 is set.

The process ID present in the packet header is compared with the value of the process ID designation register 108. If the values do not match, a context switch occurs before an acknowledge is returned, and the decrement is performed. Response waiting read request number counter 11 to be performed
4 is saved on the local memory 3 and interrupts the external microprocessor 2 to notify the OS that a remote write response packet for the saved process has been received. The OS reads the corresponding response waiting read request number counter 1 saved on the local memory 3.
Decrement 14 to return the context to the state before interrupt.

When the packet receiving unit 113 receives the remote write response packet, the process ID existing in the packet header is compared with the value of the process ID setting register. Is decremented. When the response waiting write request number counter 115 becomes 0, the status register 1
A bit indicating the completion of the remote write operation with acknowledgment 22 is set.

If the process ID existing in the packet header and the value of the process ID setting register do not match, a context switch occurs before the acknowledgment returns, and should be decremented. The response waiting write request number counter 115, which has been saved on the local memory 3, interrupts the external microprocessor 2 to notify the OS that a remote write response packet for the saved process has been received. The OS decrements the corresponding response-waiting write request counter 115 saved on the local memory 3 to return the context to the state before the interruption.

Next, an example of using the remote memory access control device of this embodiment on a parallel computer will be described. FIG.
FIG. 11 is a diagram illustrating an example of a mapping situation having a mixture of jobs of a plurality of users and substitution of a faulty PE in a two-dimensional mesh-connected parallel computer including × 9 = 72 element processors (PEs). PE0 to PE31 are assigned to the process 1. However, when trying to assign a 4 × 4 logical PE, the PE46 has failed.
E39, PE44, PE45, PE47, PE52-P
E55, PE60 to PE63, and PE70 are allocated to the process 2.

FIG. 9 is a diagram showing a logical group of PEs in the process 2. The numbers in parentheses are physical PE numbers, and the numbers outside of parentheses are logical PE numbers. Logical PE number is 0-1
Although the number is consecutive up to 5, the corresponding physical PE number is not always consecutive due to the position assigned as in Process 2 or the presence of a faulty PE.

FIG. 10 is a diagram showing a state of address conversion in process 2. In a virtual address (called a local logical address) in the transmission-side PE of the process 2, memories distributed on each PE are mapped to logically continuous areas.

However, since the physical PE numbers are not always consecutive as shown in FIG.
In a memory space (global intermediate address) in which only the E address portion is a physical address, the local logical address is converted into an irregularly discrete memory area corresponding to the discrete physical PE number irregularly. Treated as part of the header information in the outgoing and incoming packets. The connection network performs message routing based on the physical PE number in the packet header.

In the case where multi-user operation is performed in a time-division manner on each PE or virtual storage is locally performed in order to make it appear that there is more memory than a locally mounted semiconductor memory, a global intermediate The in-PE memory address expressed as a bit excluding the PE address part in the address is converted from the in-PE logical memory address to the in-PE physical memory address on the receiving PE.

In this embodiment, this conversion is realized using the TLB in the control device on the receiving side. Since the address conversion on the receiving side is an address conversion for the memory at hand, locality can be used, and TLB is effective.

On the other hand, which physical PE number is mapped in the global shared memory space should be programmed by the user without directly knowing it on the local logical address space. This is information to be managed by the OS when execution is performed, and address conversion is required. This address conversion is performed, for example, in all-to-all personalized communication in which small amounts of separate data are exchanged among all PEs.
communication).
Since the access is such that the logical PE number changes frequently, if the number of entries in the TLB is smaller than the number of the communicating PEs, the TLB becomes a series of mishits, which has an adverse effect.

FIG. 11 shows that the 16 × 16 square matrices A and B are divided in the column direction by 16 PEs and held one column at a time. It is a figure showing a communication pattern. A (k, 2) held in PE2 is assigned to B (2, k), respectively.
Here, k is an integer of 0 to 15. B (2, k) is 16
Since the PEs are divided into a plurality of PEs, the assignment operation is all communication to different PEs. Similar communication is performed for all other PEs.
, This processing is all-to-all communication. If the number of assigned PEs is about 16 as in this example, the number of entries in the TLB in the ordinary microprocessor is smaller, so that TLB can be expected to be effective even if all-to-all communication is performed. If it is assumed that all-to-all communication is repeatedly performed in a process using more than 100 PEs on a highly parallel computer or a massively parallel computer with a In a method of converting a PE address from a logical PE address to a physical PE address using a TLB having only the TLB, the TLB entry is not reused and TLB mishits occur repeatedly.

Therefore, preferably, the packet generator (1)
00), a protection information adding mechanism that cannot be directly controlled from the program in the non-privileged mode without shifting to the privileged mode is provided, and the remote memory access packet receiving side is provided with a protection information adding mechanism based on the protection information in the packet. In addition to providing a memory access protection mechanism for controlling the permission / non-permission of memory access, a translated address prepared by a system call on the transmission side as an address for specifying the remote memory is prepared in the local memory (3), and the actual remote address is prepared. At the time of memory access, the above-mentioned converted address may be reused.

That is, since the protection information can be rewritten from the privileged mode, when the processor is context-switched and switched to another user's process, the operating system is now ready for the external processor (2) or remote memory access controller (1). When a remote memory access is performed in the non-privileged mode, the packet generator embeds this information in the header of the packet.

As for the address designating the remote memory, the translated address obtained by performing the translation by the system call on the transmitting side is prepared in the local memory (3), so that the translated address on the local memory (3) is stored in the non-privileged mode. There is a risk of being rewritten. However, erroneous writing to someone else's remote memory is eliminated by checking the protection information in the packet header with the receiving memory access protection mechanism.

At the time of actual remote memory access, by reusing the above-mentioned converted address, the address conversion does not have to be performed each time communication is performed, so that the TLB and system calls are not frequently generated on the transmission side. .

In the case of a remote memory access, the destination processor number will form part of the address of the remote memory, but these are usually different processors which require different TLB entries, so In the method in which the address translation by the TLB is required one by one, if there are many destination processors, TLB mishits will occur repeatedly.
According to the above, even if the communication is repeated many times, the conversion is performed only once in advance, so that there is no such a problem.

In this embodiment, as described above, the OS is called only once before communication starts, and the global shared memory address (local logical address) to be accessed is changed to the address (global intermediate address) capable of identifying the real node. It is effective to have the local memory 3 store the packet generation information obtained based on that information in the local memory 3 and to reuse that information. Because FFT
When the access destination does not fluctuate dynamically and is repeatedly accessed in a loop many times, as is often used in scientific and technical calculations such as matrix transposition and matrix transposition, a microprocessor is used for remote memory access. Internal TL
A different page allocated on a different processor without using the TLB in the control unit for transmitting a packet, without using a TLB for translation of a remote memory address, and without causing a trap when executing a remote memory access. This is because it is possible to generate a remote memory access packet to many remote access destinations belonging to.

FIG. 12 is a diagram showing an array of write destination pointers before address conversion in the matrix transposition shown in FIG. 11 in PE2. The upper part of the address is a logical PE number, which is different for each word. The lower part of the address is the logical memory address in the PE. If the elements of the matrices A and B consist of 8 bytes of data, this value is
Is obtained by adding 8 bytes × the second element = 16 to the base address in each PE.

FIG. 13 is a diagram showing an array of write destination pointers after address conversion at the time of transposing the matrix shown in FIG. 11 in PE2 in the process 2 mapped in FIG. The upper part of the address is converted from a logical PE number to a physical PE number. This arrangement is such that, after preparing FIG. 12, the OS is executed only once before starting communication as described above.
Is obtained by calling This array corresponds to the packet generation information.

In this embodiment, the high-order address is converted into the physical PE number itself assuming that the inter-processor interconnection network performs routing using an absolute address. However, the inter-processor interconnection network performs routing using a relative address. In order to facilitate the routing, it is desirable that the address be converted into a relative address calculated from the PE address of the writing source and the PE address of the writing destination, and be used as it is as the routing information.

Next, an example of the operation when the communication shown in FIG. 11 is performed will be described. In order to perform this operation, first, an array A of 16 pieces of 64-bit data for transmission, an array B of data for reception, and an array C of packet generation information shown in FIG. 13 are prepared in the memory.

“16” is set in the reception write processing completion counter 116 and “16” is set in the packet number designation register 105.
Is stored in the burst length designation register 107 as “2” (32 bi).
t means 2 words), the “header of array A” is stored in the packet data section pointer specification register 106, and the packet generation information pointer specification register 10
4, the "start address of array C" is set from the microprocessor 2 and the "remote write request address without acknowledgment" is accessed from the microprocessor 2 to start the remote write operation.

The packet generating section 100 generates 16 packets having a header and a 64-bit data portion as shown in FIG. When all 16 packets have been generated, a bit indicating the end of packet generation is set in the status register 122.

The same operation is performed in other PEs other than PE2, and PE2 receives the above-mentioned packets one by one from 16 PEs including itself. In turn, when the data portion of the 16 remote write request packets without acknowledgment is written to the physical memory address obtained by converting the logical memory address in the PE included in the header portion by the TLB 103 in the control device, the number of times of completion of the reception write process The counter 116 becomes “0” and the status register 122
The remote write operation is completed by setting the bit of the completion of the receive write operation.

The microprocessor 2 polls the completion of the reception and writing, and when detecting that the reception and writing completion bit of the status register 122 is set, executes the barrier synchronization. When all the PEs execute the barrier synchronization, all-to-all communication is completed, and a matrix B obtained by transposing the matrix A is obtained.

In the above operation example, the realization is performed by using remote writing without acknowledgment. However, the same can be realized by using remote writing with acknowledgment or remote reading. In this operation example, since the write destination and the number of times of writing are statically determined, the use of remote writing without acknowledgment is the fastest mode of implementation because no response message is required.

(Modification 1) Next, a modification of the first embodiment will be described. In the first modification, in the remote memory access control device of the first embodiment, as the packet generation information stored in the local memory 3, the number of data words to be carried by the packet is prepared in addition to the memory address. A remote memory burst access packet for reading and writing the continuous data of is generated.

When generating a remote memory access packet, the packet generation section 100 refers to the number of data words (burst length) to be carried by the packet in the packet generation information, and reads the burst length in the header if it is read. A remote memory read request packet including a remote memory write request packet is generated based on the data read from the local memory for the burst length, based on the data read from the local memory. Since the burst length is included in the packet generation information for each packet, it is possible to access with a different burst length depending on the access destination.

FIG. 14 is a diagram for explaining this modification, and shows an example of an array corresponding to packet generation information having a burst length that can be specified for each message. Although the value on the burst length designation register 107 in the remote memory access control device 1 in FIG. 1 has been used for generating all packets, when generating a packet as in the example here, FIGS. The individual burst length in the packet generation information in FIG. 14 is used as the burst length in the packet header in FIG.

For example, the packet corresponding to the first element in the packet generation information in FIG. 14 generates a packet for writing or reading two words of data for the PE having the physical PE number 36, and corresponds to the second element. The packet generates a packet for writing or reading three words of data for the PE having the physical PE number 37.

Whether the packet generation information in the format of FIG. 7 or the packet generation information in the format of FIG. 14 is used is determined by mapping an address for starting remote access to another address. Can be distinguished.

(Embodiment 2) First, the main configuration of this embodiment and the operation and effect thereof will be described. <Main Configuration of Second Embodiment> In this embodiment, a distributed shared memory (2) that is physically distributed and is logically accessible as a shared memory from a plurality of vector processors (25).
In the parallel computer having 3), information corresponding to the number of generated packets is specified by the vector length specifying register (236), and a part or all of the packet generation information for generating a packet for accessing the remote memory is desired. Are stored in a local memory (23) outside the remote memory access control device (21) and at hand based on a predetermined format, corresponding to the number of packets.
The vector processor (25) loads the packet generation information from the local memory (23) into the vector register (234) by the vector instruction fetched, and uses them to execute the remote memory access packet by the vector instruction fetched by the vector processor (25). Generate

<Main Functions and Effects of Second Embodiment> The remote memory access control device (21) of this embodiment is realized as a part of the function of the vector processor (25) in a distributed shared memory type parallel computer. First, part or all of the packet generation information for generating a packet for accessing the remote memory is stored in the local memory (23) based on a predetermined format in an amount corresponding to a desired number of packets. . Next, the vector register (23) is read from the local memory (23) by the vector load instruction fetched by the vector processor (25) itself.
In 4), loading of packet generation information is started. next,
With the information corresponding to the number of generated packets set in the vector length designation register (236), the vector processor (25) activates the vector remote memory access instruction fetched.

In this way, the memory access unit and the packet generation unit (200) operate in a pipeline simultaneously, and remote memory access packets are generated one after another by one vector remote memory access instruction. After notifying the vector processor of the permission to start the remote memory access, the external processor (22) can perform another processing without being busy for the processing of the remote memory access. Also, since the packet generation information does not pass through the cache (41) of the external processor (22),
The hit rate of the cache (41) of the external processor (22) is also improved.

Further, since the result from the vector operation unit can be used as the data of the remote message without going into and out of the memory when it comes out, high-speed remote memory access avoiding a memory access bottleneck is realized.

Hereinafter, the present embodiment will be described more specifically. <Specific Description of Second Embodiment> FIGS. 15 and 16 are diagrams showing a configuration of an element processor of a parallel computer having the coprocessor of the present embodiment.

A parallel computer is realized by interconnecting a plurality of element processors by an inter-processor connection network. In this embodiment, the interconnection scheme of the element processors is arbitrary.

As shown in the figure, in each element processor, a coprocessor 21 in which a remote memory access control device and a vector processor are integrated, a main processor 22 realized by a general-purpose microprocessor, and a local memory 23 are connected by an external bus 24. It is connected. The remote memory access control device 21 is directly connected to the inter-processor connection network. In this embodiment, the local memory 23 of each element processor is used as a distributed shared memory.

As in the first embodiment, the microprocessor 2
2 includes a cache memory 41 and a TLB 42 for address conversion in addition to the CPU 43. Further, the main processor 22 accesses the internal resources of the local memory 23 and the vector processor 21 via the external bus 24. In the present embodiment, an OS and a part of a user program operate on the main processor 22, and the vector processor 21 is positioned as a coprocessor.

However, unlike the present embodiment, it is not necessary to implement the combination of the main processor (22) and the coprocessor (21) implemented by a general-purpose microprocessor having a TLB and a cache memory. It may be realized by a scalar processor and a vector processor which are more tightly coupled than in this embodiment by design.

As shown in FIGS. 15 and 16, the coprocessor 21 includes a vector operation pipeline 232, a load store pipeline 233, a vector register 234, a scalar register 235, a vector length designation register 236,
Status register 222, process ID designation register 208, instruction buffer 241, instruction control unit 242, vector processor connection network 243, bus interface unit 201, inter-processor connection network interface unit 217,
A packet generation unit 200, a packet reception unit 213, a transmission packet buffer 211, a reception packet buffer 212,
Response waiting read request counter 214, response waiting write request counter 215, reception / write processing completion counter 216, TLB 203 in coprocessor, reply packet data section buffer 221, memory address generation section 20
2 is provided.

In the local memory 23, the main body of the page table to be cached in the TLB 42 in the main processor and the TLB 21 in the coprocessor, the program for the vector processor 21 and the main processor 2
2 and other data such as an array of packet generation information and a transmission packet data section.

Here, preferably, the packet generation unit (2
00), a protection information adding mechanism that cannot be directly controlled from the program in the non-privileged mode without shifting to the privileged mode is provided, and the remote memory access packet receiving side is provided with a protection information adding mechanism based on the protection information in the packet. It is preferable to provide a memory access protection mechanism for controlling permission / denial of memory access. Since the protection information can be rewritten from the privileged mode, when the processor context switches and switches to another user's process, the operating system sets a process ID to allocate resources such as an external processor and a remote memory access control device. When the remote memory is accessed in the non-privileged mode, the packet generator embeds this information in the header of the packet. In other words, even if the information for packet generation on the local memory (23) is rewritten in the non-privileged mode, if it is erroneously written to the remote memory of another person, the protection information in the packet header is checked by the memory access protection mechanism on the receiving side. Will be eliminated by

In this embodiment, it is assumed that the above-described mechanism is adopted, and the TLB 203 in the coprocessor and the process I
D designation register 208 is mapped to an area that can be accessed only in the privileged mode of the microprocessor.

The in-coprocessor TLB 203 holds a process ID assigned to each entry of the TLB 42 in addition to a set of a physical address and a logical address used for address conversion at the time of memory access generated by the memory address generation unit 202. I do. The entity is located in an area of the local memory 23 which can be accessed only in the privileged mode, and TL
The information is cached in B42. Here, the process is a unit for allocating a virtual address space, and there are a plurality of threads using the same process ID across a plurality of processors.

The process ID designation register 208 holds the process ID assigned to the coprocessor 21, and the header of the packet generated by the packet generation unit 200 has the process ID on the process ID designation register 208 embedded therein. It is. Thus, the TLB 203 in the coprocessor and the process ID designation register 208
Direct access is not allowed from the user mode because it is a mechanism that holds protection information and prevents unauthorized memory access.

The vector processor in the coprocessor 21 reads the coprocessor program in the local memory 23 into the instruction buffer based on the instruction from the main processor 22 and decodes the instruction in the instruction buffer 241. Then, the vector operation pipeline 232, the load store pipeline 233,
A control signal is sent to the scalar register 235, the vector register 234, and the like, and operates independently from the main processor 22 unless a synchronization command or interrupt processing occurs.

The instructions of the coprocessor 21 include an instruction to load data from the local memory 23 and the instruction buffer 241 to the scalar register 235 and the vector length designation register 236, and a vector load to load data from the local memory 23 to the vector register 234. Instruction VL
D and the local memory 23 from the vector register 234.
, A vector operation instruction for executing an operation such as addition or multiplication between values on the vector register 234 and the scalar register 235 and storing the result in the vector register 234, and a vector transmission instruction described later. There are instructions.

The connection network 243 in the vector processor is a data transfer path between resources in the vector processor controlled by the operand specified by the vector instruction. Here, it is preferable that the storage format of the packet generation information stored in the local memory (23) be an array. In such a case, the packet generation information can be easily and quickly loaded into the vector register by the vector load instruction.

In this embodiment, the storage format of the packet generation information stored in the local memory 23 is an array. In the case of performing interprocessor communication by the coprocessor 21 of the present embodiment, first, in the case of performing remote access, it is necessary to load packet generation information into the vector register 234 in both writing and reading. In this embodiment, the storage format of the packet generation information is arrayed, and information necessary for generating header information of n packets is first stored in the local memory 2.
3 is prepared as an array on a continuous area.

Here, preferably, the packet generation unit (2
00), a protection information adding mechanism that cannot be directly controlled from the program in the non-privileged mode without shifting to the privileged mode is provided, and the remote memory access packet receiving side is provided with a protection information adding mechanism based on the protection information in the packet. A memory access protection mechanism for controlling permission / rejection of memory access is provided, and a translated address prepared by a system call on the transmission side as an address specifying a remote memory is prepared in a local memory,
At the time of actual remote memory access, the address after the conversion may be reused.

That is, since the protection information can be rewritten from the privileged mode, when the processor performs a context switch to switch to another user's process, the operating system can now allocate resources such as an external processor and a remote memory access control device. When a remote memory access is performed in the non-privileged mode, the packet generator embeds this information in the header of the packet when an ID or the like is set.

As for the address for specifying the remote memory, the translated address obtained by performing the translation by the system call on the transmitting side is prepared in the local memory. Therefore, there is a risk that the translated address on the local memory is rewritten in the non-privileged mode. is there. However, erroneous writing to someone else's remote memory is eliminated by checking the protection information in the packet header with the receiving memory access protection mechanism.

At the time of actual remote memory access, by reusing the address after the above-mentioned conversion, the address conversion does not have to be performed every time the communication is executed. Therefore, it is possible to eliminate the frequent occurrence of TLB and system calls on the transmission side. it can.

In the case of remote memory access, the processor number of the destination forms part of the address of the remote memory, and these usually require different TLB entries if another processor is used. In a method in which the address translation by the TLB is required one by one, if there are many destination processors, TLB mishits occur continuously.
According to the above, even if the communication is repeated many times, the conversion is performed only once in advance, so that there is no such a problem.

In this embodiment, as in the first embodiment,
After preparing FIG. 7, before starting communication as described above, 1
The array of FIG. 8 obtained by calling the OS only once corresponds to the packet generation information.

The start address of the array corresponding to the packet generation information is set in the scalar register SR1, the number of packet generation information data to be loaded, that is, the number of packets n is set in the vector length designation register 236, An access step number 1 indicating that access is performed is set in the scalar register SR2, and a vector load instruction (VLD SR1) for the vector register VR1 is set.
SR2 VR1).

Then, the vector length, the start address, and the number of access steps are transferred to the load / store pipeline 233, where a memory address is generated, an access request reaches the bus interface unit 201, and the result of reading the memory 23 is stored in the vector processor. Via the connection network 243, n packets are loaded into the vector register VR1 as packet generation information.

In this embodiment, the storage format of the packet generation information is arrayed, and the information necessary to generate the header information of the n packets is prepared in the local memory 23 as an array. The packet generation information is efficiently loaded into the vector register 234 by the load instruction.

When remote writing is performed, it is necessary to further prepare write data in the vector register 234. Here, it is assumed that each transmission packet has data of a burst length of L words and that n transmission packets are generated by one activation.

When remote writing is performed, data to be written is prepared on the local memory 23 and a vector load instruction is loaded into the vector register 234 by executing a vector load instruction by a vector processor.
Vector register 2 only if the result of operation by the vector processor can be used directly as transmission data
34, there is a method using the calculation result.

First, the operation in the case of executing by the former method will be described. In FIG. 15 and FIG. 16, data to be written is stored in a logically continuous area as an L × n two-dimensional array. Since an instruction for performing a list access is generally prepared, data need not always be stored in a continuous area.

In this example, the start address of the two-dimensional array is set in the scalar register SR3, the number L × n of data to be loaded is set in the vector length designation register 236, and the number of access steps indicating that continuous access is to be performed is 1 Is set in the scalar register SR2, and the vector load instruction (VLD SR3 SR2V
When R2) is executed, transmission packet data for n packets is loaded into the vector register VR2.

Preferably, the information for packet generation is stored in the first vector register, and the load data from the local memory or the write data obtained as an output from the vector operation unit is stored in the second vector register. Alternatively, the remote memory access control device itself may generate a remote memory write packet by a vector instruction fetched by combining the corresponding packet generation information and write data. Since the write data is stored in the vector register, the result from the vector calculator is held in the vector register. If it is specified as the source operand of the vector remote memory write instruction, the packet generation unit combines the packet generation information loaded from the local memory into another vector register without passing through the memory to generate the remote memory write packet. Generated one after another.

Preferably, the remote memory burst access packet includes a vector instruction capable of designating the number of data words (burst length) to be carried by the packet, and reads and writes continuous data of one or more words in one packet. It may be generated. The burst length can be specified by, for example, a general-purpose scalar register or a dedicated register, and the vector remote memory burst access instruction is started with the value set therein. Then, the packet generation unit generates a remote memory burst access packet for reading and writing the number of words specified by the register. In this way, block-like data specified by the burst length is carried for one packet header including a remote memory address, etc., so that remote memory access is always performed one word at a time. And the memory bandwidth is effectively used.

In this embodiment, the vector instructions VSENDW and VSENDR for generating a remote access request packet using the packet generation information on the vector register 234
The VSENDW generates a remote write request packet from the packet generation information on the vector register 234 and the transmission data.

In particular, the instruction VSENDW of the present embodiment
Is the burst length L set on the scalar register 235
To generate a packet requesting a remote memory burst write by attaching one header to the continuous L word data on the vector register 234.

The difference between VSENDW and VSENDR is that the packet generation information VSENDW is used for remote writing consisting of one word, whereas the write data on the vector register 234 is appended with L words to form a transmission packet.
The SENDR is different in that the information for packet generation is composed of two words per packet, has no data part, and is used for a remote read request requiring a read result storage address.

The packet generation information and the transmission data are loaded into the vector registers VR1 and VR2, respectively.
When remote writing is performed using the ENDW instruction, the burst length L is set in the scalar register SR4, the number of packets n is set in the vector length designation register 236, and the vector transmission instruction (VSENDW SR4 VR1 VR2)
To generate n remote burst write request packets.

The above operation is shown in a time chart, which is similar to FIG. 4 which is a time chart at the time of the pipeline operation of the first embodiment. Here, one clock is used to generate the address of the array, the delay of random memory access is two clocks,
Hereinafter, it is assumed that the memory read transfer rate during continuous access is one clock per word, and two clocks are required to generate a packet of one word in the header part and one word in the data part.

As described above, in the method of loading the transmission data from the local memory, the effect of integrating the vector processor and the remote memory access control device is that the functions of the packet generation information buffer and the transmission packet data buffer are shared by the vector register. The point is that the hardware is used effectively.

On the other hand, as long as the operation result of the vector processor can be directly used as the transmission data, the one using the operation result on the vector register 234 is a bottleneck of the communication performance unlike the first embodiment. Since the number of times of memory access is greatly reduced, higher-speed communication becomes possible.

FIG. 17 is a diagram showing the transmitting-side operation of remote memory writing using the operation result by the vector processor directly as transmission data. Here, one clock is used for generating the address of the array, the delay for random memory access is two clocks, the memory read transfer rate for continuous access is one clock per word, and two packets are used for generating one word header part and one word data part. It is shown as a clock.

If the result data from the vector operation unit is written to the vector register before or in parallel with the memory access for the header, the packet generation can be started from the fifth cycle, and the packet transmission can be performed as compared with the case of FIG. Packet generation starts at a time earlier than the time required for data loading, and in this example, processing that took 17 cycles has been completed in 12 cycles.

Also, unlike the first embodiment, in this embodiment, the loading of the packet generation information and the packet generation are composed of different instructions. If the external bus 24 is free, it is possible to finish loading the packet generation information while the vector computing unit is calculating the data for transmission. In that case, the result from the vector computing unit As soon as the message appears, packet generation is started immediately, which may further increase the speed.

Here, preferably, a counter for counting the number of times of completion of the processing for the received remote write request may be provided. When performing a process whose number of times of writing from the remote node is known by the program to be written, the value of the counter counted by the receiving side is returned to the receiving side without returning a response packet to the remote writing. Checking that the processing of the write packet of a predetermined number of times is completed, and then performing barrier synchronization as a whole, so that the completion of the write is immediately transmitted to the side that transmitted the remote write packet, so that the speed is high. The burden on the inter-processor connection network is also reduced.

The header of the transmission packet is the same as in the first embodiment, and is as shown in FIGS. 5, 6, and 7. Also, the operations at the time of packet generation and reception, the operations of the response waiting read request number counter 214 and the response waiting write request number counter 215, and the operation of the reception write processing completion number counter 216 are the same as those of the first embodiment.

(Modification 2) Next, a modification of the second embodiment will be described. In the second modification, in the remote memory access control device of the second embodiment, the packet generation information stored in the local memory and loaded into the vector register includes:
In addition to the memory address, the number of data words (burst length) to be carried by the packet is prepared, and a remote memory burst access packet that reads and writes a different number of words for each one or more packets is generated.

When generating a remote memory access packet, the packet generation unit 200 refers to the number of data words (burst length) in the packet generation information and reads the remote memory access packet including the burst length in the header if reading is performed. However, if a request packet is generated and is to be written, a remote memory write packet included in the data portion is generated for the burst length based on the data in the vector register. Since the burst length is included in the packet generation information for each packet, it is possible to access with a different burst length depending on the access destination.

An array corresponding to packet generation information having a burst length that can be specified for each message is shown in FIG.
Can be used. FIG. 15 and FIG.
The value on the scalar register 235 for specifying the burst length in the coprocessor 32 of No. 6 has been used for generating all packets.
The individual burst length in the packet generation information of FIG. 14 is used as the burst length in the packet header of FIGS. 1, 12, and 13.

For example, the packet corresponding to the first element in the packet generation information of FIG.
A packet corresponding to the second element is written or read, and a packet corresponding to the second element generates a packet writing or reading three data words for the PE having the physical PE number 37.

Whether the packet generation information in the format of FIG. 7 is used or the packet generation information in the format of FIG. 14 is determined by mapping an address for starting remote access to another address. Can be distinguished. Further, the present invention is not limited to the above-described embodiments, and can be implemented with various modifications without departing from the scope of the invention.

[0195]

According to the present invention, information necessary for packet generation is stored in the local memory in advance, and a plurality of remote memory access packets are stored based on the information in response to one request from the processor. Since it is generated,
While realizing global multiple virtual storage, it is possible to eliminate an increase in cache misses due to a remote memory address generation from a processor and an interrupt generation due to a TLB mishit, and to provide a remote memory which can only be generated intermittently. Memory access packets can be generated continuously.

Further, the processor can perform another process after starting up the remote memory access control device without being busy with the remote memory access process as in the related art.

Further, since memory access to a plurality of packets can be performed at once, burst memory access is promoted, effective memory bandwidth is improved, and high-speed remote memory access is realized. In particular, when the packet generation information is arranged, high-speed memory access can be realized with a small amount of hardware.

When the vector processor and the remote memory access device are integrated, the number of times of memory access in the remote memory writing process can be reduced, and higher-speed remote memory writing can be realized.

Further, according to the present invention, a counter for counting the completed remote write requests is provided on the receiving side. Therefore, in the case of processing in which the number of writes on the receiving side can be statically determined, the completion of the writing is promptly completed. It is possible to know
The adverse effect that the connection network is congested by the response packet is eliminated.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of an element processor of a parallel computer including a remote memory access control device according to a first embodiment of the present invention;

FIG. 2 is a diagram showing a configuration of the element processor.

FIG. 3 is a diagram showing an example of a pipeline operation at the time of transmitting a packet;

FIG. 4 is a diagram showing another example of pipeline operation at the time of packet transmission.

FIG. 5 is a diagram showing an example of a remote read request packet;

FIG. 6 is a diagram showing an example of a remote write request packet header with an acknowledgment;

FIG. 7 is a diagram showing an example of a remote write request packet header without acknowledgment;

FIG. 8 is a diagram showing an example of a mapping situation in a two-dimensional mesh-connected parallel computer.

FIG. 9 is a diagram for explaining a logical PE group;

FIG. 10 is a diagram showing a state of address conversion.

FIG. 11 illustrates an example of a communication pattern.

FIG. 12 is a diagram showing an example of an array of write destination pointers before address conversion.

FIG. 13 is a diagram showing an example of an array of write destination pointers after address conversion.

FIG. 14 is a diagram showing an example of an array corresponding to packet generation information having a burst length that can be specified for each message;

FIG. 15 is a diagram illustrating a configuration of an element processor of a parallel computer including a coprocessor according to a second embodiment of the present invention.

FIG. 16 is a diagram showing a configuration of the same element processor.

FIG. 17 is a diagram illustrating a transmitting-side operation of remote memory writing that directly uses an operation result by a vector processor as transmission data.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Remote memory access control device, 2 ... Microprocessor, 3 ... Local memory, 4 ... External bus, 31 ... Cache memory, 32 ... TLB, 33 ... CPU, 100 ...
Packet generator 101, bus interface 10
2. Memory address generation unit 103: TL in control device
B, 104: packet generation information pointer designation register, 105: packet number designation register, 106: packet data part pointer designation register, 107: burst length designation register, 108: process ID designation register, 1
09: packet generation information buffer 110: transmission packet data buffer 111: transmission packet buffer 112: reception packet buffer 113: packet reception unit 114: response waiting read request number counter 1,
15: Response waiting write request number counter, 116: Receive / write processing completion number counter, 117: Inter-processor connection network interface unit, 121: Return packet buffer, 122: Status register, 21: Coprocessor, 22: Main processor, 23 ... Local memory,
24 external bus, 25 vector processor, 41 cache memory, 42 TLB, 43 CPU, 200
... Packet generation unit, 201 ... Bus interface unit, 2
02: memory address generation unit, 203: TLB in coprocessor, 208: process ID designation register, 211 ...
Transmission packet buffer, 212: reception packet buffer, 213: packet reception unit, 214: response wait read request counter, 215: response wait write request counter, 216 ... reception write processing completion number counter, 21
7: Inter-processor connection network interface unit, 221: Return packet data unit buffer, 222: Status register, 232: Vector operation pipeline, 233: Load store pipeline, 234: Vector register
235: scalar register, 236: vector length designation register, 241: instruction buffer, 242: instruction control unit, 2
43 ... connection network in vector processor

Claims (15)

(57) [Claims] 1. A parallel processor comprising a plurality of processors and a distributed shared memory physically mounted in a distributed manner and logically accessible as a shared memory from the plurality of processors. A remote memory access control device for controlling remote memory access, comprising: a pointer designating unit for designating an address of packet generation information; a packet number designating unit for designating the number of packets to be generated; Access request detecting means for detecting a memory access request; and, when the remote memory access request is detected, a predetermined format of a predetermined format from a local memory of the corresponding processor based on the address indicated by the pointer designating means. Local to read packet generation information Memory access means; and packet generation means for generating packets for remote memory access to the distributed shared memory by the number of packets specified by the packet number designation means based on the read packet generation information. A remote memory access control device comprising: 2. The remote memory access control device according to claim 1, wherein a storage format of the packet generation information stored in the local memory is arranged. 3. The apparatus according to claim 2, further comprising: first buffer means for temporarily storing the packet generation information; and second buffer means for temporarily storing write data. When the memory access request is a remote memory write request, the information is fetched from the local memory into the first and second buffer means, and the packet generation means determines that the remote memory access request is a remote memory write request. When a set of the corresponding packet generation information and the write data is completed, a remote memory write packet is generated based on the contents of the set, and the local memory access unit and the The packet generation unit is operated simultaneously in a pipeline manner. Remote memory access control apparatus according to claim 1. 4. The local memory access means accesses a local memory having a high speed of continuous access, and stores the packet generation information and the write data for a plurality of packets by the continuous access from the local memory. 4. The remote memory access control device according to claim 3, wherein the data is taken into the first and second buffer means. 5. A remote memory burst access packet for reading and writing the number of words specified by the burst length designating means, further comprising burst length designating means for designating the number of data words of information to be carried by the packet. The remote memory access control device according to claim 1, wherein: 6. A remote memory burst access packet for reading and writing continuous data having a desired number of words for each packet, wherein the information for packet generation has the number of data words of information to be carried by the packet in addition to the memory address. 2. The remote memory access control device according to claim 1, wherein the remote memory access control device generates 7. The packet generating means includes a protection information adding mechanism that cannot be directly controlled from a program in a non-privileged mode without shifting to a privileged mode, and a remote memory from another processor. 2. The remote memory access control device according to claim 1, further comprising a memory access protection mechanism for controlling permission / rejection of memory access based on protection information in the access packet when the access packet is received. 8. A distributed shared memory of a vector processor in a parallel computer comprising: a plurality of vector processors; a distributed shared memory mounted physically and logically accessible as a shared memory from the plurality of vector processors. A remote memory access control device for controlling remote memory access to a packet processor, wherein packet generation is started by a vector instruction fetched by a vector processor, and packet generation information loaded from a local memory to a vector register by a vector load means of the vector processor. A remote memory access control apparatus, comprising: packet generating means for generating packets for remote memory access to the distributed shared memory for the number of packets specified by the vector length specifying means based on the packet length. 9. A method for storing packet generation information in at least one of said vector registers, and writing write data loaded from said local memory or write data obtained as an output from a vector operation unit to another vector of said vector register. 9. The remote memory access control device according to claim 8, wherein a remote memory write packet is generated by the fetched vector instruction by combining the corresponding packet generation information and write data stored in a register. 10. The remote memory access control device according to claim 8, wherein a storage format of said packet generation information stored in said local memory is arranged. 11. A remote memory burst access packet in which a vector instruction capable of specifying the number of data words of information to be carried by the packet is provided, and a predetermined number of words of continuous data are read and written in one packet. Item 9. A remote memory access control device according to item 8. 12. A remote memory burst access packet for reading and writing a desired number of words of continuous data for each packet, wherein the packet generation information has the number of data words of information to be carried by the packet in addition to the memory address. 10. The remote memory access control device according to claim 8, wherein the remote memory access control device is configured to generate 13. The packet generating means is provided with a protection information adding mechanism that cannot be directly controlled from a program in a non-privileged mode without shifting to a privileged mode, and a remote memory from another processor. 9. The remote memory access control device according to claim 8, further comprising a memory access protection mechanism for controlling permission / rejection of memory access based on protection information in the access packet when the access packet is received. 14. A translated address prepared by a system call on the transmission side as an address for designating a remote memory is prepared in the local memory, and the translated address is reused in actual remote memory access. 14. The remote memory access control device according to claim 7, wherein the control is performed. 15. A parallel processor comprising a plurality of processors and a distributed shared memory physically mounted in a distributed manner and logically accessible from the plurality of processors as a shared memory. A remote memory access control device for controlling remote memory access, comprising: access request detection means for detecting a remote write request from the corresponding processor; and generating a corresponding packet in response to the remote write request; Means for transmitting to a processor of the same; means for receiving a packet corresponding to the remote write request from a processor other than the corresponding processor; and a corresponding local memory according to the contents of the packet corresponding to the remote write request. Means for writing data to the Remote memory access control apparatus characterized by comprising a counter means for counting the number of times of completion of the processing for the packet corresponding to the request write attempts.