CN116795647A - Method, device, equipment and medium for managing and scheduling heterogeneous resources of database - Google Patents
Method, device, equipment and medium for managing and scheduling heterogeneous resources of database Download PDFInfo
- Publication number
- CN116795647A CN116795647A CN202310777226.1A CN202310777226A CN116795647A CN 116795647 A CN116795647 A CN 116795647A CN 202310777226 A CN202310777226 A CN 202310777226A CN 116795647 A CN116795647 A CN 116795647A
- Authority
- CN
- China
- Prior art keywords
- acceleration
- kernel
- target
- data
- communication bandwidth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000001133 acceleration Effects 0.000 claims abstract description 194
- 238000004891 communication Methods 0.000 claims abstract description 88
- 238000013500 data storage Methods 0.000 claims abstract description 37
- 238000012544 monitoring process Methods 0.000 claims abstract description 23
- 238000007781 pre-processing Methods 0.000 claims abstract description 17
- 230000015654 memory Effects 0.000 claims description 58
- 230000006870 function Effects 0.000 claims description 21
- 238000007726 management method Methods 0.000 claims description 20
- 238000012545 processing Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004458 analytical method Methods 0.000 claims description 14
- 230000006872 improvement Effects 0.000 claims description 10
- 239000002699 waste material Substances 0.000 abstract description 8
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 7
- 238000000354 decomposition reaction Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device, equipment and a medium for managing and scheduling heterogeneous resources of a database, which relate to the technical fields of databases and heterogeneous computing and comprise the following steps: receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; the method comprises the steps that a data storage address is sent to a target acceleration kernel, so that the target acceleration kernel reads data to be operated on the basis of the data storage address and carries out operation; and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result. When the acceleration kernel is called, the corresponding quantity of acceleration kernels are deployed in a self-adaptive mode through monitoring the idle communication bandwidth on the data acceleration card, so that the communication bandwidth is utilized to the maximum extent, the data exchange efficiency is improved, and the heterogeneous resource waste is reduced.
Description
Technical Field
The present application relates to the field of databases and heterogeneous computing technologies, and in particular, to a method, an apparatus, a device, and a medium for managing and scheduling heterogeneous resources of a database.
Background
Databases are used as an integrated system for data record storage, employing a variety of different models to organize data, which store data in different forms such as rows, columns, tables, etc. The database supports the omnibearing storage, searching and analysis of data and is widely applied to the fields of business, industry, intelligent home, medical care and the like. With the advent of the artificial intelligence era, large data and informatization processes have been continuously advanced, the installed capacity of databases has been exponentially increased, and the speed requirements for database data query have been increasing.
The database in the present stage mostly realizes the query processing of user access data through a CPU (Central Processing Unit/Processor, central processing unit) carried by a server. However, when the CPU processes the computationally intensive task, it will cause a great burden to the process and the memory, and affect the task scheduling and processing speed of other processes of the server, so it is an emerging technical direction to use FPGA (Field-Programmable Gate Array, field programmable gate array) and GPU (Graphics Processing Unit, graphics processor) as coprocessors to share the database computationally intensive task. The scheduling of heterogeneous resources and deployment of acceleration kernel (operating system kernel) by the cpu+x heterogeneous system are usually determined by monitoring the resource requirement and kernel idle state of the acceleration card, but the limitation of the communication bandwidth on the parallel acceleration kernel is often ignored, so that a plurality of parallel acceleration kernels can only be sequentially executed because of the limitation of the communication bandwidth, and the parallel acceleration can not be realized and the additional heterogeneous resources are wasted.
Disclosure of Invention
In view of the above, the present application aims to provide a method, an apparatus, a device, and a medium for managing and scheduling heterogeneous resources of a database, which can maximize the utilization of communication bandwidth, improve data exchange efficiency, and reduce heterogeneous resource waste. The specific scheme is as follows:
in a first aspect, the present application discloses a method for managing and scheduling heterogeneous resources of a database, which is applied to a database server, and includes:
receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator;
reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel;
sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation;
and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result.
Optionally, the performing a preset preprocessing operation on the query instruction to obtain a target operator includes:
decomposing the query instruction into a target number of operators;
processing each operator through a preset quality improvement algorithm to obtain a corresponding target operator; wherein the preset quality improvement algorithm comprises an uncorrelated algorithm.
Optionally, after the processing each operator by using a preset quality improvement algorithm to obtain the corresponding target operator, the method further includes:
analyzing the target operator through a preset instruction analysis operation so as to translate the target operator into a target code; the target code is bottom code which can be processed by the central processing unit.
Optionally, before the reading the idle communication bandwidth monitored by the data acceleration card, the method further includes:
and monitoring idle information of heterogeneous resources of the data acceleration card in real time through the bandwidth and the resource monitoring area on the data acceleration card so as to obtain the idle communication bandwidth.
Optionally, before the reading the idle communication bandwidth monitored by the data acceleration card, the method further includes:
and transmitting the data to be operated corresponding to the query instruction from a host memory to an on-chip memory through direct memory access so as to obtain the data storage address corresponding to the data to be operated.
Optionally, the performing an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel includes:
calling a corresponding API function based on the target operator, and acquiring the bandwidth utilization rate corresponding to the current data bus based on the idle communication bandwidth;
and executing acceleration kernel deployment operation based on the API function and the bandwidth utilization rate to obtain the target acceleration kernel.
Optionally, the executing the accelerating kernel deployment operation based on the API function and the bandwidth utilization to obtain the target accelerating kernel includes:
determining a current acceleration kernel set based on the API function;
matching the to-be-deployed acceleration kernel set meeting the current requirement from the current acceleration kernel set based on the bandwidth utilization rate;
and executing acceleration kernel deployment operation on the acceleration kernel set to be deployed so as to obtain the corresponding target acceleration kernel.
In a second aspect, the present application discloses a heterogeneous resource management and scheduling device for a database, which is applied to a database server, and includes:
the operator acquisition module is used for receiving a query instruction sent by a user and executing preset preprocessing operation on the query instruction to obtain a target operator;
the acceleration kernel deployment module is used for reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth so as to obtain a target acceleration kernel;
the data operation module is used for sending a data storage address to the target acceleration kernel so that the target acceleration kernel can read data to be operated on the basis of the data storage address and operate;
the operation result reading module is used for receiving a completion signal returned by the target acceleration kernel after the operation result is obtained and reading the operation result.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the database heterogeneous resource management and scheduling method as disclosed above.
In a fourth aspect, the present application discloses a computer readable medium for storing a computer program; wherein the computer program when executed by a processor implements a database heterogeneous resource management and scheduling method as previously disclosed.
The application provides a method for managing and scheduling heterogeneous resources of a database, which comprises the following steps: receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation; and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result. Therefore, when the acceleration kernel is called, the corresponding quantity of acceleration kernels are deployed in a self-adaptive mode through monitoring the idle communication bandwidth on the data acceleration card, so that the communication bandwidth is utilized to the maximum extent, the data exchange efficiency is improved, and the heterogeneous resource waste is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for managing and scheduling heterogeneous resources of a database;
FIG. 2 is a diagram of the overall architecture of the method for managing heterogeneous resources of a database according to the present application;
fig. 3 is a schematic diagram of implementation of a method for managing and scheduling heterogeneous resources of a database based on bandwidth monitoring according to the present application;
FIG. 4 is a flowchart of a method for managing and scheduling heterogeneous resources of a database according to the present application;
fig. 5 is a schematic diagram of communication bandwidth allocation according to the present disclosure;
FIG. 6 is a schematic diagram of a heterogeneous database resource management and scheduling device according to the present application;
fig. 7 is a block diagram of an electronic device according to the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
At present, the database mostly realizes the query processing of user access data through a CPU carried by a server. However, when the CPU processes the computationally intensive task, a huge burden is caused to the process and the memory, which affects the task scheduling and processing speed of other processes of the server, so that the use of FPGA and GPU as coprocessors to share the computationally intensive task of the database becomes an emerging technical direction. The scheduling of the heterogeneous resources and the deployment of the acceleration kernel by the CPU+X heterogeneous system are usually determined by monitoring the resource requirement of the acceleration card and the kernel idle state, but the limitation of the communication bandwidth on the parallel acceleration kernel is often ignored, so that a plurality of parallel acceleration kernels can only be sequentially executed because of the limitation of the communication bandwidth, and the parallel acceleration can not be realized and the additional heterogeneous resources are wasted. Therefore, the application provides a method for managing and scheduling heterogeneous resources of a database, which can maximally utilize communication bandwidth, improve data exchange efficiency and reduce heterogeneous resource waste.
The embodiment of the application discloses a method for managing and scheduling heterogeneous resources of a database, which is shown in fig. 1 and is applied to a database server, and the method comprises the following steps:
step S11: and receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator.
In this embodiment, a query instruction sent by a user is received, and a preset preprocessing operation is performed on the query instruction to obtain a target operator. Specifically, the query instruction is decomposed into a target number of operators; processing each operator through a preset quality improvement algorithm to obtain a corresponding target operator; wherein the preset quality improvement algorithm comprises an uncorrelated algorithm. It can be understood that, as shown in fig. 2, the heterogeneous resource management method of the database mainly comprises a user application layer, an instruction parsing layer, an acceleration interface layer, a task management layer, a device driving layer and a query acceleration layer. The user application layer provides database management application for the database server, and simultaneously, the user sets instructions for the database behaviors at the application layer so as to receive query instructions sent by the application layer; the instruction analysis layer comprises instruction analysis, instruction decomposition and instruction optimization; the acceleration interface layer includes a sequencing API (Application Program Interface ), a connection API, an aggregation API, and a filtering API; the task management layer comprises resource management, data transmission and task scheduling; the device driver layer includes Xilinx Runtime Library, XDMA and PCIE (PCI-Express, bus and interface standard); the query acceleration layer includes bandwidth and resource monitoring, data loading, sort kernel, join kernel, aggregation kernel, filter kernel.
After receiving a query instruction sent by a user, firstly decomposing the original query instruction into a plurality of operators through the instruction decomposition of an instruction analysis layer, and cooperatively completing the instruction function by the operators; and then optimizing the decomposed operators through an algorithm (namely a preset quality improvement algorithm such as incoherent parallelization and algorithm optimization) through instruction optimization to obtain target operators, so that the running efficiency and the acceleration efficiency of the instructions are improved maximally.
It will be appreciated that the device driver layer includes the device drivers required for the proper operation of the data accelerator card, as well as the communication protocols and drivers required for data transfer between the database CPU and the data accelerator card, such as XRT, PCIE, XDMA, etc.
Step S12: and reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel.
In this embodiment, a query instruction sent by a user is received, a preset preprocessing operation is performed on the query instruction to obtain a target operator, an idle communication bandwidth monitored by a data acceleration card is read, and an acceleration kernel deployment operation is performed based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel. Specifically, a corresponding API function is called based on the target operator, and the bandwidth utilization rate corresponding to the current data bus is obtained based on the idle communication bandwidth; and executing acceleration kernel deployment operation based on the API function and the bandwidth utilization rate to obtain the target acceleration kernel. Wherein the executing the acceleration kernel deployment operation based on the API function and the bandwidth utilization to obtain the target acceleration kernel includes: determining a current acceleration kernel set based on the API function; matching the to-be-deployed acceleration kernel set meeting the current requirement from the current acceleration kernel set based on the bandwidth utilization rate; and executing acceleration kernel deployment operation on the acceleration kernel set to be deployed so as to obtain the corresponding target acceleration kernel.
It can be understood that the acceleration interface layer contains an API function for the corresponding query operation, and the corresponding API of the acceleration interface layer is called according to the operator required by the instruction to deploy an acceleration kernel, and the user completes the dynamic reconfiguration of the logic function of the data acceleration card by calling the API function. The task scheduling in the task management layer carries out comprehensive scheduling management on the processes of operator operation, data encryption, compression, transmission and the like, and ensures orderly execution of instruction execution and data processing processes; the resource management reasonably distributes heterogeneous resources according to the bandwidth of the query acceleration layer and the resource monitoring result, reasonably deploys acceleration kernel according to the bandwidth utilization rate, and maximally utilizes communication bandwidth and heterogeneous resources; the data transfer ensures data transfer of data between the CPU and the accelerator card.
Step S13: and sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation.
In this embodiment, after performing an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel, a data storage address is sent to the target acceleration kernel, so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation. It can be understood that, as shown in fig. 3, the specific implementation process of the method for managing and scheduling heterogeneous resources of the database based on bandwidth monitoring is as follows: after receiving the inquiry operation instruction, the database server CPU analyzes and optimizes the instruction; the database CPU transfers the data to be operated from the host memory to the on-chip memory through DMA (Direct Memory Access ); the CPU of the database reads the idle communication bandwidth monitored by the data acceleration card at the moment, and completes the dynamic deployment of the coprocessor acceleration kernel according to the required communication bandwidth of the kernel corresponding to the query operation; after the deployment of the kernel is completed, the CPU sends the memory address of the data to be operated in the on-chip memory to the kernel, and the kernel is started; after the kernel is started, the data to be operated is read from the corresponding memory address, operation is carried out, then the operation result is rewritten into the memory, and a finishing signal is returned.
Step S14: and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result.
In this embodiment, a data storage address is sent to the target acceleration core, so that after the target acceleration core reads data to be operated on the basis of the data storage address and performs operation, a completion signal returned by the target acceleration core after an operation result is obtained is received, and the operation result is read. It can be understood that after the database CPU receives the completion signal, the operation result is read back to the host memory from the corresponding memory address, and the query operation is completed.
The database server architecture of the application is of a CPU+X (FPGA, GPU, DPU and the like) heterogeneous structure, the CPU is responsible for overall task scheduling of the database, low-density computing tasks, logic management processes and other controllable tasks, and the computation-density tasks are completed by X, so that the advantages of the CPU and the X are fully utilized, and the performance of the database is improved. According to the application, the bandwidth utilization rate of the coprocessor and the on-chip memory on the data acceleration card is monitored, and when the CPU invokes the acceleration kernel, the acceleration kernel with corresponding quantity is adaptively deployed according to the current bandwidth idle bandwidth and the bandwidth required by the kernel, so that the communication bandwidth is utilized to the maximum extent, the data exchange efficiency is improved, and the heterogeneous resource waste caused by the overflow of the communication bandwidth is saved.
The application provides a method for managing and scheduling heterogeneous resources of a database, which comprises the following steps: receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation; and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result. Therefore, when the acceleration kernel is called, the corresponding quantity of acceleration kernels are deployed in a self-adaptive mode through monitoring the idle communication bandwidth on the data acceleration card, so that the communication bandwidth is utilized to the maximum extent, the data exchange efficiency is improved, and the heterogeneous resource waste is reduced.
Referring to fig. 4, an embodiment of the present application discloses a method for managing and scheduling heterogeneous resources of a database, and compared with the previous embodiment, the present embodiment further describes and optimizes a technical solution.
Step S21: and receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator.
Step S22: analyzing the target operator through a preset instruction analysis operation so as to translate the target operator into a target code.
In this embodiment, after receiving a query instruction sent by a user and performing a preset preprocessing operation on the query instruction to obtain a target operator, analyzing the target operator through a preset instruction analysis operation so as to translate the target operator into a target code; the object code is bottom code which can be processed by the central processing unit. It can be understood that after the instruction decomposition and instruction optimization operation of the instruction analysis layer, the target operator is analyzed through the instruction analysis of the instruction analysis layer, so that the query instruction after the instruction decomposition and instruction optimization is translated into the bottom code which can be understood by the CPU.
Step S23: and monitoring idle information of heterogeneous resources of the data acceleration card in real time through the bandwidth and the resource monitoring area on the data acceleration card so as to obtain the idle communication bandwidth.
In this embodiment, after the target operator is translated into the target code, idle information of heterogeneous resources of the data acceleration card is monitored in real time through the bandwidth and the resource monitoring area on the data acceleration card, so as to obtain the idle communication bandwidth. It can be understood that the query acceleration layer is a data acceleration card, and includes bandwidth and resource monitoring, data loading and acceleration kernel. The data loading module receives data to be operated sent by the database CPU, and the bandwidth and resource monitoring module monitors the idle condition of heterogeneous resources of the data acceleration card.
Step S24: and transmitting the data to be operated corresponding to the query instruction from a host memory to an on-chip memory through direct memory access so as to obtain the data storage address corresponding to the data to be operated.
In this embodiment, after the idle communication bandwidth is obtained, the data to be operated corresponding to the query instruction is transferred from the host memory to the on-chip memory through direct memory access, so as to obtain the data storage address corresponding to the data to be operated. It can be understood that after the database CPU transmits the data to be operated to the on-chip memory, the current free bandwidth of the memory is analyzed, and the communication bandwidth is dynamically deployed according to the communication bandwidth required by the acceleration kernel, so as to maximize the utilization of the communication bandwidth.
Step S25: and reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel.
In this embodiment, the bandwidth utilization rate of the current data bus is fed back when the kernel is deployed, the acceleration kernel is corresponding to the function of the API interface layer according to the bandwidth utilization rate, and the corresponding acceleration kernel is deployed according to different API functions called by the user application layer. Specifically, the use condition of heterogeneous resources of a coprocessor on a data acceleration card and the bandwidth utilization rate of a data bus are monitored in real time, when a CPU executes a computationally intensive task, a kernel is dynamically deployed on the data acceleration card by calling a corresponding operation operator API, and a task management layer synthesizes the idle condition of a memory on the heterogeneous acceleration card and dynamically deploys the quantity of the kernel by the idle condition of the bandwidth and the current bandwidth required by deployment of the kernel, so that the communication bandwidth of the memory on a chip is fully utilized, and unnecessary heterogeneous resources are saved.
In a specific embodiment, the database CPU analyzes the current free bandwidth of the memory after transmitting the data to be operated to the on-chip memory, and dynamically deploys according to the communication bandwidth required by the acceleration kernel, and maximizes the utilization of the communication bandwidth, where the free bandwidth of the memory 1 is set to B1, the bandwidth required by the kernel is set to B1, so that only one kernel is deployed to communicate with the memory 1, the free bandwidth of the memory 3 is 2B3/3, and the bandwidth required by the kernel is set to B3/3, and thus two kernels can be deployed to communicate with the memory 3. If the communication bandwidth is exceeded, for example, 3 kernel are deployed to communicate with the memory 3, because the communication bandwidth reaches the maximum value at this time, 3 kernel can only run two of them at the same time, and the third kernel can continue to run after waiting for the completion of the first two kernel runs and releasing the bandwidth, so that the purpose of the ideal 3 kernel parallel running acceleration is not achieved, and additional heterogeneous resources are wasted to deploy unnecessary kernel.
For example, as shown in fig. 5, the data accelerator card has three on-chip memories, such as DDR4 (new generation memory specification) or HBM, with total communication bandwidths of B1, B2, and B3, respectively, and if the database CPU needs to perform the sorting operation, the communication bandwidth required by the sorting kernel is quantized to B1, B2/2, and B3/3 by the memory bandwidth, and the free communication bandwidths of the on-chip memories are B1, B2/2,2B3/3, respectively. The free bandwidth of the memory 1 is B1, and the bandwidth required by the kernel is B1, so that one kernel is deployed to communicate with the memory 1; the free bandwidth of the memory 2 is B2/2, and the bandwidth required by the kernel is B2/2, so that one kernel is deployed to communicate with the memory 2; memory 3 has a free bandwidth of 2B3/3 and a required bandwidth of B3/3, so two kernels can be deployed to communicate with memory 3. At this time, if the communication bandwidth is exceeded, for example, 3 kernel and the memory 3 are deployed for communication, since the communication bandwidth reaches the maximum value, kernel 1, kernel 2 and kernel 5 can only operate two of them at the same time, and third, the kernel 1 or kernel 2 needs to wait for the completion of operation and release the bandwidth before continuing to operate, so that the ideal purpose of 3 acceleration kernel parallel operations cannot be achieved, and additional heterogeneous resources are wasted.
Step S26: and sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation.
Step S27: and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result.
For the specific content of the steps S21, S26, S27, reference may be made to the corresponding content disclosed in the foregoing embodiment, and no further description is given here.
It can be seen that, in the embodiment of the present application, by receiving a query instruction sent by a user, and executing a preset preprocessing operation on the query instruction, a target operator is obtained; analyzing the target operator through a preset instruction analysis operation so as to translate the target operator into a target code; real-time monitoring idle information of heterogeneous resources of the data acceleration card through the bandwidth and resource monitoring area on the data acceleration card to obtain the idle communication bandwidth; transmitting the data to be operated corresponding to the query instruction from a host memory to an on-chip memory through direct memory access to obtain the data storage address corresponding to the data to be operated; reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation; and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result, so that the communication bandwidth is utilized to the maximum extent, the data exchange efficiency is improved, and the heterogeneous resource waste is reduced.
Referring to fig. 6, the embodiment of the application also correspondingly discloses a device for managing and scheduling heterogeneous resources of a database, which is applied to a database server and comprises:
the operator acquisition module 11 is configured to receive a query instruction sent by a user, and perform a preset preprocessing operation on the query instruction to obtain a target operator;
the acceleration kernel deployment module 12 is configured to read an idle communication bandwidth monitored by the data acceleration card, and execute an acceleration kernel deployment operation based on the target operator and the idle communication bandwidth, so as to obtain a target acceleration kernel;
a data operation module 13, configured to send a data storage address to the target acceleration kernel, so that the target acceleration kernel reads data to be operated on based on the data storage address and performs operation;
the operation result reading module 14 is configured to receive a completion signal returned by the target acceleration kernel after obtaining the operation result, and read the operation result.
It can be seen that the present application includes: receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator; reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel; sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation; and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result. Therefore, when the acceleration kernel is called, the corresponding quantity of acceleration kernels are deployed in a self-adaptive mode through monitoring the idle communication bandwidth on the data acceleration card, so that the communication bandwidth is utilized to the maximum extent, the data exchange efficiency is improved, and the heterogeneous resource waste is reduced.
In some specific embodiments, the operator obtaining module 11 specifically includes:
the instruction receiving unit is used for receiving a query instruction sent by a user;
the instruction decomposition unit is used for decomposing the query instruction into a target number of operators;
an operator processing unit, configured to process each operator through a preset quality improvement algorithm, so as to obtain a corresponding target operator; wherein the preset quality improvement algorithm comprises a non-correlation algorithm;
the instruction analysis unit is used for analyzing the target operator through a preset instruction analysis operation so as to translate the target operator into a target code; the target code is bottom code which can be processed by the central processing unit.
In some embodiments, the acceleration kernel deployment module 12 specifically includes:
the idle communication bandwidth acquisition unit is used for monitoring idle information of heterogeneous resources of the data acceleration card in real time through the bandwidth on the data acceleration card and the resource monitoring area so as to obtain the idle communication bandwidth;
the data storage address acquisition unit is used for transmitting the data to be operated corresponding to the query instruction from the host memory to the on-chip memory through direct memory access so as to obtain the data storage address corresponding to the data to be operated;
the idle communication bandwidth reading unit is used for reading the idle communication bandwidth monitored by the data acceleration card;
an API function calling unit for calling the corresponding API function based on the target operator;
the bandwidth utilization rate acquisition unit is used for acquiring the bandwidth utilization rate corresponding to the current data bus based on the idle communication bandwidth;
a current acceleration kernel set determining unit, configured to determine a current acceleration kernel set based on the API function;
the to-be-deployed acceleration kernel set determining unit is used for matching the to-be-deployed acceleration kernel set meeting the current requirement from the current acceleration kernel set based on the bandwidth utilization rate;
the target acceleration kernel acquisition unit is used for executing acceleration kernel deployment operation on the acceleration kernel set to be deployed so as to obtain the corresponding target acceleration kernel.
In some embodiments, the data operation module 13 specifically includes:
and the data operation unit is used for sending a data storage address to the target acceleration kernel so that the target acceleration kernel can read the data to be operated on based on the data storage address and operate.
In some embodiments, the operation result reading module 14 specifically includes:
the operation result reading unit is used for receiving a completion signal returned by the target acceleration kernel after the operation result is obtained and reading the operation result.
Further, the embodiment of the application also provides electronic equipment. Fig. 7 is a block diagram of an electronic device 20, according to an exemplary embodiment, and is not intended to limit the scope of use of the present application in any way.
Fig. 7 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, which is loaded and executed by the processor 21 to implement relevant steps in the database heterogeneous resource management and scheduling method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a computer program 222, and the like, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and computer programs 222, which may be Windows Server, netware, unix, linux, etc. The computer program 222 may further comprise a computer program capable of performing other specific tasks in addition to the computer program capable of performing the database heterogeneous resource management and scheduling method performed by the electronic device 20 disclosed in any of the foregoing embodiments.
Further, the embodiment of the application also discloses a medium, wherein the medium stores a computer program, and when the computer program is loaded and executed by a processor, the steps of the method for managing and scheduling the heterogeneous resources of the database disclosed in any embodiment are realized.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The method, the device, the equipment and the medium for managing and scheduling the heterogeneous resources of the database provided by the application are described in detail, and specific examples are applied to the explanation of the principle and the implementation mode of the application, and the explanation of the above examples is only used for helping to understand the method and the core idea of the application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (10)
1. The heterogeneous resource management and scheduling method for the database is characterized by being applied to a database server and comprising the following steps of:
receiving a query instruction sent by a user, and executing preset preprocessing operation on the query instruction to obtain a target operator;
reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target acceleration kernel;
sending a data storage address to the target acceleration kernel so that the target acceleration kernel reads data to be operated on the basis of the data storage address and performs operation;
and receiving a completion signal returned by the target acceleration kernel after the operation result is obtained, and reading the operation result.
2. The method for managing and scheduling heterogeneous resources of a database according to claim 1, wherein the performing a preset preprocessing operation on the query instruction to obtain a target operator includes:
decomposing the query instruction into a target number of operators;
processing each operator through a preset quality improvement algorithm to obtain a corresponding target operator; wherein the preset quality improvement algorithm comprises an uncorrelated algorithm.
3. The method for managing and scheduling heterogeneous resources of a database according to claim 2, wherein after said processing each of said operators by a preset quality-improving algorithm to obtain a corresponding target operator, further comprising:
analyzing the target operator through a preset instruction analysis operation so as to translate the target operator into a target code; the target code is bottom code which can be processed by the central processing unit.
4. The method for managing and scheduling heterogeneous resources in a database according to claim 1, wherein before the idle communication bandwidth monitored by the read data accelerator card, further comprising:
and monitoring idle information of heterogeneous resources of the data acceleration card in real time through the bandwidth and the resource monitoring area on the data acceleration card so as to obtain the idle communication bandwidth.
5. The method for managing and scheduling heterogeneous resources in a database according to claim 1, wherein before the idle communication bandwidth monitored by the read data accelerator card, further comprising:
and transmitting the data to be operated corresponding to the query instruction from a host memory to an on-chip memory through direct memory access so as to obtain the data storage address corresponding to the data to be operated.
6. The method for managing and scheduling heterogeneous resources of a database according to any one of claims 1 to 5, wherein the performing an accelerated kernel deployment operation based on the target operator and the idle communication bandwidth to obtain a target accelerated kernel includes:
calling a corresponding API function based on the target operator, and acquiring the bandwidth utilization rate corresponding to the current data bus based on the idle communication bandwidth;
and executing acceleration kernel deployment operation based on the API function and the bandwidth utilization rate to obtain the target acceleration kernel.
7. The method for managing and scheduling heterogeneous resources of a database according to claim 6, wherein said performing an acceleration kernel deployment operation based on said API function and said bandwidth utilization to obtain said target acceleration kernel comprises:
determining a current acceleration kernel set based on the API function;
matching the to-be-deployed acceleration kernel set meeting the current requirement from the current acceleration kernel set based on the bandwidth utilization rate;
and executing acceleration kernel deployment operation on the acceleration kernel set to be deployed so as to obtain the corresponding target acceleration kernel.
8. A heterogeneous database resource management and scheduling device, applied to a database server, comprising:
the operator acquisition module is used for receiving a query instruction sent by a user and executing preset preprocessing operation on the query instruction to obtain a target operator;
the acceleration kernel deployment module is used for reading the idle communication bandwidth monitored by the data acceleration card, and executing acceleration kernel deployment operation based on the target operator and the idle communication bandwidth so as to obtain a target acceleration kernel;
the data operation module is used for sending a data storage address to the target acceleration kernel so that the target acceleration kernel can read data to be operated on the basis of the data storage address and operate;
the operation result reading module is used for receiving a completion signal returned by the target acceleration kernel after the operation result is obtained and reading the operation result.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the database heterogeneous resource management and scheduling method according to any one of claims 1 to 7.
10. A computer readable medium for storing a computer program; wherein the computer program when executed by a processor implements the method of heterogeneous resource management and scheduling of a database according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310777226.1A CN116795647A (en) | 2023-06-28 | 2023-06-28 | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310777226.1A CN116795647A (en) | 2023-06-28 | 2023-06-28 | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116795647A true CN116795647A (en) | 2023-09-22 |
Family
ID=88047855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310777226.1A Pending CN116795647A (en) | 2023-06-28 | 2023-06-28 | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116795647A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117331945A (en) * | 2023-10-31 | 2024-01-02 | 中科驭数（北京）科技有限公司 | Data processing method, device, computer storage medium and acceleration card |
| CN117331970A (en) * | 2023-10-31 | 2024-01-02 | 中科驭数（北京）科技有限公司 | Data query method, device, computer storage medium and acceleration card |
| CN118377750A (en) * | 2024-06-25 | 2024-07-23 | 山东浪潮科学研究院有限公司 | FPGA-based heterogeneous acceleration kernel configuration system for database |
-
2023
- 2023-06-28 CN CN202310777226.1A patent/CN116795647A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117331945A (en) * | 2023-10-31 | 2024-01-02 | 中科驭数（北京）科技有限公司 | Data processing method, device, computer storage medium and acceleration card |
| CN117331970A (en) * | 2023-10-31 | 2024-01-02 | 中科驭数（北京）科技有限公司 | Data query method, device, computer storage medium and acceleration card |
| CN118377750A (en) * | 2024-06-25 | 2024-07-23 | 山东浪潮科学研究院有限公司 | FPGA-based heterogeneous acceleration kernel configuration system for database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831562B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| CN116795647A (en) | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database | |
| CN112035238B (en) | Task scheduling processing method and device, cluster system and readable storage medium | |
| CN111258744A (en) | Task processing method based on heterogeneous computation and software and hardware framework system | |
| CN112738060B (en) | Method and device for processing micro-service data, micro-service processing platform and medium | |
| CN111679911B (en) | Management method, device, equipment and medium of GPU card in cloud environment | |
| CN115756822A (en) | Method and system for optimizing performance of high-performance computing application | |
| CN108563509A (en) | Data query implementation method, device, medium and electronic equipment | |
| CN115562846A (en) | Resource scheduling method and device and computing node | |
| CN114912618A (en) | Quantum computing task scheduling method and device and quantum computer operating system | |
| CN110442446B (en) | Method for real-time processing high-speed digital signal data stream | |
| CN111400034B (en) | Multi-core processor-oriented waveform resource allocation method | |
| CN116361120B (en) | Method, device, equipment and medium for managing and scheduling heterogeneous resources of database | |
| CN113094395A (en) | Data query method, computer device and storage medium | |
| CN114466088B (en) | Data transmission method and device of sweeping robot, storage medium and terminal | |
| CN117271122A (en) | Task processing method, device, equipment and storage medium based on separation of CPU and GPU | |
| CN111190731A (en) | Cluster task scheduling system based on weight | |
| CN111190607B (en) | Task plugin processing method and device, task scheduling server and storage medium | |
| CN116383240A (en) | Multi-database acceleration query method, device, equipment and medium based on FPGA | |
| CN115712524A (en) | Data recovery method and device | |
| CN115829826A (en) | Model storage optimization method and electronic equipment | |
| CN115167985A (en) | Virtualized computing power providing method and system | |
| CN115292053B (en) | CPU, GPU and NPU unified scheduling method of mobile terminal CNN | |
| CN116032928B (en) | Data collaborative computing method, device, system, electronic device and storage medium | |
| CN117435367B (en) | User behavior processing method, device, equipment, storage medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |