CN112685275A - Algorithm strategy searching method and device, electronic equipment and storage medium - Google Patents
Algorithm strategy searching method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112685275A CN112685275A CN202011611065.1A CN202011611065A CN112685275A CN 112685275 A CN112685275 A CN 112685275A CN 202011611065 A CN202011611065 A CN 202011611065A CN 112685275 A CN112685275 A CN 112685275A
- Authority
- CN
- China
- Prior art keywords
- target
- parameter information
- tested
- software
- sdk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 191
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000012360 testing method Methods 0.000 claims abstract description 274
- 238000011056 performance test Methods 0.000 claims abstract description 19
- 230000015654 memory Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 description 30
- 238000013527 convolutional neural network Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 101800000618 Protein kinase C delta type catalytic subunit Proteins 0.000 description 3
- 102100021004 Protein sidekick-1 Human genes 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 125000004122 cyclic group Chemical group 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The application provides an algorithm strategy searching method, an algorithm strategy searching device, electronic equipment and a storage medium, and relates to the technical field of software. The method obtains the search result by searching whether the historical test records contain the target test records of the SDK of the target network model on the tested platform with the target software and hardware parameter information, so that the historical test records can be searched before algorithm strategy search is carried out, when the historical test records contain the corresponding target test records, the target optimal algorithm strategy in the target test records can be directly reused as the optimal algorithm strategy of the target SDK running on the target platform, algorithm strategy search is not needed, the time of algorithm strategy search can be saved, the time of the whole performance test is reduced, and the test efficiency is improved.
Description
Technical Field
The application relates to the technical field of software, in particular to an algorithm strategy searching method, an algorithm strategy searching device, electronic equipment and a storage medium.
Background
At present, the performance of a network model running on a platform by using different algorithm strategies is usually different, so in order to optimize the performance of the network model running on a certain platform, search of the algorithm strategies needs to be performed, that is, the optimal algorithm strategies are searched out for the network model to use.
In the prior art, when the network model is subjected to performance tests on different platforms, the search process of the algorithm strategy is required, the search process is long in time consumption, a large amount of test time is required, and the test efficiency is low. Moreover, when a large number of network models are subjected to performance tests on a large number of platforms, if algorithm strategy search is required each time, a long time is consumed.
Disclosure of Invention
An object of the embodiments of the present application is to provide an algorithm policy search method, an algorithm policy search device, an electronic device, and a storage medium, so as to solve the problem in the prior art that a long test time is consumed due to the fact that algorithm policy search needs to be performed each time in a performance test process of a network model on a platform, which results in low test efficiency.
In a first aspect, an embodiment of the present application provides an algorithm policy search method, where the method includes: determining a target Software Development Kit (SDK) which needs to perform performance test on a target platform with target software and hardware parameter information at present, wherein the target SDK comprises a target network model; searching whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists in the historical test record or not to obtain a search result; if the search result is that the target test record exists in the historical test record, obtaining a target optimal algorithm strategy obtained by testing the tested SDK in the target test record on the tested platform; and determining the target optimal algorithm strategy as an optimal algorithm strategy of the network model in the target SDK running on the target platform.
In the implementation process, the search result is obtained by searching whether the historical test records contain the target test record of the SDK of the target network model on the tested platform with the target software and hardware parameter information, so that the historical test records can be searched before algorithm strategy search is carried out, when the historical test records contain the corresponding target test record, the target optimal algorithm strategy in the target test record can be directly reused as the optimal algorithm strategy of the target SDK running on the target platform, algorithm strategy search is not needed, the algorithm strategy search time can be saved, the whole performance test time is reduced, and the test efficiency is improved.
Optionally, the method further comprises:
and if the search result is that the target test record does not exist in the historical test records, carrying out algorithm strategy search.
In the implementation process, when no corresponding target test record exists in the historical test records, the algorithm strategy search is performed, so that whether a reusable algorithm strategy exists in the historical test records or not is searched before the algorithm strategy search is performed, and when the algorithm strategy search is not required, the time for searching the algorithm strategy can be saved, so that the test time of the target network model is reduced.
Optionally, the historical test record includes the type of the network model included in each tested SDK and software and hardware parameter information corresponding to each tested platform; the method for determining whether the historical test records have the target test records of the tested SDK containing the target network model on the tested platform with the target software and hardware parameter information comprises the following steps:
matching the type of the network model contained in each tested SDK in the historical test record with the type of the target network model, and matching software and hardware parameter information corresponding to each tested platform with the target software and hardware parameter information;
if the type of the target network model is consistent with the type of the network model contained in one tested SDK in the historical test records, and the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records, determining that the historical test records contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information; otherwise, determining that the historical test records do not contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information.
In the implementation process, the type of the target network model is compared with the types of the network models recorded in the historical test records, and the target software and hardware parameter information is compared with the software and hardware parameter information of the tested platforms recorded in the historical test records, so that whether the historical test records contain the corresponding target test records can be determined more accurately.
Optionally, the determining whether the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records includes:
judging whether the deviation between the target software and hardware parameter information and the software and hardware parameter information of one tested platform in the historical test records is within a preset range or not;
and if so, determining that the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records.
In the implementation process, whether the deviation between the target software and hardware parameter information and the software and hardware parameter information of one tested platform is within a preset range or not is judged, so that the algorithm strategy search is not needed for the target platform with the deviation within the preset range, the algorithm strategy corresponding to one tested platform can be directly reused, and the test time can be greatly saved when a large number of SDKs are tested on different platforms.
Optionally, after the performing of the algorithm policy search, the method further includes:
obtaining a search result after algorithm strategy search, wherein the search result comprises an optimal algorithm strategy used by a network model in the target SDK on the target platform;
and storing the search result, the target software and hardware parameter information of the target platform and the target network model in the target SDK into the historical test record.
In the implementation process, the obtained search results and other information are stored in the historical test records, so that more data references can be provided for the subsequent performance test of the network model.
Optionally, the storing the search result in the historical test record includes:
reading the search result from a first folder created in advance;
and migrating the search result from the first folder to a second folder for storing the historical test records.
In the implementation process, the current search result is stored in another folder, so that the search result of each time can be known conveniently, and the test records for each network model are presented in the historical test records.
In a second aspect, an embodiment of the present application provides an algorithm policy search apparatus, where the apparatus includes:
the system comprises an information determining module, a target Software Development Kit (SDK) and a target network model, wherein the information determining module is used for determining the SDK of the target software development kit which needs to perform performance test on a target platform with target software and hardware parameter information at present;
the information searching module is used for searching whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists in the historical test record or not to obtain a searching result;
an algorithm strategy obtaining module, configured to obtain an optimal target algorithm strategy obtained by testing the tested SDK in the target test record on the tested platform if the search result is that the target test record is included in the historical test record;
and the algorithm strategy determining module is used for determining the target optimal algorithm strategy as the optimal algorithm strategy of the network model in the target SDK, which runs on the target platform.
Optionally, the algorithm policy obtaining module is further configured to perform algorithm policy search if the search result indicates that the target test record does not exist in the historical test records.
Optionally, the historical test record includes the type of the network model included in each tested SDK and software and hardware parameter information corresponding to each tested platform; the information searching module is used for matching the type of the network model contained in each tested SDK in the historical test record with the type of the target network model, and matching the software and hardware parameter information corresponding to each tested platform with the target software and hardware parameter information; if the type of the target network model is consistent with the type of the network model contained in one tested SDK in the historical test records, and the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records, determining that the historical test records contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information; otherwise, determining that the historical test records do not contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information.
Optionally, the information search module is configured to determine whether a deviation between the target software and hardware parameter information and software and hardware parameter information of one tested platform in the historical test records is within a preset range; and if so, determining that the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records.
Optionally, the apparatus further comprises:
the recording module is used for obtaining a search result after algorithm strategy search is carried out, and the search result comprises an optimal algorithm strategy used by a network model in the target SDK on the target platform; and storing the search result, the target software and hardware parameter information of the target platform and the target network model in the target SDK into the historical test record.
Optionally, the recording module is configured to read the search result from a first folder created in advance; and migrating the search result from the first folder to a second folder for storing the historical test records.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method as provided in the first aspect are executed.
In a fourth aspect, embodiments of the present application provide a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps in the method as provided in the first aspect.
Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a flowchart of an algorithm policy search method according to an embodiment of the present disclosure;
fig. 2 is a block diagram of an algorithm policy search apparatus according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device for executing an algorithm policy search method according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.
Before describing particular embodiments of the present application, the following discussion is intended to provide an understanding of the present application with a general background.
Software Development Kit (SDK) is generally a collection of Development tools used by some Software engineers to build application Software for a particular Software package, Software framework, hardware platform, operating system, etc. In the embodiment of the application, in order to facilitate performance testing of each network model on each platform by using different algorithm strategies, the network model may be packaged in the SDK, so that the performance of the network model is tested by running the SDK on the platform.
The platform includes a plurality of algorithm strategies used by the network model, and an algorithm strategy may be understood as a combined calculation strategy for describing operators corresponding to each node in the network model, for example, a certain node in the network model may be represented as y (x) ═ a × x2+ b x + c, where x is the input, a, b, c are the weighting parameters of the node, and the algorithm strategy is x2、b*x、a*x2How these operators operate on the platform, e.g. b x, a x can be calculated on the platform's cpu2X can be computed on gpu of the platform2Or selecting other parameters participating in the calculation, and the like, wherein the parameters refer to software and hardware parameters of a platform which can influence the calculation speed, and how to select the combination of the software and hardware parameters to operate the operators can be understood as the search of the algorithm strategy.
One combination of each software and hardware parameter can be called as an algorithm strategy, when the SDK runs on a platform, the SDK can run according to the multiple algorithm strategies, and then the algorithm strategy with the highest running speed is determined as the optimal algorithm strategy.
For example, if 8 algorithm strategies are set for the convolutional neural network model in the platform, the performance of the convolutional neural network model may be different when different algorithm strategies are selected for the convolutional neural network model to run on the platform, and therefore, an optimal algorithm strategy needs to be selected from the algorithm strategies as the algorithm strategy used when the convolutional neural network model finally runs on the platform. In the embodiment of the application, the optimal algorithm strategy can be searched and obtained through the algorithm strategy searching mechanism of the embodiment of the application, so that the optimal performance is realized.
It should be noted that the algorithm policies set for the same network model on different platforms may be different, so that the optimal algorithm policies determined when the performance of the same network model is optimal on different platforms may also be different.
In the embodiment of the application, in order to test the operation performance of the network model on different platforms by using different algorithm strategies, the performance of the network model when the network model operates on the platforms by using different algorithm strategies can be obtained, for example, indexes such as the calculation speed and the calculation time of the network model are tested, so that the optimal algorithm strategy used by the network model can be obtained.
Continuous Integration (CI), i.e. continuously performing Integration, construction, and automation test of code during the code construction process; with the CI tool, the introduced errors can be discovered as early as possible through unit testing and the like in the process of code submission; and (2) Continuous Delivery (CD), after the code is constructed, the new version can be conveniently deployed on line, so that quick iteration is facilitated and the product is delivered.
A GitLab CI/CD (GitLab CI for short) is a set of CI/CD system based on the GitLab, and can allow developers to pass through.
At present, the above GitLab CI/CD is provided for some SDKs, such as picture quality SDKs, in order to ensure the code quality, find out the code error in advance and solve the problem in advance. Therefore, if the performance of the SDK tested in the CI/CD is expected to be optimal, the existing algorithm strategy search mechanism needs to be used again on the platform for each submitted code to determine the optimal algorithm strategy used by the network model in the SDK.
The existing algorithm strategy searching mechanism is divided into two steps, firstly, algorithm strategy searching is carried out, a plurality of algorithm strategies are obtained, then, the optimal algorithm strategy is found out and stored in an algo _ policy folder, then, a cache is generated by utilizing the file, and when the algorithm strategy is used next time, the optimal algorithm strategy can be directly and quickly read from the cache. However, in the process of obtaining the optimal algorithm strategy by the currently adopted algorithm strategy search mechanism, an average time is about 15 minutes, and assuming that 20 times of submitting new SDKs (where there is an operation of updating the model) are performed in a day, in order to optimize the performance of the 20 CI/CD tests, 20 times of search processes of the algorithm strategy need to be performed, that is, 20 × 15 is 300 minutes, but assuming that there are 3 platforms, 4 SDK products need to be tested, 3 × 4 × 300 is 3600 minutes, and obviously, a long time is required.
The above prior art solutions have shortcomings which are the results of practical and careful study of the inventor, and therefore, the discovery process of the above problems and the solutions proposed by the following embodiments of the present invention to the above problems should be the contribution of the inventor to the present invention in the course of the present invention.
In order to solve the problem that the time consumption of a test process in the prior art is long, an embodiment of the present application provides an algorithm strategy search method, and a search result is obtained by searching whether a test record of a tested SDK including a target network model on a tested platform having target software and hardware parameter information exists in a history test record, so that the history test record can be searched before algorithm strategy search is performed, and therefore, when the history test record includes a corresponding target test record, a target optimal algorithm strategy in the target test record can be directly reused as an optimal algorithm strategy for the target SDK to operate on the target platform, search of the algorithm strategy is not required, time for algorithm strategy search can be saved, time for whole performance test can be reduced, and test efficiency can be improved.
Referring to fig. 1, fig. 1 is a flowchart of an algorithm policy search method according to an embodiment of the present application, where the method includes the following steps:
step S110: determining a target SDK which needs to perform performance test on a target platform with target software and hardware parameter information at present, wherein the target SDK comprises a target network model.
The target network model may refer to any type of network model, such as a convolutional neural network model, a cyclic neural network model, and the like, and the performance of the target network model running on the target platform using different algorithm strategies is usually different. Therefore, the optimal performance of the target network model running on the target platform needs to be tested to find the optimal algorithm strategy.
For example, for a convolutional neural network model and a cyclic neural network model, the models each include different algorithm operations such as a convolution operation and a pooling operation, and these operations are performed on different platforms, and may be performed using different algorithm policies, for example, different operations are performed using different processors, such as directly using a CPU, or using cuda of a GPU, or using Opencl, in these implementations, the setting and selection of the size of a partition, Opencl work groups, and the like, all affect the performance of the model on the platform, and these selections may be referred to as part of algorithm policy search.
For example, on a certain platform, there are 5 corresponding algorithm strategies, and when performing algorithm strategy search, all 5 algorithm strategies are traversed once, and then the algorithm strategy with the fastest execution speed is searched as the optimal algorithm strategy.
That is, different algorithm strategies are used by the network model to run on the platform, and the performance of the network model generally differs, so that in order to optimize the performance of the network model running on the platform, the optimal algorithm strategy needs to be found through an algorithm strategy searching process.
In practical application, the platform may refer to chips of different models, different types of mobile phones, or hardware platforms such as different servers, and may, of course, refer to a software platform, such as a different operating system.
The software and hardware parameter information may refer to related parameter information of the platform, for example, for the case that the platform is a mobile phone, the software and hardware parameter information may include related information of software and hardware such as a CPU, a GPU, an operating system, and the like, such as information of a model, a memory, an operation speed, and the like of the CPU and the GPU, a model of the operating system, and the like, that is, the software and hardware parameter information may refer to software and hardware information of the platform itself, and the like. Of course, when the platform is a software platform, the corresponding software and hardware parameter information may include parameter information corresponding to the software platform and/or hardware information of a hardware device operated by the software platform.
In practical situations, the software and hardware parameter information corresponding to each platform can be set according to requirements. For different types of platforms, the types of corresponding software and hardware parameter information may also be different.
In addition, the mode of obtaining the target software and hardware parameter information may be obtained through user input, or may be obtained by searching the prestored software and hardware parameter information corresponding to each platform. The target SDK may be determined by marking the SDK currently to be tested, and the marking may be performed by the user according to the requirement.
It should be noted that, if a plurality of SDKs need to be tested on a plurality of platforms, when a plurality of SDKs are tested on the same platform, a corresponding test sequence can be set according to requirements, so that each SDK is tested on each platform in sequence, and the problem of test confusion caused by simultaneous testing can be avoided. The testing process of each SDK on the corresponding platform is similar to the testing process of the target SDK on the target platform, and the target SDK is tested on the target platform in this embodiment of the present application.
Step S120: and searching whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists in the historical test record or not to obtain a search result.
The historical test records include test records of the tested SDKs containing the network models for performing performance tests on the tested platforms with different software and hardware parameter information, that is, the historical test records include a plurality of test records, and each test record may include test information of the tested SDKs containing the network models for performing performance tests on the tested platforms with corresponding software and hardware parameter information.
Therefore, whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists can be searched from a plurality of test records in the historical test records, and a corresponding search result is obtained.
The obtained search result may be that the historical test record does not include the target test record, or the search result is that the historical test record includes the target test record.
If the target SDK is tested on other platforms and needs to be tested on the target platform currently, the historical test record comprises the relevant information of the target SDK; or, if another SDK has been tested on the target platform, the historical test record may include related information of the target platform, such as software and hardware parameter information of the target platform; if the target SDK is a new untested SDK and the target platform is a newly added platform, whether a tested SDK matched with the target SDK exists or not and a tested platform matched with the target platform can be searched from the historical test records.
Step S130: and if the search result is that the target test record exists in the historical test record, obtaining a target optimal algorithm strategy obtained by testing the tested SDK in the target test record on the tested platform.
If the search result is that the historical test record contains the target test record of the tested SDK containing the target network model on the tested platform with the target software and hardware parameter information, the algorithm strategy search is not needed, the target SDK is indicated to be tested on the target platform without performing the algorithm strategy search, the target test record in the historical test record can be directly reused, and if the target optimal algorithm strategy of the tested SDK containing the target network model and recorded in the target test record running on the tested platform with the target software and hardware parameter information can be obtained, the target optimal algorithm strategy is used as the optimal algorithm strategy of the target SDK running on the target platform, so that the test time can be saved.
Step S140: and determining the target optimal algorithm strategy as an optimal algorithm strategy of the network model in the target SDK running on the target platform.
For example, if the target network model is a convolutional neural network model, a target optimal algorithm strategy when the SDK1 containing the convolutional neural network model runs on the platform 1 with the same target software and hardware parameters is obtained from the test record 1 in the historical test records, such as the algorithm strategy 1, at this time, the test record 1 is the above-mentioned target test record, the SDK1 is a tested SDK, and the platform 1 is a tested platform, and then the algorithm strategy 1 is determined as the optimal algorithm strategy when the target SDK runs on the target platform. In this case, that is, when the running performance of the test target SDK on the target platform is expressed, the algorithm policy 1 recorded in the test record 1 may be directly reused without performing the search of the algorithm policy, so that the search time of the algorithm policy of 15 minutes may be saved, and the test time may be effectively reduced.
In the implementation process, the search result is obtained by searching whether the historical test records contain the target test record of the SDK of the target network model on the tested platform with the target software and hardware parameter information, so that the historical test records can be searched before algorithm strategy search is carried out, when the historical test records contain the corresponding target test record, the target optimal algorithm strategy in the target test record can be directly reused as the optimal algorithm strategy of the target SDK running on the target platform, algorithm strategy search is not needed, the algorithm strategy search time can be saved, the whole performance test time is reduced, and the test efficiency is improved.
In some embodiments, if the search result is that the historical test record does not include the target test record of the tested SDK including the target network model on the tested platform with the target software and hardware parameter information, algorithm strategy search is performed, that is, the target SDK is operated, and then the optimal algorithm strategy corresponding to the optimal operation performance of the target SDK is found.
In the implementation process, when no corresponding target test record exists in the historical test records, the algorithm strategy search is performed, so that whether a reusable algorithm strategy exists in the historical test records or not is searched before the algorithm strategy search is performed, and when the algorithm strategy search is not required, the time for searching the algorithm strategy can be saved, so that the test time of the target network model is reduced.
In some embodiments, the method for determining whether there is a target test record of the tested SDK including the target network model on the tested platform having the target software and hardware parameter information includes:
matching the type of the network model contained in each tested SDK in the historical test record with the type of a target network model, and matching software and hardware parameter information corresponding to each tested platform with the target software and hardware parameter information;
if the type of the target network model is consistent with the type of the network model contained in one tested SDK in the historical test records, and the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records, determining that the historical test records contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information;
and if the type of the target network model is inconsistent with the type of the network model contained in any tested SDK in the historical test record, and/or the target software and hardware parameter information is inconsistent with the software and hardware parameter information of any tested platform in the historical test record, determining that the historical test record does not contain the target test record of the tested SDK containing the target network model on the tested platform with the target software and hardware parameter information.
Wherein, each test record recorded in the historical test records can be as shown in the following table 1:
TABLE 1
It can be understood that each test record in the history test records includes the type of the network model included in the SDK and the software and hardware parameter information of the platform, so that the type of the target network model included in the target SDK can be compared with the type of the network model recorded in each test record to determine whether the types are the same, and the target software and hardware parameter information can be compared with the software and hardware parameter information recorded in each test record to determine whether the software and hardware parameter information is the same.
In order to facilitate the comparison of the types of the network models, the same type of network models can be marked with the same identifier when the test records are stored, so that the identifiers of the network models can be compared when the types of the network models are compared. For example, the identifiers corresponding to different types of network models are stored in advance, such as the identifier corresponding to the convolutional neural network model is 1, the identifier corresponding to the cyclic neural network model is 2, and the identifier corresponding to the long-term and short-term memory network model is 3, … …, so that when the historical test record is stored, the corresponding identifier can be searched according to the type of the network model included in each SDK during each test, and then the identifier is recorded in the historical test record. After the target SDK is obtained, the type of the target network model included in the target SDK may be obtained first, then an identifier corresponding to the target network model of the type is searched, then the identifier is compared with the identifier of the network model recorded in each test record in the historical test records, if the two identifiers are the same, the type of the target network model is considered to be consistent with the type of the network model recorded in the test record, and if the two identifiers are different, the type of the target network model is considered to be inconsistent with the type of the network model recorded in the test record.
When comparing the software and hardware parameter information, each parameter information can be compared one by one, wherein each parameter information of the same type is compared, if the target software and hardware parameter information comprises the model, the running speed and the memory of the CPU and the GPU, the model of the CPU can be extracted from the software and hardware parameter information recorded in each test record and compared with the model of the CPU in the target software and hardware parameter information, the running speed of the CPU in the test record is compared with the running speed of the CPU in the target software and hardware parameter information, and the memory of the CPU in the test record is compared with the memory of the CPU in the target software and hardware parameter information; the same is true for the comparison of the GPU, and according to this way, each parameter information in the target software and hardware parameter information can be compared one by one, and when all the parameter information are compared and consistent, it is determined that the target software and hardware parameter information is consistent with the software and hardware parameter information recorded in the test record, and if any parameter information is not compared and consistent, it is determined that the target software and hardware parameter information is inconsistent with the software and hardware parameter information recorded in the test record.
In some other embodiments, when comparing the software and hardware parameter information, if the target software and hardware parameter information includes a plurality of parameter information, if the parameter information in the plurality of parameter information includes more than a preset number of parameter information, and the software and hardware parameter information recorded in a certain test record are compared and consistent, it is determined that the target software and hardware parameter information is compared and consistent with the software and hardware parameter information recorded in the certain test record.
The preset number can be flexibly set according to the number of the actual parameter information, for example, when the parameter information includes 3, the preset number can be set to 2 or 3, and in an actual situation, the value of the preset number can be flexibly set.
For example, when the type of the target network model is compared with the type of the network model in the test record 1 to determine that the comparison is consistent, the target software and hardware parameter information is compared with the software and hardware parameter information in the test record 1 to determine that the comparison is consistent, it is determined that the historical test records include the target test record 1 of the tested SDK including the target network model on the tested platform having the target software and hardware parameter information, and in this case, algorithm strategy search is not required.
It should be noted that, when comparing the network models, not only the types of the network models but also other relevant information such as the structural information of the network models may be compared; when the software and hardware parameter information is carried out, only some parameter information can be selected for comparison, and all parameter information does not need to be compared. That is, in the specific implementation process, the information that needs to be compared may be set according to the actual requirement, for example, the size of the input/output data of the network model may also be compared.
In the implementation process, the type of the target network model is compared with the types of the network models recorded in the historical test records, and the target software and hardware parameter information is compared with the software and hardware parameter information of the tested platforms recorded in the historical test records, so that whether the historical test records contain the corresponding target test records can be determined more accurately.
In some embodiments, when comparing the software and hardware parameter information, it may be further determined whether a deviation between the target software and hardware parameter information and the software and hardware parameter information of one of the tested platforms in the historical test records is within a preset range, and if so, it is determined that the target software and hardware parameter information is consistent with the software and hardware parameter information of one of the tested platforms in the historical test records.
The corresponding preset range can be preset for each parameter information, the preset range can be flexibly set according to actual conditions, and the preset ranges set for different parameter information can be different. For example, if the deviation between each piece of target software and hardware parameter information and the corresponding piece of parameter information in the software and hardware parameter information in the test record 1 in table 1 is within the corresponding preset range, the software and hardware parameter information of the two pieces of parameter information are compared to be consistent.
In the implementation process, whether the deviation between the target software and hardware parameter information and the software and hardware parameter information of one tested platform is within a preset range or not is judged, so that the algorithm strategy search is not needed for the target platform with the deviation within the preset range, the algorithm strategy corresponding to one tested platform can be directly reused, and the test time can be greatly saved when a large number of SDKs are tested on different platforms.
In some embodiments, the historical test records may further include an optimal algorithm policy for the network model in each SDK to run on each platform, as shown in table 2 below:
TABLE 2
Therefore, after the algorithm strategy search is not needed in the determination, the target optimal algorithm strategy can be obtained from the target test record, and then the target optimal algorithm strategy is determined as the optimal algorithm strategy of the network model in the target SDK running on the target platform.
For example, if the type of the target network model is a convolutional neural network model, and the target software and hardware parameter information of the target network model is matched with the software and hardware parameter information in the platform 1, the optimal algorithm strategy of the SDK1 when running on the platform 1, that is, the algorithm strategy 1, may be obtained from the test record 1 (that is, the target test record), and then the algorithm strategy 1 is determined as the optimal algorithm strategy of the target SDK when running on the target platform. In this case, that is, when the running performance of the test target SDK on the target platform is expressed, the algorithm policy 1 recorded in the test record 1 may be directly reused without performing the search of the algorithm policy, so that the search time of the algorithm policy of 15 minutes may be saved, and the test time may be effectively reduced.
In the implementation process, the target optimal algorithm strategy is multiplexed, the searching process of the algorithm strategy is not needed, and the testing time is reduced.
In some embodiments, when it is determined that algorithm policy search is required, algorithm policy search operation may be performed, and an optimal algorithm policy when the target SDK runs on the target platform is obtained through search, that is, a search result after algorithm policy search is performed may be obtained, where the search result includes an optimal algorithm policy used by a network model in the target SDK on the target platform, and then the search result, target software and hardware parameter information of the target platform, and a target network model in the target SDK may be stored in a history test record.
The storage of the target network model in the historical test record may refer to storing information such as an identifier of the target network model or a type of the target network model, and may also store parameter information such as a network structure of a real target network model.
That is, the search result, the target software and hardware parameter information, and the target network model may be stored in the historical test records according to the format shown in table 2, and accumulated as the historical test records, so that more data may be provided for the reuse of the algorithm strategy in the subsequent test process.
In some embodiments, the search results may be stored in a first folder created in advance, and the historical test records stored in a second folder, such as a folder maintaining a globally unique name in the system, called big _ algo _ policy, for storing the historical test records. And then, an empty folder named empty _ algo _ policy can be preset and created to store new search results obtained after algorithm strategy search is performed each time, namely, the search results can be read from a first folder created in advance, and then the search results are migrated from the first folder to a second folder for storing historical test records. Of course, the search results may also be copied into the second folder.
For example, when determining whether algorithm policy search is needed, the historical test records may be read from the big _ algo _ policy folder, and then corresponding test records may be searched from the historical test records, and if it is determined that algorithm policy search is needed, algorithm policy search operation is performed. And then storing the search result obtained by searching into an empty _ algo _ policy folder, then checking the file size of the empty _ algo _ policy through an ls-l command, if the file size is not 0, determining that the search result is in the folder, splicing the content in the folder to the end of the big _ algo _ policy folder, namely, moving the search result into the end of the big _ algo _ policy folder, for example, adding a new line to the last of the table 2, and then adding the search result, the target software and hardware parameter information and the type of the target network model into the new line to the last of the table 2.
Of course, in some other embodiments, it is also possible to create a new row directly at the last in the table storing the historical test records, and then directly store the obtained search results, the target software and hardware parameter information, and the type of the target network model in the new row at the last.
In the implementation process, the current search result is stored in another folder, so that the search result of each time can be known conveniently, and the test records for the various network models are presented in the historical test records.
In some embodiments, the historical test records may also be stored in a git repository, with additional git repositories storing the historical test records, so that each time a performance test is performed, the historical test records may be read from the git repository and then subjected to a subsequent information matching process. Thus, when the problems are found manually, the problems can be located by searching the historical test records stored in the git warehouse.
Referring to fig. 2, fig. 2 is a block diagram of an algorithm policy search apparatus 100 according to an embodiment of the present disclosure, where the apparatus 100 may be a module, a program segment, or code on an electronic device. It should be understood that the apparatus 100 corresponds to the above-mentioned embodiment of the method of fig. 1, and can perform various steps related to the embodiment of the method of fig. 1, and the specific functions of the apparatus 100 can be referred to the above description, and the detailed description is appropriately omitted here to avoid redundancy.
Optionally, the apparatus 100 comprises:
an information determining module 110, configured to determine a target software development kit SDK that currently needs to perform a performance test on a target platform with target software and hardware parameter information, where the target SDK includes a target network model;
the information searching module 120 is configured to search whether a historical test record includes a target test record of the tested SDK of the target network model on the tested platform having the target software and hardware parameter information, and obtain a search result;
an algorithm policy obtaining module 130, configured to obtain a target optimal algorithm policy obtained by testing the tested SDK in the target test record on the tested platform if the search result is that the target test record is included in the historical test record;
and an algorithm policy determining module 140, configured to determine the target optimal algorithm policy as an optimal algorithm policy for the network model in the target SDK to run on the target platform.
Optionally, the algorithm policy obtaining module 130 is further configured to perform algorithm policy search if the search result indicates that the target test record does not exist in the historical test records.
Optionally, the historical test record includes the type of the network model included in each tested SDK and software and hardware parameter information corresponding to each tested platform; the information search module 120 is configured to match the type of the network model included in each tested SDK in the historical test record with the type of the target network model, and match software and hardware parameter information corresponding to each tested platform with the target software and hardware parameter information; if the type of the target network model is consistent with the type of the network model contained in one tested SDK in the historical test records, and the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records, determining that the historical test records contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information; otherwise, determining that the historical test records do not contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information.
Optionally, the information search module 120 is configured to determine whether a deviation between the target software and hardware parameter information and software and hardware parameter information of one tested platform in the historical test records is within a preset range; and if so, determining that the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records.
Optionally, the apparatus 100 further comprises:
the recording module is used for obtaining a search result after algorithm strategy search is carried out, and the search result comprises an optimal algorithm strategy used by a network model in the target SDK on the target platform; and storing the search result, the target software and hardware parameter information of the target platform and the target network model in the target SDK into the historical test record.
Optionally, the recording module is configured to read the search result from a first folder created in advance; and migrating the search result from the first folder to a second folder for storing the historical test records.
It should be noted that, for the convenience and brevity of description, the specific working procedure of the above-described apparatus may refer to the corresponding procedure in the foregoing method embodiment, and the description is not repeated herein.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device for executing an algorithm policy search method according to an embodiment of the present application, where the electronic device may include: at least one processor 210, such as a CPU, at least one communication interface 220, at least one memory 230, and at least one communication bus 240. Wherein the communication bus 240 is used for realizing direct connection communication of these components. In the embodiment of the present application, the communication interface 220 of the device is used for performing signaling or data communication with other node devices. Memory 230 may be a high-speed RAM memory or a non-volatile memory (e.g., at least one disk memory). Memory 230 may optionally be at least one memory device located remotely from the aforementioned processor. The memory 230 stores computer readable instructions, which when executed by the processor 210, cause the electronic device to perform the method processes described above with reference to fig. 1.
Embodiments of the present application provide a readable storage medium, and when being executed by a processor, the computer program performs the method processes performed by an electronic device in the method embodiment shown in fig. 1.
The present embodiments disclose a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the methods provided by the above-described method embodiments, for example, comprising: determining a target Software Development Kit (SDK) which needs to perform performance test on a target platform with target software and hardware parameter information at present, wherein the target SDK comprises a target network model; searching whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists in the historical test record or not to obtain a search result; if the search result is that the target test record exists in the historical test record, obtaining a target optimal algorithm strategy obtained by testing the tested SDK in the target test record on the tested platform; and determining the target optimal algorithm strategy as an optimal algorithm strategy of the network model in the target SDK running on the target platform.
In summary, according to the algorithm strategy searching method, device, electronic device and storage medium provided in the embodiments of the present application, a search result is obtained by searching whether a history test record includes a target test record of an SDK including a target network model on a tested platform having target software and hardware parameter information, so that the history test record can be searched before performing algorithm strategy search, and thus when the history test record includes a corresponding target test record, a target optimal algorithm strategy in the target test record can be directly reused as an optimal algorithm strategy of the target SDK running on the target platform, search of the algorithm strategy is not required, time for algorithm strategy search can be saved, time for whole performance test can be reduced, and test efficiency can be improved.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. An algorithmic policy search method, the method comprising:
determining a target Software Development Kit (SDK) which needs to perform performance test on a target platform with target software and hardware parameter information at present, wherein the target SDK comprises a target network model;
searching whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists in the historical test record or not to obtain a search result;
if the search result is that the target test record exists in the historical test record, obtaining a target optimal algorithm strategy obtained by testing the tested SDK in the target test record on the tested platform;
and determining the target optimal algorithm strategy as an optimal algorithm strategy of the network model in the target SDK running on the target platform.
2. The method of claim 1, further comprising:
and if the search result is that the target test record does not exist in the historical test records, carrying out algorithm strategy search.
3. The method of claim 2, wherein the historical test records comprise the types of the network models included in each tested SDK and the software and hardware parameter information corresponding to each tested platform; the method for determining whether the historical test records have the target test records of the tested SDK containing the target network model on the tested platform with the target software and hardware parameter information comprises the following steps:
matching the type of the network model contained in each tested SDK in the historical test record with the type of the target network model, and matching software and hardware parameter information corresponding to each tested platform with the target software and hardware parameter information;
if the type of the target network model is consistent with the type of the network model contained in one tested SDK in the historical test records, and the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records, determining that the historical test records contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information; otherwise, determining that the historical test records do not contain the target test records of the tested SDK of the target network model on the tested platform with the target software and hardware parameter information.
4. The method of claim 3, wherein determining whether the target hardware and software parameter information is consistent with the hardware and software parameter information of one of the tested platforms in the historical test records comprises:
judging whether the deviation between the target software and hardware parameter information and the software and hardware parameter information of one tested platform in the historical test records is within a preset range or not;
and if so, determining that the target software and hardware parameter information is consistent with the software and hardware parameter information of one tested platform in the historical test records.
5. The method of claim 2, wherein after performing the algorithmic policy search, further comprising:
obtaining a search result after algorithm strategy search, wherein the search result comprises an optimal algorithm strategy used by a network model in the target SDK on the target platform;
and storing the search result, the target software and hardware parameter information of the target platform and the target network model in the target SDK into the historical test record.
6. The method of claim 5, wherein storing the search results in the historical test record comprises:
reading the search result from a first folder created in advance;
and migrating the search result from the first folder to a second folder for storing the historical test records.
7. An algorithmic policy search apparatus, the apparatus comprising:
the system comprises an information determining module, a target Software Development Kit (SDK) and a target network model, wherein the information determining module is used for determining the SDK of the target software development kit which needs to perform performance test on a target platform with target software and hardware parameter information at present;
the information searching module is used for searching whether a target test record containing the tested SDK of the target network model on the tested platform with the target software and hardware parameter information exists in the historical test record or not to obtain a searching result;
an algorithm strategy obtaining module, configured to obtain an optimal target algorithm strategy obtained by testing the tested SDK in the target test record on the tested platform if the search result is that the target test record is included in the historical test record;
and the algorithm strategy determining module is used for determining the target optimal algorithm strategy as the optimal algorithm strategy of the network model in the target SDK, which runs on the target platform.
8. The apparatus according to claim 7, wherein the algorithm policy obtaining module is further configured to perform an algorithm policy search if the search result indicates that the target test record does not exist in the historical test records.
9. An electronic device comprising a processor and a memory, the memory storing computer readable instructions that, when executed by the processor, perform the method of any of claims 1-6.
10. A readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011611065.1A CN112685275B (en) | 2020-12-30 | 2020-12-30 | Algorithm policy search method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011611065.1A CN112685275B (en) | 2020-12-30 | 2020-12-30 | Algorithm policy search method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112685275A true CN112685275A (en) | 2021-04-20 |
| CN112685275B CN112685275B (en) | 2024-07-30 |
Family
ID=75455109
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011611065.1A Active CN112685275B (en) | 2020-12-30 | 2020-12-30 | Algorithm policy search method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112685275B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113938473A (en) * | 2021-10-12 | 2022-01-14 | 平安银行股份有限公司 | Automatic Mock method, device, equipment and storage medium based on flow |
| CN115225497A (en) * | 2022-07-13 | 2022-10-21 | 上海壁仞智能科技有限公司 | Server, client, system, method, device and medium for optimizing policy |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090292952A1 (en) * | 2008-05-23 | 2009-11-26 | Microsoft Corporation | Techniques for dynamically determining test platforms |
| CN110633211A (en) * | 2019-08-30 | 2019-12-31 | 深圳壹账通智能科技有限公司 | Multi-interface testing method, device, server and medium |
| CN110659210A (en) * | 2019-09-20 | 2020-01-07 | 深圳创维－Rgb电子有限公司 | Information acquisition method and device, electronic equipment and storage medium |
| CN111552633A (en) * | 2020-03-27 | 2020-08-18 | 深圳壹账通智能科技有限公司 | Interface abnormal call testing method and device, computer equipment and storage medium |
| WO2020233374A1 (en) * | 2019-05-21 | 2020-11-26 | 深圳壹账通智能科技有限公司 | Business platform cache strategy test method and apparatus |
| CN112149266A (en) * | 2020-10-23 | 2020-12-29 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining network model quantization strategy |
-
2020
- 2020-12-30 CN CN202011611065.1A patent/CN112685275B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090292952A1 (en) * | 2008-05-23 | 2009-11-26 | Microsoft Corporation | Techniques for dynamically determining test platforms |
| WO2020233374A1 (en) * | 2019-05-21 | 2020-11-26 | 深圳壹账通智能科技有限公司 | Business platform cache strategy test method and apparatus |
| CN110633211A (en) * | 2019-08-30 | 2019-12-31 | 深圳壹账通智能科技有限公司 | Multi-interface testing method, device, server and medium |
| CN110659210A (en) * | 2019-09-20 | 2020-01-07 | 深圳创维－Rgb电子有限公司 | Information acquisition method and device, electronic equipment and storage medium |
| CN111552633A (en) * | 2020-03-27 | 2020-08-18 | 深圳壹账通智能科技有限公司 | Interface abnormal call testing method and device, computer equipment and storage medium |
| CN112149266A (en) * | 2020-10-23 | 2020-12-29 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining network model quantization strategy |
Non-Patent Citations (2)
| Title |
|---|
| 邢行;尚颖;赵瑞莲;李征;: "面向多目标测试用例优先排序的蚁群算法信息素更新策略", 计算机应用, no. 09 * |
| 邢行;尚颖;赵瑞莲;李征;: "面向多目标测试用例优先排序的蚁群算法信息素更新策略", 计算机应用, no. 09, 10 September 2016 (2016-09-10) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113938473A (en) * | 2021-10-12 | 2022-01-14 | 平安银行股份有限公司 | Automatic Mock method, device, equipment and storage medium based on flow |
| CN115225497A (en) * | 2022-07-13 | 2022-10-21 | 上海壁仞智能科技有限公司 | Server, client, system, method, device and medium for optimizing policy |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112685275B (en) | 2024-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109933520B (en) | Software development testing method and device, computer device and storage medium | |
| JP2012185642A (en) | Test case generation method, program, and system | |
| CN112685275A (en) | Algorithm strategy searching method and device, electronic equipment and storage medium | |
| CN115827636B (en) | Method for storing and reading simulation data of logic system design from waveform database | |
| US11663113B2 (en) | Real time fault localization using combinatorial test design techniques and test case priority selection | |
| CN113094252B (en) | Test case generation method and device, computer equipment and storage medium | |
| CN112231229A (en) | Web UI (user interface) automatic testing method and system, electronic equipment and readable storage medium | |
| CN115617819B (en) | Data storage method, system, computer device and storage medium | |
| CN109359093A (en) | A kind of rule file update method and system | |
| CN115543435A (en) | Software development quality measuring method, device and equipment based on development contribution value | |
| CN114564228A (en) | Application program updating method and device, computer equipment and storage medium | |
| CN110866492B (en) | Baseline branch identification method and device and computer system | |
| CN115587048A (en) | Regression testing method, terminal device and computer readable storage medium | |
| CN111008140B (en) | Cross-platform UI (user interface) automatic testing method and device | |
| CN115168124A (en) | Stability test method and device for concurrently executing multiple files on hard disk, computer equipment and medium | |
| CN115098362A (en) | Page testing method and device, electronic equipment and storage medium | |
| CN112559331A (en) | Test method and device | |
| CN118051779B (en) | Automatic parameter searching method and device for large model training and electronic equipment | |
| CN110795297B (en) | Electronic equipment testing method and device | |
| CN111078543B (en) | System dynamic test method and test device | |
| JP6902513B2 (en) | Source code generation support device and source code generation support method | |
| CN118708230A (en) | Java-based object relationship determination method, device, equipment and storage medium | |
| CN118606186A (en) | Application software testing method and device, storage medium and computer equipment | |
| CN115904940A (en) | Self-adaptive regression random test method, device, terminal and medium | |
| CN115904906A (en) | Hard disk performance index data testing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |