CN110737572A - Big data platform resource preemption test method, system, terminal and storage medium - Google Patents
Big data platform resource preemption test method, system, terminal and storage medium Download PDFInfo
- Publication number
- CN110737572A CN110737572A CN201910820062.XA CN201910820062A CN110737572A CN 110737572 A CN110737572 A CN 110737572A CN 201910820062 A CN201910820062 A CN 201910820062A CN 110737572 A CN110737572 A CN 110737572A
- Authority
- CN
- China
- Prior art keywords
- task
- time
- task queue
- resource
- preemption
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides big data platform resource preemption testing method, system, terminal and storage medium, which comprises the steps of respectively setting task queue resource occupation ratio and second task queue resource occupation ratio, setting task data and constructing query tasks for the task data, respectively submitting the query tasks to a task queue and a second task queue under the condition of cluster no-load to obtain processing time and second processing time required by task completion and ensure that the processing time is equal to the second processing time, submitting the query tasks to a task queue and submitting the query tasks to the second task queue when the query tasks of a task queue are executed to a preset progress to obtain a third time for completing the tasks of the task queue and a fourth time for completing the tasks of the second task queue, and monitoring resource preemption events according to a comparison relation among the time, the second time, the third time, the fourth time, the task queue resource occupation ratio and the second task queue resource occupation ratio.
Description
Technical Field
The invention relates to the technical field of big data service platforms, in particular to big data platform resource preemption testing methods, systems, terminals and storage media.
Background
The big data component YARN has three scheduling modes, Fair scheduling (Fair Scheduler) is which is better in applicability than scheduling mode, and the mode supports resource preemption, so that the processing speed is high when the tasks are few, and Fair scheduling is performed when the tasks are concurrent, so that the resource preemption function is important functions of the component.
Disclosure of Invention
In view of the above-mentioned deficiencies of the prior art, the present invention provides methods, systems, terminals and storage media for preemptive testing of large data platform resources, so as to solve the above-mentioned technical problems.
, the invention provides a big data platform resource preemption test method, which comprises the following steps:
th task queue resource occupation ratio and second task queue resource occupation ratio are respectively set;
setting task data and constructing a query task for the task data;
under the condition that the cluster is idle, submitting the query task to an th task queue and a second task queue respectively to obtain th processing time and second processing time required by task completion, and ensuring that the th processing time and the second processing time are equal;
submitting a query task to an th task queue, submitting the query task to a second task queue when the query task in a th task queue is executed to a preset progress, and acquiring a third time for the th task queue to complete the task and a fourth time for the second task queue to complete the task;
and monitoring the resource preemption event according to the comparison relationship among th time, second time, third time, fourth time, th task queue resource occupation ratio and the second task queue resource occupation ratio.
, the setting and second task queue resource occupation ratios respectively includes:
setting a proportion difference according to the discrimination requirement of the task execution time;
setting a scheduling mode of a big data platform as fair scheduling;
and setting th task queue resource occupation ratio as a low occupation ratio, setting the resource occupation ratio of the second task queue as a high occupation ratio, wherein the difference between the low occupation ratio and the high occupation ratio is not less than the preset occupation ratio difference.
, the submitting the query task to the task queue and submitting the query task to the second task queue when the query task in the task queue is executed to a preset schedule includes:
setting a preemption trigger progress, and converting preemption trigger time according to the preemption trigger progress;
submitting the query task to a th task queue and recording the submission time;
and when the execution time of the th task queue reaches the preemption trigger time, submitting a query task to the second task queue.
, the monitoring the resource preemption event according to the comparison among th time, the second time, the third time, the fourth time, the th task queue resource occupation ratio and the second task queue resource occupation ratio includes:
confirming that the third time and the fourth time are both less than the th time and the second time;
and generating a proportional relation between the task execution time and the resource ratio according to the third time, the fourth time, the th task queue resource ratio and the second task queue ratio, and judging that a resource preemption event occurs if the proportional relation is a direct ratio.
In a second aspect, the present invention provides big data platform resource preemption testing systems, including:
the queue setting unit is configured to set th task queue resource occupation ratio and a second task queue resource occupation ratio respectively;
the task construction unit is configured for setting task data and constructing a query task of the task data;
the no-load execution unit is configured to submit the query task to the th task queue and the second task queue respectively under the condition that the cluster is no-load, so that the th processing time and the second processing time required by task completion are obtained, and the th processing time and the second processing time are ensured to be equal;
the preemption simulation unit is configured to submit a query task to the th task queue and submit the query task to the second task queue when the query task in the th task queue is executed to a preset progress, and obtain a third time for the th task queue to complete the task and a fourth time for the second task queue to complete the task;
and the preemption monitoring unit is configured to monitor the resource preemption event according to the comparison relationship among th time, second time, third time, fourth time, th task queue resource occupation ratio and the second task queue resource occupation ratio.
Further , the queue setting unit includes:
the difference setting module is configured for setting a ratio difference according to the discrimination requirement on the task execution time;
the mode setting module is configured for setting a scheduling mode of the big data platform as fair scheduling;
and the duty ratio setting module is configured to set the resource duty ratio of the th task queue to a low-equal duty ratio and set the resource duty ratio of the second task queue to a high-equal duty ratio, wherein the difference between the low-equal duty ratio and the high-equal duty ratio is not less than a preset duty ratio difference value.
Further to , the preemptive simulate unit includes:
the trigger setting module is configured for setting a preemption trigger progress and converting preemption trigger time according to the preemption trigger progress;
the initial execution module is configured to submit the query task to the th task queue and record the submission time;
and the trigger execution module is configured to submit the query task to the second task queue when the execution time of the th task queue reaches the preemption trigger time.
, the preemption monitoring unit includes:
a preliminary confirmation module configured to confirm that the third time and the fourth time are both less than the th time and the second time;
and the proportion judging module is configured to generate a proportional relation between the task execution time and the resource proportion according to the third time, the fourth time, the th task queue resource proportion and the second task queue proportion, and if the proportional relation is a direct proportion, the occurrence of the resource preemption event is judged.
In a third aspect, terminals are provided, including:
a processor, a memory, wherein,
the memory is used for storing a computer program which,
the processor is used for calling and running the computer program from the memory so as to make the terminal execute the method of the terminal.
In a fourth aspect, there are computer storage media having instructions stored thereon that, when executed on a computer, cause the computer to perform the method of the above aspects.
The beneficial effect of the invention is that,
the invention provides a big data platform resource preemption test method, a system, a terminal and a storage medium, under a fair scheduling mode, 2 queues with larger allocable resource difference are arranged, and at different idle time periods of a cluster, a reference task is submitted to the two queues, the reference task can be executed for a longer time, the execution time is recorded, then a scene of resource preemption capture is constructed, the reference task is submitted to the queue with smaller allocable resource, and in the execution process of the small queues, the reference task is submitted to the queue with larger allocable resource and the time is respectively recorded. And finally, whether the resource preemption event occurs or not can be analyzed through the comparison of the four execution times. The invention can provide a specific resource preemption test method for testers, thereby solving the problem of the lack of the resource preemption test method for YARN component fair scheduling, providing method innovation for cluster function test, and providing a reference basis for a platform to support resource preemption scheduling for customers, thereby ensuring the quality of products.
In addition, the invention has reliable design principle, simple structure and very wide application prospect of .
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of embodiments of the invention.
Fig. 2 is a schematic block diagram of a system of embodiments of the invention.
Fig. 3 is a schematic structural diagram of terminals according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a partial embodiment of of the present invention, rather than a whole embodiment.
FIG. 1 is a schematic flow chart of a method of embodiments of the invention, wherein the execution subject of FIG. 1 can be a large data platform resource preemption test system.
As shown in fig. 1, the method 100 includes:
step 110, respectively setting th task queue resource occupation ratio and second task queue resource occupation ratio;
step 120, setting task data and constructing a query task for the task data;
130, under the condition that the cluster is in no-load, submitting the query task to an th task queue and a second task queue respectively to obtain th processing time and second processing time required by task completion, and ensuring that the th processing time and the second processing time are equal;
step 140, submitting a query task to the th task queue, submitting the query task to the second task queue when the query task in the th task queue is executed to a preset schedule, and acquiring a third time for the th task queue to complete the task and a fourth time for the second task queue to complete the task;
and 150, monitoring a resource preemption event according to the comparison relationship among th time, second time, third time, fourth time, th task queue resource occupation ratio and the second task queue resource occupation ratio.
Optionally, as embodiments of the present invention, the setting of the th task queue resource proportion and the second task queue resource proportion respectively includes:
setting a proportion difference according to the discrimination requirement of the task execution time;
setting a scheduling mode of a big data platform as fair scheduling;
and setting th task queue resource occupation ratio as a low occupation ratio, setting the resource occupation ratio of the second task queue as a high occupation ratio, wherein the difference between the low occupation ratio and the high occupation ratio is not less than the preset occupation ratio difference.
Optionally, as embodiments of the present invention, the submitting a query task to the task queue and submitting the query task to the second task queue when the query task in the task queue executes to a preset schedule includes:
setting a preemption trigger progress, and converting preemption trigger time according to the preemption trigger progress;
submitting the query task to a th task queue and recording the submission time;
and when the execution time of the th task queue reaches the preemption trigger time, submitting a query task to the second task queue.
Optionally, as embodiments of the present invention, the monitoring the resource preemption event according to an alignment relationship among a th time, a second time, a third time, a fourth time, a th task queue resource duty ratio and a second task queue resource duty ratio includes:
confirming that the third time and the fourth time are both less than the th time and the second time;
and generating a proportional relation between the task execution time and the resource ratio according to the third time, the fourth time, the th task queue resource ratio and the second task queue ratio, and judging that a resource preemption event occurs if the proportional relation is a direct ratio.
In order to facilitate understanding of the present invention, the following describes a method for preemptively testing resources of a big data platform according to the principle of the method for preemptively testing resources of a big data platform according to the present invention in combination with a process of performing simulation triggering and monitoring on a resource preemption event in the embodiment.
In this embodiment, taking a cloud sea Insight big data platform as an example, specifically, the method for preempting and testing the resources of the big data platform includes:
and S1, respectively setting the st and second task queue resource occupation ratios.
The scheduling mode of the YARN component is set as Fair scheduling (Fair Scheduler). firstly, a resource ratio difference value is set according to the screening requirement of the execution time, namely the difference value of a high resource ratio and a low resource ratio cannot be smaller than a preset difference value, so that the task execution time difference of two task queues is large, and the task execution time is convenient for a tester to screen.A resource ratio difference value is set to be 40%, th task queue (the queue name can be self-named, such as queue dev) is set to have a resource ratio of 20%, and a second task queue (queue poc) is set to have a resource ratio of 80%.
And S2, setting task data and constructing a query task for the task data.
Constructing an executable benchmark task, ensuring that the task can be executed for time periods, for example, the task can be executed for about 10min, constructing 300GB data by using a tool TPC-DS, and querying by using Query _64.sql, namely a Query _64.sql Query task.
And S3, under the condition that the cluster is in no-load, submitting the query task to the task queue and the second task queue respectively to obtain the processing time and the second processing time required by task completion, and ensuring that the processing time is equal to the second processing time.
In the case of no load of the cluster, the Query _64.sql task is submitted to the queue dev, and the total TIME1 for the task to run to completion is recorded. In the case of an empty cluster, the Query _64.sql task is submitted to queue poc and the total TIME of completion of the task run is recorded TIME 2. If TIME1 is equal to TIME2, it indicates that the large data platform is in fair scheduling mode, and the test can be continued. If TIME1 is not equal to TIME2, it indicates that the big data platform is not in fair scheduling mode, and it needs to exit the test and prompt a mode error.
S4, submitting the query task to the task queue, submitting the query task to the second task queue when the query task in the task queue is executed to the preset progress, and acquiring third time for the task queue to complete the task and fourth time for the second task queue to complete the task.
For example, the resource occupation ratios set in this embodiment are respectively 20% of the dev resource occupation ratio of the queue and 80% of the poc resource occupation ratio of the queue, and then the queue poc needs to start to execute the task when the dev task progress of the queue reaches third.
If the TIME1 is TIME2> TIME3 and the TIME1 is TIME2> TIME4, executing next judgments, otherwise, judging that the resource preemption event does not occur;
and generating a proportional relation between the task execution time and the resource ratio according to the third time, the fourth time, the th task queue resource ratio and the second task queue ratio:
if TIME4 is more than TIME3, the proportional relation is in a direct proportion, and a resource preemption event is judged to occur; and if the TIME3 is greater than the TIME4, the proportional relation is in inverse proportion, and the resource preemption event is judged not to occur.
As shown in fig. 2, the system 200 includes:
the queue setting unit 210 is configured to set th task queue resource occupation ratio and a second task queue resource occupation ratio respectively;
a task construction unit 220 configured to set task data and construct a query task for the task data;
the no-load execution unit 230 is configured to submit the query task to the th task queue and the second task queue respectively under the condition that the cluster is no-load, so as to obtain th processing time and second processing time required by task completion, and ensure that the th processing time and the second processing time are equal;
the preemption simulation unit 240 is configured to submit a query task to the th task queue and submit the query task to the second task queue when the query task in the th task queue is executed to a preset progress, so as to obtain a third time for the th task queue to complete the task and a fourth time for the second task queue to complete the task;
and the preemption monitoring unit 250 is configured to monitor the resource preemption event according to the comparison relationship among the th time, the second time, the third time, the fourth time, the th task queue resource occupation ratio and the second task queue resource occupation ratio.
Optionally, as embodiments of the present invention, the queue setting unit includes:
the difference setting module is configured for setting a ratio difference according to the discrimination requirement on the task execution time;
the mode setting module is configured for setting a scheduling mode of the big data platform as fair scheduling;
and the duty ratio setting module is configured to set the resource duty ratio of the th task queue to a low-equal duty ratio and set the resource duty ratio of the second task queue to a high-equal duty ratio, wherein the difference between the low-equal duty ratio and the high-equal duty ratio is not less than a preset duty ratio difference value.
Optionally, as embodiments of the present invention, the preemptive simulate unit includes:
the trigger setting module is configured for setting a preemption trigger progress and converting preemption trigger time according to the preemption trigger progress;
the initial execution module is configured to submit the query task to the th task queue and record the submission time;
and the trigger execution module is configured to submit the query task to the second task queue when the execution time of the th task queue reaches the preemption trigger time.
Optionally, as embodiments of the present invention, the preemption monitoring unit includes:
a preliminary confirmation module configured to confirm that the third time and the fourth time are both less than the th time and the second time;
and the proportion judging module is configured to generate a proportional relation between the task execution time and the resource proportion according to the third time, the fourth time, the th task queue resource proportion and the second task queue proportion, and if the proportional relation is a direct proportion, the occurrence of the resource preemption event is judged.
Fig. 3 is a schematic structural diagram of terminal systems 300 according to an embodiment of the present invention, where the terminal system 300 may be used to execute the big data platform resource preemption testing method according to the embodiment of the present invention.
The terminal system 300 may include a processor 310, a memory 320, and a communication unit 330, which communicate via buses, and it will be understood by those skilled in the art that the structure of the server shown in the figure is not a limitation of the present invention, and may be a bus structure, a star structure, a combination of more or less components than those shown, or a different arrangement of components.
The memory 320 may be used for storing instructions executed by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 320, when executed by processor 310, enable terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 320 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 310 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 330, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
The present invention further provides computer storage media, wherein the computer storage media can store programs, and the programs can include some or all of the steps in the embodiments of the present invention when executed.
Therefore, in the fair scheduling mode, 2 queues with larger allocable resource difference are set, and at different idle time periods of the cluster, a reference task is submitted to the two queues, the reference task can be executed for a longer time, the execution time is recorded, then a scene that the resource seizes back is constructed, the reference task is submitted to the queue with smaller allocable resource, and in the execution process of the small queue, the reference task is submitted to the queue with larger allocable resource, and the time is recorded respectively. And finally, whether the resource preemption event occurs or not can be analyzed through the comparison of the four execution times. The invention can provide a specific resource preemption test method for testers, thereby solving the problem of the lack of the resource preemption test method for YARN component fair scheduling, providing method innovation for cluster function test, and providing a reference basis for a platform to support resource preemption scheduling for customers, thereby ensuring the product quality.
Based on the understanding that the technical solutions in the embodiments of the present invention or portions thereof contributing to the prior art can be embodied in the form of software products stored in storage media such as a usb disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes, which include instructions for causing computer terminals (which may be personal computers, servers, or secondary terminals, network terminals, etc.) to execute all or part of the steps of the methods described in the embodiments of the present invention.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
For example, the above-described system embodiments are merely illustrative, e.g., the division of the units into logical functional divisions, and other divisions may be possible in actual practice, e.g., multiple units or components may be combined or integrated into another systems, or features may be omitted or not implemented.at , the shown or discussed coupling or direct coupling or communication connection between each other may be through interfaces, and the indirect coupling or communication connection of the systems or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in places, or may also be distributed on multiple network units.
In addition, functional units in the embodiments of the present invention may be integrated into processing units, or each unit may exist alone physically, or two or more units are integrated into units.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1, big data platform resource preemption test method, characterized by comprising:
th task queue resource occupation ratio and second task queue resource occupation ratio are respectively set;
setting task data and constructing a query task for the task data;
under the condition that the cluster is idle, submitting the query task to an th task queue and a second task queue respectively to obtain th processing time and second processing time required by task completion, and ensuring that the th processing time and the second processing time are equal;
submitting a query task to an th task queue, submitting the query task to a second task queue when the query task in a th task queue is executed to a preset progress, and acquiring a third time for the th task queue to complete the task and a fourth time for the second task queue to complete the task;
and monitoring the resource preemption event according to the comparison relationship among th time, second time, third time, fourth time, th task queue resource occupation ratio and the second task queue resource occupation ratio.
2. The big data platform resource preemption test method of claim 1, wherein the setting th and second task queue resource proportions, respectively, comprises:
setting a proportion difference according to the discrimination requirement of the task execution time;
setting a scheduling mode of a big data platform as fair scheduling;
and setting th task queue resource occupation ratio as a low occupation ratio, setting the resource occupation ratio of the second task queue as a high occupation ratio, wherein the difference between the low occupation ratio and the high occupation ratio is not less than the preset occupation ratio difference.
3. The big data platform resource preemption test method of claim 1, wherein the submitting a query task to an th task queue and submitting the query task to a second task queue when the query task in a th task queue executes to a preset schedule comprises:
setting a preemption trigger progress, and converting preemption trigger time according to the preemption trigger progress;
submitting the query task to a th task queue and recording the submission time;
and when the execution time of the th task queue reaches the preemption trigger time, submitting a query task to the second task queue.
4. The big data platform resource preemption test method of claim 1, wherein the monitoring resource preemption events according to an alignment between th time, second time, third time, fourth time, th task queue resource occupancy and second task queue resource occupancy comprises:
confirming that the third time and the fourth time are both less than the th time and the second time;
and generating a proportional relation between the task execution time and the resource ratio according to the third time, the fourth time, the th task queue resource ratio and the second task queue ratio, and judging that a resource preemption event occurs if the proportional relation is a direct ratio.
The resource preemption test system for the large data platform of types is characterized by comprising the following steps:
the queue setting unit is configured to set th task queue resource occupation ratio and a second task queue resource occupation ratio respectively;
the task construction unit is configured for setting task data and constructing a query task of the task data;
the no-load execution unit is configured to submit the query task to the th task queue and the second task queue respectively under the condition that the cluster is no-load, so that the th processing time and the second processing time required by task completion are obtained, and the th processing time and the second processing time are ensured to be equal;
the preemption simulation unit is configured to submit a query task to the th task queue and submit the query task to the second task queue when the query task in the th task queue is executed to a preset progress, and obtain a third time for the th task queue to complete the task and a fourth time for the second task queue to complete the task;
and the preemption monitoring unit is configured to monitor the resource preemption event according to the comparison relationship among th time, second time, third time, fourth time, th task queue resource occupation ratio and the second task queue resource occupation ratio.
6. The big data platform resource preemption test system of claim 5, wherein the queue setting unit comprises:
the difference setting module is configured for setting a ratio difference according to the discrimination requirement on the task execution time;
the mode setting module is configured for setting a scheduling mode of the big data platform as fair scheduling;
and the duty ratio setting module is configured to set the resource duty ratio of the th task queue to a low-equal duty ratio and set the resource duty ratio of the second task queue to a high-equal duty ratio, wherein the difference between the low-equal duty ratio and the high-equal duty ratio is not less than a preset duty ratio difference value.
7. The big data platform resource preemption test system of claim 5, wherein the preemption simulation unit comprises:
the trigger setting module is configured for setting a preemption trigger progress and converting preemption trigger time according to the preemption trigger progress;
the initial execution module is configured to submit the query task to the th task queue and record the submission time;
and the trigger execution module is configured to submit the query task to the second task queue when the execution time of the th task queue reaches the preemption trigger time.
8. The big data platform resource preemption test system of claim 5, wherein the preemption monitoring unit comprises:
a preliminary confirmation module configured to confirm that the third time and the fourth time are both less than the th time and the second time;
and the proportion judging module is configured to generate a proportional relation between the task execution time and the resource proportion according to the third time, the fourth time, the th task queue resource proportion and the second task queue proportion, and if the proportional relation is a direct proportion, the occurrence of the resource preemption event is judged.
A terminal of the type 9, , comprising:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform the method of any of claims 1-4.
10, computer-readable storage medium storing a computer program, characterized in that the program, when being executed by a processor, carries out the method according to any of claims 1-4 .
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910820062.XA CN110737572B (en) | 2019-08-31 | 2019-08-31 | Big data platform resource preemption test method, system, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910820062.XA CN110737572B (en) | 2019-08-31 | 2019-08-31 | Big data platform resource preemption test method, system, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737572A true CN110737572A (en) | 2020-01-31 |
| CN110737572B CN110737572B (en) | 2023-01-10 |
Family
ID=69267419
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910820062.XA Active CN110737572B (en) | 2019-08-31 | 2019-08-31 | Big data platform resource preemption test method, system, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737572B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112418588A (en) * | 2020-09-03 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Task flow display method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108228324A (en) * | 2016-12-21 | 2018-06-29 | 天脉聚源（北京）科技有限公司 | A kind of server cluster seizes the method and system of distribution task |
| CN108769254A (en) * | 2018-06-25 | 2018-11-06 | 星环信息科技（上海）有限公司 | Resource-sharing application method, system and equipment based on preemption scheduling |
| CN109901921A (en) * | 2019-02-22 | 2019-06-18 | 北京致远互联软件股份有限公司 | Task queue running time prediction method, apparatus and realization device |
-
2019
- 2019-08-31 CN CN201910820062.XA patent/CN110737572B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108228324A (en) * | 2016-12-21 | 2018-06-29 | 天脉聚源（北京）科技有限公司 | A kind of server cluster seizes the method and system of distribution task |
| CN108769254A (en) * | 2018-06-25 | 2018-11-06 | 星环信息科技（上海）有限公司 | Resource-sharing application method, system and equipment based on preemption scheduling |
| CN109901921A (en) * | 2019-02-22 | 2019-06-18 | 北京致远互联软件股份有限公司 | Task queue running time prediction method, apparatus and realization device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112418588A (en) * | 2020-09-03 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Task flow display method and device |
| CN112418588B (en) * | 2020-09-03 | 2023-08-01 | 上海哔哩哔哩科技有限公司 | Task flow display method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737572B (en) | 2023-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3447642B1 (en) | System and method for predicting application performance for large data size on big data cluster | |
| CN106776280B (en) | Configurable performance testing device | |
| CN110377520B (en) | Transaction scenario testing method and device, electronic equipment and readable storage medium | |
| CN109144700B (en) | Method and device for determining timeout duration, server and data processing method | |
| US8606905B1 (en) | Automated determination of system scalability and scalability constraint factors | |
| CN108737218B (en) | Automatic verification method and device for message push arrival rate | |
| CN110012062B (en) | Multi-computer-room task scheduling method and device and storage medium | |
| EP4095700A1 (en) | Method and system for micro-service testing, and storage medium | |
| CN114546738A (en) | Server general test method, system, terminal and storage medium | |
| CN114064208A (en) | Method and device for detecting application service state, electronic equipment and storage medium | |
| CN111258913A (en) | Automatic algorithm testing method and device, computer system and readable storage medium | |
| US20140023185A1 (en) | Characterizing Time-Bounded Incident Management Systems | |
| CN113238930A (en) | Software system testing method and device, terminal equipment and storage medium | |
| CN108829519A (en) | Method for scheduling task, cloud platform and computer readable storage medium based on cloud platform | |
| CN108616668B (en) | Method for monitoring dialing task, electronic device and computer readable storage medium | |
| CN114817050A (en) | Task execution method and device, electronic equipment and computer readable storage medium | |
| CN110737572B (en) | Big data platform resource preemption test method, system, terminal and storage medium | |
| CN111124854A (en) | Method, system, terminal and storage medium for distributing smoking test cases | |
| EP4152715A1 (en) | Method and apparatus for determining resource configuration of cloud service system | |
| CN108763052A (en) | A kind of method for testing performance and system of virtual reality software memory reclaim mechanism | |
| CN112256588A (en) | Resource allocation method for application program test, computer readable storage medium and tester | |
| CN112052070A (en) | Application containerization evaluation method and device, electronic equipment and storage medium | |
| CN111639022A (en) | Transaction testing method and device, storage medium and electronic device | |
| CN107391262B (en) | Job scheduling method and device | |
| CN113535439B (en) | Service request processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |