WO2022267733A1

WO2022267733A1 - Method for dynamically adjusting frame-dropping threshold value, and related devices

Info

Publication number: WO2022267733A1
Application number: PCT/CN2022/092369
Authority: WO
Inventors: 贾睿
Original assignee: 荣耀终端有限公司
Priority date: 2021-06-25
Filing date: 2022-05-12
Publication date: 2022-12-29
Also published as: CN113473229A; CN113473229B

Abstract

Disclosed in the present application are a method for dynamically adjusting a frame-dropping threshold value, and related devices. The method comprises: a second electronic device being able to receive a video frame that is sent by a first electronic device, and recording the time at which the video frame is received; the second electronic device determining a frame-receiving time difference, wherein the frame-receiving time difference is the difference between the time at which the video frame is received and the time at which an Mth video frame is received, and the Mth video frame is a video frame that is sent by the first electronic device before the first electronic device sends said video frame; if the frame-receiving time difference is not less than a frame-dropping threshold value, the second electronic device further being able to adjust the frame-dropping threshold value; in addition, after adjusting the frame-dropping threshold value, the second electronic device further being able to perform a threshold value test, that is, determining whether a decoding delay and a display delay after the frame-dropping threshold value is adjusted are reduced. By means of the above method, not only can the problem of video frame blocking under different network conditions and in different device states be handled, but the effect of adjusting the frame-dropping threshold value can also be fed back in a timely manner, thereby ensuring that the problem of video frame blocking can be effectively solved on the basis of the current frame-dropping threshold value.

Description

A method and related equipment for dynamically adjusting the frame loss threshold

This application claims the priority of the Chinese patent application with the application number 202110713481.0 and the application title "A Method for Dynamically Adjusting the Frame Loss Threshold and Related Equipment" submitted to the China Patent Office on June 25, 2021, the entire contents of which are incorporated by reference incorporated in this application.

technical field

The present application relates to the field of screen mirroring, and in particular to a method and related equipment for dynamically adjusting a frame loss threshold.

Background technique

With the continuous development of Internet technology, screen projection technology has been widely used, bringing great convenience to people's work and life. Screen projection is divided into two types: mirror projection and non-mirror projection. Mirror projection refers to the real-time projection of the screen of one device to another device for display. It is especially useful in scenarios such as business meetings.

There will be video frame transmission during the mirroring process, and the transmission of video frames is easily affected by the network. If the network condition is not good, the device that needs to receive the video frame cannot actually receive the video frame. After the subsequent network recovery, the device A large number of video frames will be received in a short period of time, which exceeds the decoding capability of the device's decoder, resulting in a backlog of a large number of video frames in the decoder. The decoder cannot decode these video frames in time, which increases the screen projection delay.

Therefore, how to effectively solve the problem of video frame blocking under different network conditions and different device states is an urgent problem to be solved at present.

Contents of the invention

This application provides a method for dynamically adjusting the frame loss threshold. The initial frame loss threshold can be set, and the frame loss threshold can be dynamically adjusted according to the time difference of receiving video frames by the second electronic device, that is, the current network status can be judged according to the time difference of receiving video frames and Correspondingly adjust the frame loss threshold, and then perform the frame loss operation, which can effectively solve the video frame blocking problem under different network conditions. In addition, after the frame loss is completed, judge whether the decoding delay and the display delay after adjusting the frame loss threshold are reduced, and determine whether the frame loss threshold needs to be adjusted again according to the judgment result, so that the effect of the frame loss threshold can be fed back in time , to ensure that the current frame loss threshold can effectively solve the problem of video frame blocking.

In a first aspect, the present application provides a method for dynamically adjusting a frame loss threshold. The method can be applied to the second electronic device. The method may include: receiving the video frame sent by the first electronic device; determining the frame receiving time difference; the frame receiving time difference is the difference between the time of receiving the video frame and the time of receiving the Mth video frame; the Mth video frame is The video frame sent by the first electronic device before sending the video frame; when the frame receiving time difference is not less than the frame loss threshold, if the first duration does not reach the preset time, reduce the frame loss threshold; the first duration is above Adjust the duration from the frame loss threshold to the current moment once; the frame loss threshold is used to determine whether to discard the video frame; or, when the frame receiving time difference is not less than the frame loss threshold, if the first duration reaches the preset time, and the frame loss If the frame threshold is less than the upper threshold, increase the frame loss threshold; if the first duration reaches a preset time, and the frame loss threshold is not less than the upper threshold, reduce the frame loss threshold; the upper threshold is the The maximum value of the frame loss threshold; record the decoding delay and display delay of N frames of video frames received after receiving the video frame; the decoding delay is the time from when the video frame arrives at the decoder to the completion of decoding; when sending the display Delay is the time from the completion of decoding to the time when the video frame is displayed on the display screen; judge whether N is equal to the preset number of frames; if N is equal to the preset number of frames, judge whether the adjusted frame loss threshold is valid; if the adjusted frame loss If the threshold is invalid, reduce the adjusted frame loss threshold and stop the threshold test; the threshold test is used to determine whether the adjustment to the frame loss threshold is valid.

In the solution provided by the present application, the second electronic device can adjust the frame loss threshold to adapt to video frame blocking problems under different network conditions and device states. In addition, after the frame loss is completed, the second electronic device can also perform a threshold test, that is, determine whether the decoding delay and the display delay after adjusting the frame loss threshold are reduced, and determine whether it is necessary to adjust the frame loss threshold again according to the judgment result , so that the effect of the frame loss threshold can be fed back in time, ensuring that the current frame loss threshold can effectively solve the problem of video frame blocking.

With reference to the first aspect, in a possible implementation manner of the first aspect, the video frame is a video frame not used for a threshold test.

In the solution provided by the present application, the video frames received by the second electronic device and sent by the first electronic device may not be used for the threshold test. At this time, the second electronic device can judge whether to adjust the frame loss threshold according to the time of receiving the video frame, which can avoid the inaccurate threshold test caused by adjusting the frame loss threshold when the video frame is used for the threshold test.

With reference to the first aspect, in a possible implementation manner of the first aspect, judging whether the adjusted frame loss threshold is valid includes: determining the average decoding delay and the average display delay in the current full time period; The current full time period is a period of time from receiving the first video frame to determining the average decoding delay and average display delay; if the decoding delay and display delay of N frames of video frames are respectively higher than the average decoding The delay and the average display delay are reduced by at least c%, and it is determined that the adjusted frame loss threshold is valid.

In the solution provided by this application, the second electronic device can adjust the average decoding delay and the average decoding delay and average Send to display delay to judge whether the adjusted frame loss threshold can effectively solve the problem of video frame blocking, that is, whether it can reduce decoding delay and display delay.

With reference to the first aspect, in a possible implementation manner of the first aspect, the second electronic device may discard the video frame when the frame receiving time difference is smaller than the frame loss threshold.

In the solution provided by the present application, when the frame receiving time difference is smaller than the frame loss threshold, the second electronic device judges that a large number of video frames have been received in a short period of time, so the video frames just received can be directly discarded. This can reduce the waiting time for decoding, thereby solving the problem of video frame blocking.

With reference to the first aspect, in a possible implementation manner of the first aspect, after the second electronic device receives the video frame sent by the first electronic device, it may also record the time when the video frame is received, and perform a check on the frame loss threshold Initialize processing. It can be understood that the order in which the second electronic device records the time of receiving the video frame and initializes the frame loss threshold is not limited.

In the solution provided by the present application, the second electronic device may record the time when the video frame is received, so as to subsequently calculate the frame receiving time difference.

With reference to the first aspect, in a possible implementation of the first aspect, after recording the time of receiving the video frame, the second electronic device may also store the time of receiving the video frame in the first queue; The time when the second electronic device receives the Mth video frame is stored in a queue.

In the solution provided by the present application, a queue may be set to store the time when the second electronic device receives the video frame, so as to facilitate processing the time when the second electronic device receives the video frame.

With reference to the first aspect, in a possible implementation manner of the first aspect, before the recording delay of decoding and display delay of N video frames received after receiving the video frame, the second electronic device further The first written element in the first queue may be removed, and the time at which the video frame is received is written into the first queue.

In the solution provided by the present application, the second electronic device can adjust the first queue in time to facilitate the calculation of the frame receiving time difference next time.

In a second aspect, the present application provides an electronic device, which may include a display screen, a memory, and one or more processors, wherein the memory is used to store computer programs; the processor is used to call the A computer program, so that the electronic device executes: receiving the video frame sent by the first electronic device; determining the frame receiving time difference; the frame receiving time difference is the difference between the time of receiving the video frame and the time of receiving the Mth video frame; the second M frames of video frames are video frames sent by the first electronic device before sending the video frame; in the case that the frame receiving time difference is not less than the frame loss threshold, if the first duration does not reach the preset time, reduce the frame loss threshold; The first duration is the duration from the last time the frame loss threshold was adjusted to the current moment; the frame loss threshold is used to determine whether to discard the video frame; or, when the frame receiving time difference is not less than the frame loss threshold, if the first duration reaches the predetermined Set the time, and the frame loss threshold is less than the upper threshold, increase the frame loss threshold; if the first duration reaches the preset time, and the frame loss threshold is not less than the upper threshold, reduce the frame loss threshold; The upper limit of the threshold is the maximum value of the frame loss threshold; record the decoding delay and the display delay of the N frames of video frames received after receiving the video frame; the decoding delay is from the arrival of the video frame to the completion of the decoding Time; the display delay is the time from the completion of decoding to the time when the video frame is displayed on the display screen; judge whether N is equal to the preset number of frames; if N is equal to the preset number of frames, judge whether the adjusted frame loss threshold is valid; if The adjusted frame loss threshold is invalid, reduce the adjusted frame loss threshold, and stop the threshold test; the threshold test is used to determine whether the adjustment to the frame loss threshold is valid.

With reference to the second aspect, in a possible implementation manner of the second aspect, the video frame is a video frame not used for a threshold test.

With reference to the second aspect, in a possible implementation manner of the second aspect, the processor is configured to call the computer program, so that the electronic device executes when judging whether the adjusted frame loss threshold is valid, specifically using Invoking the computer program, so that the electronic device executes: determining the average decoding delay and the average display delay in the current full time period; the current full time period is from receiving the first video frame to determining the average decoding delay A period of time until the delay and the average display delay; if the decoding delay and the display delay of N frames of video frames are respectively reduced by at least c% compared with the average decoding delay and the average display delay, determine the adjusted loss Frame Threshold is in effect.

With reference to the second aspect, in a possible implementation manner of the second aspect, the processor may be further configured to discard the video frame when the frame receiving time difference is less than a frame loss threshold.

With reference to the second aspect, in a possible implementation manner of the second aspect, after the processor is used to invoke the computer program so that the electronic device executes receiving the video frame sent by the first electronic device, further It can be used to call the computer program, so that the electronic device executes: recording the time of receiving the video frame; and initializing the frame loss threshold.

With reference to the second aspect, in a possible implementation manner of the second aspect, after the processor is used to call the computer program so that the electronic device executes recording the time at which the video frame is received, the processor may further The method is used to call the computer program, so that the electronic device executes: storing the time of receiving the video frame in the first queue; storing the time when the second electronic device receives the Mth video frame in the first queue.

With reference to the second aspect, in a possible implementation manner of the second aspect, the processor is configured to call the computer program, so that the electronic device performs recording of N frames received after receiving the video frame Before the decoding delay of the video frame and the display delay, it can also be used to call the computer program, so that the electronic device performs: remove the first written element in the first queue, and write the time of receiving the video frame first queue.

In a third aspect, the present application provides a computer storage medium, including an instruction, which, when the instruction is run on an electronic device, causes the electronic device to execute any possible implementation manner in the first aspect above.

In a fourth aspect, an embodiment of the present application provides a chip, the chip is applied to an electronic device, and the chip includes one or more processors, and the processor is used to invoke computer instructions so that the electronic device executes any one of the above first aspects a possible implementation.

In a fifth aspect, an embodiment of the present application provides a computer program product including instructions, which, when the computer program product is run on a device, cause the electronic device to execute any possible implementation manner in the first aspect above.

It can be understood that the electronic device provided by the second aspect, the computer storage medium provided by the third aspect, the chip provided by the fourth aspect, and the computer program product provided by the fifth aspect are all used to execute the method provided by the embodiment of the present application. Therefore, the beneficial effects that it can achieve can refer to the beneficial effects in the corresponding method, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of a mirror projection screen provided by an embodiment of the present application;

FIG. 2 is a schematic flow diagram of a mirror projection screen provided by an embodiment of the present application;

FIG. 3A is a schematic diagram of decoding provided by an embodiment of the present application when the network condition is good;

FIG. 3B is a schematic diagram of decoding when the network condition is not good provided by the embodiment of the present application;

FIG. 3C is a schematic diagram of decoding after frame loss provided by the embodiment of the present application;

FIG. 4 is a flow chart of a method for dynamically adjusting the frame loss threshold provided by an embodiment of the present application;

FIG. 5 is a flowchart of another method for dynamically adjusting the frame loss threshold provided by the embodiment of the present application;

FIG. 6 is a flow chart for adjusting the frame loss threshold provided by the embodiment of the present application;

FIG. 7A is a schematic diagram of adjusting the frame loss threshold provided by the embodiment of the present application;

FIG. 7B is another schematic diagram of adjusting the frame loss threshold provided by the embodiment of the present application;

FIG. 7C is another schematic diagram of adjusting the frame loss threshold provided by the embodiment of the present application;

FIG. 8 is a schematic flow diagram of an audio-video synchronization provided by an embodiment of the present application;

FIG. 9 is a flowchart of another method for dynamically adjusting the frame loss threshold provided by the embodiment of the present application;

FIG. 10 is a schematic diagram of a hardware structure of an electronic device 100 provided by an embodiment of the present invention;

FIG. 11 is a schematic diagram of a software structure of an electronic device 100 provided by an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

First, some terms and related technologies involved in this application are explained, so as to facilitate the understanding of those skilled in the art.

Application (Application, APP), generally refers to mobile phone software, mainly refers to the software installed on the smart phone, to improve the shortcomings of the original system and personalization. It is the main means to make the mobile phone improve its functions and provide users with a richer experience. The operation of mobile phone software requires a corresponding mobile phone system. As of June 1, 2017, the main mobile phone systems: Apple's iOS, Google's Android (Android) system, Symbian platform and Microsoft platform.

The Window class represents the top-level window of a view, which manages the top-level View in this view, and provides some standard UI processing strategies such as background, title bar, and default buttons. At the same time, it has the only Surface for drawing its own content. When the app creates a Window through WindowManager, WindowManager will create a Surface for each Window and pass the Surface to the app so that the application can draw content on it. In simple terms, it can be considered that there was a one-to-one relationship between Window and Surface.

Window Management Service (WindowManagerService, WMS) is a system-level service, started by SystemService, and implements the IWindowManager.AIDL interface.

The WindowManager controls window objects, which are containers for view objects. Window objects are always backed by Surface objects. The WindowManager oversees the lifecycle, input and focus events, screen orientation, transitions, animations, position, transformations, Z-order, and many other aspects of the window. WindowManager will send all window metadata to SurfaceFlinger so that SurfaceFlinger can use this data to compose a Surface on the screen.

A Surface typically represents the producer side of a buffer queue consumed by SurfaceFlinger. When rendered onto a Surface, the resulting result goes into the associated buffer, which is passed to the consumer. To put it simply, Surface can be regarded as a layer of view, images are drawn on Surface, and the BufferQueue in it produces view data, which is then handed over to its consumer SurfaceFlinger to be synthesized with other view layers, and finally displayed on the screen.

SurfaceFlinger is a service of Android, which is responsible for managing the Surface on the application side and compounding all Surfaces. SurfaceFlinger is a layer between the graphics library and the application. Its role is to accept graphics display data from multiple sources, synthesize them, and then send them to the display device. For example, when you open an application, there are usually three layers of display, the top status bar, the bottom or side navigation bar, and the application interface. Each layer is updated and rendered separately. These interfaces are synthesized by SurfaceFlinger and refreshed to the hardware display. Bufferqueue is used in the display process, and SurfaceFlinger is used as the consumer. For example, the Surface managed by WindowManager is used as the producer to generate pages, which are then synthesized by SurfaceFlinger.

(Hardware Composer, HWC) is used to determine the most efficient way to compose buffers with the available hardware. As a HAL, its implementation is device-specific and is usually done by the display device hardware original equipment manufacturer (OEM).

The display subsystem (Display Sub System, DSS) is a piece of hardware dedicated to image synthesis, and the image of the main screen of the mobile phone is synthesized using DSS.

VirtualDisplay refers to virtual display, which is one of the multiple screens supported by Android (screens supported by Android include: main display, external display, and virtual display). There are many usage scenarios of VirtualDisplay, such as screen recording, WFD display, etc. Its function is to grab the content displayed on the screen. VirtualDisplay captures screen content, and there are many ways to implement it. ImageReader is provided in the API to read the content in VirtualDisplay.

MediaCodec can be used to obtain the underlying multimedia encoding of Android, which can be used for encoding and decoding. It is an important part of the Android low-level multimedia infrastructure framework. The role of MediaCodec is to process input data to generate output data. First generate an input data buffer, fill the data into the buffer and provide it to the codec. The codec will process the input data in an asynchronous manner, and then provide the filled output buffer to the consumer. After the consumer consumes the buffer Return to codec.

Transmission Control Protocol (Transmission Control Protocol, TCP) is a connection-oriented, reliable, byte stream-based transport layer communication protocol, defined by RFC 793 of IETF. TCP was designed to accommodate a layered protocol hierarchy supporting multiple network applications. TCP is used to provide reliable communication services between pairs of processes in host computers connected to different but interconnected computer communication networks. TCP assumes that it can obtain simple, possibly unreliable, datagram service from lower-level protocols. In principle, TCP should be able to operate over a variety of communication systems from hardwired connections to packet-switched or circuit-switched networks.

Internet Protocol (Internet Protocol, IP) is the abbreviation of the network layer protocol in the TCP/IP system. The purpose of designing IP is to improve the scalability of the network: one is to solve Internet problems and realize the interconnection and intercommunication of large-scale and heterogeneous networks; Independent development. According to the end-to-end design principle, IP only provides a connectionless, unreliable, best-effort data packet transmission service for the host.

Packet (Packet) is the data unit in the communication transmission of TCP/IP protocol, generally also called "data packet".

Real-time Transport Protocol (RTP/RTTP) is a network transmission protocol. The RTP protocol specifies a standard packet format for delivering audio and video over the Internet. It was originally designed as a multicast protocol, but has since been used in many unicast applications. The RTP protocol is often used in streaming media systems (with RTCP protocol), video conferencing and push to talk (Push to Talk) systems (with H.323 or SIP), making it the technical basis of the IP telephony industry. The RTP protocol is used with the RTP control protocol RTCP, and it is built on the user datagram protocol.

RTP itself does not provide a timely delivery mechanism or other Quality of Service (QoS) guarantees, it relies on low-level services to achieve this process. RTP does not guarantee delivery or prevent out-of-order delivery, nor does it determine the reliability of the underlying network. RTP implements orderly transmission. The serial number in RTP allows the receiver to reassemble the packet sequence of the sender. At the same time, the serial number can also be used to determine the appropriate packet location. For example, in video decoding, sequential decoding is not required.

The RTP standard defines two sub-protocols, RTP and RTCP.

The data transfer protocol RTP is used to transfer data in real time. Information provided by the protocol includes: timestamp (for synchronization), sequence number (for packet loss and reordering detection), and payload format (for specifying the encoding format of the data).

The control protocol RTCP is used for QoS feedback and synchronous media flow, that is, RTCP is used to monitor the quality of service and transmit relevant information about ongoing session participants. Compared with RTP, the bandwidth occupied by RTCP is very small, usually only 5%. The functionality of the second aspect of RTCP is sufficient for "loosely controlled" sessions, that is, it does not have to be used to support all of an application's control communication requests without explicit membership control and organization.

In the transmission of information between TCP nodes, the content of each transmission is a structure, so each time it is transmitted, the data in the structure must be packaged, and then when one end receives the data, the received buf The data in the parameter is unpacked.

Frames Per Second (FPS) is the definition in the image field, which refers to the number of frames that the chip can or actually renders per second. Generally speaking, it refers to the number of visually animated or video frames. FPS is a measure of the amount of information used to save and display dynamic video. The more frames per second, the smoother the displayed motion will be. Generally, 30 is the minimum you need to avoid jerky motion. Some computer video formats can only provide 15 frames per second.

PTS (Presentation Time Stamp): Displays the timestamp, which is used to tell the player when to display the data of this frame.

Vertical synchronization (Vertical synchronization, VSYNC) is also known as field synchronization. From the display principle of CRT (Cathode Ray Tube) display, a single pixel forms a horizontal scanning line, and the accumulation of horizontal scanning lines in the vertical direction forms a complete picture. The refresh rate of the display is controlled by the graphics card DAC (digital-to-analog converter, also known as the D/A converter), and the graphics card DAC will generate a vertical synchronization signal after scanning one frame. What we usually say to turn on vertical synchronization refers to sending the signal to the 3D graphics processing part of the graphics card, so that the graphics card is restricted by the vertical synchronization signal when generating 3D graphics.

In daily life, mirroring screen projection brings great convenience to users' work and life. on the screen so that other participants can watch it, and there is no need to perform corresponding operations on the large screen, which greatly improves the user experience.

FIG. 1 is a schematic diagram of a mirroring screen projection provided by an embodiment of the present application. The mirroring screen projection process specifically projects content displayed on a first electronic device to a second electronic device for display. On the setting interface of the first electronic device, you can find the "Wireless Screen Projection" option. Turn on "Wireless Screen Projection". After the first electronic device establishes a communication connection with other devices (for example, the second electronic device), you can perform mirror projection. Screen. As shown in Figure 1, the first electronic device is a mobile phone, and the second electronic device is a smart screen. After a communication connection is established between the mobile phone and the smart screen, the mirror image of the video played on the mobile phone can be displayed on the smart screen. It can be understood that if the interface of the mobile phone (the picture displayed on the display screen of the mobile phone) changes due to some user operations in the subsequent process, the interface on the smart screen will also change accordingly.

It can be understood that the first electronic device may be one of electronic devices such as a mobile phone, a tablet computer, and a PC, and the second electronic device may be one of electronic devices such as a tablet computer, a PC, and a smart screen.

In addition, a communication connection between the first electronic device and the second electronic device may be established in various ways. Optionally, wireless communication technology can be used to establish a communication connection between the first electronic device and the second electronic device, for example, connecting the first electronic device and the second electronic device through a wireless fidelity (Wireless Fidelity, Wi-Fi) network It is also possible to use wired communication technology to establish a communication connection between the first electronic device and the second electronic device, for example, to use media such as coaxial cables, twisted pairs, and optical fibers to connect the first electronic device and the second electronic device.

The specific process of screen mirroring is introduced below according to FIG. 2 .

FIG. 2 is a schematic flow chart of a mirroring screen projection provided by an embodiment of the present application. As shown in FIG. 1 , the mirroring screen projection process is to mirror the content displayed on the first electronic device to the second electronic device.

First, on the side of the first electronic device, two operations need to be performed:

One is the display screen.

Specifically, the APP on the first electronic device creates Window through WMS, and creates a Surface for each Window, and passes the corresponding Surface to the application so that the application can draw graphics data on the Surface, that is, Layers are presented via WMS. WMS provides SurfaceFlinger with buffer and window metadata (drawn Surface), and SurfaceFlinger can use this information to synthesize Surface, obtain the synthesized image, and then display the synthesized image to the second through its own display system (HWC/DSS). On the screen of an electronic device, that is, the image on the first electronic device that the user can see.

The second is to transmit the screen to be projected to the second electronic device.

Specifically, after SurfaceFlinger synthesizes images, it needs to capture the screen content through VirtualDisplay for virtual display. It is understandable that the captured screen content can be audio and video data, such as H.264 data, H.265 data, VP9 data, AV1 data , AAC data, etc. Then encode the captured audio and video data through an encoder (for example, MediaCodec, etc.), then perform encryption (Encryption) and multi-layer packaging, for example, perform RTP packaging and VTP/TCP packaging, and finally pack the obtained data packets sent to the second electronic device.

It can be understood that the data packet may be sent to the second electronic device through wireless communication (for example, WiFi), and the data packet may also be sent to the second electronic device through wired communication.

It should be noted that the data packet sent by the first electronic device to the second electronic device includes audio data or video data, that is, the audio data and video data are transmitted independently, wherein the audio data may include audio frames, and the video data may include video frame. Therefore, the process of the first electronic device sending the data packet to the second electronic device is the process of the first electronic device sending video frames and audio frames to the second electronic device.

Secondly, related operations also need to be performed on the side of the second electronic device in order to successfully cast the screen.

Specifically, after the second electronic device receives the data packet sent by the first electronic device, it performs corresponding unpacking (RTP unpacking and VTP/TCP unpacking) operations, decryption operations, and decoding (MediaCodec decoding) operations, and then the obtained The audio data and video data are synchronized, and then sent to display by MediaCodec, and finally the layers are synthesized by SurfaceFlinger and displayed on the screen of the second electronic device.

It should be noted that, correspondingly, the process of the second electronic device receiving the data packet sent by the first electronic device is that the second electronic device receives the video frame and the audio frame sent by the first electronic device.

It can be understood that the transmission of data packets is an important part of screen mirroring. Whether the first electronic device can send the data packets to the second electronic device smoothly and in time directly affects the screen projection effect. Since the transmission of audio data is relatively stable (it can also be transmitted relatively stably in the case of poor network conditions), this application mainly considers the transmission of video data, that is, the transmission of video frames.

If the network condition is not good, for example, the network fluctuates or the network is weak for a short time, etc., the second electronic device may not be able to receive the video frame sent by the first electronic device in time. A lot of video frames will be received, because the decoder is serial decoding, and the time required for decoding is longer than the time required for unpacking and decrypting, so the video frames received later can only be decoded after the decoding of the previously received video frames is completed. Video frame blocking issues may occur.

The difference in processing video frames by the second electronic device under different network environments is introduced below (FIG. 3A-FIG. 3C). As shown in Figure 3A, when the network condition is good, the second electronic device receives video frames A, B, C, and D at a stable rate (the time difference of receiving video frames is relatively stable), and also sequentially receives video frames A at a stable rate. , B, C, and D are decoded, and the decoding time is equal to the actual decoding time. Wherein, the decoding time refers to the time required for the second electronic device to actually start decoding the video frame to complete the decoding; the actual decoding time refers to the time from the second electronic device receiving the video frame to the completion of decoding. For example, if the second electronic device takes 30 milliseconds (ms) from receiving the video frame to completing the decoding of the video frame, and the time required from actually starting to decode the video frame to the completion of decoding is 10 ms, then the decoding The time is 10ms, and the actual decoding time is 30ms. After the decoding is completed, the second electronic device can synchronize the decoded video frames A, B, C, and D with the audio frames at a stable rate, and send the synchronized video frames to display, that is, to transmit the video frames to the display screen to display. Generally, after sending to display, you can check the display status of the video frame by sending to display callback. As shown in FIG. 3A , when the network condition is good, video frames can be displayed on the display screen of the second electronic device at a stable rate.

When the network condition is poor, the video frame cannot be transmitted to the second electronic device in time, and when the network is restored to stability, the second electronic device may receive multiple video frames in a short period of time, so that it cannot be decoded in time, that is to say , when the decoding of the current video frame has not been completed, the next video frame has been received by the second electronic device, and the video frame received later can only start decoding after the previous video frame has been decoded.

Exemplarily, as shown in FIG. 3B , the second electronic device receives video frames A, B, and C within a short period of time, and since video frame A is received first, video frame A is decoded first. However, in the process of decoding video frame A, the second electronic device receives video frame B, and video frame B should be decoded immediately, but since the decoding of video frame A has not been completed, video frame B only needs to be decoded. I can wait for a while. After video frame A is decoded, video frame B is decoded. Similarly, video frame C also needs to wait for video frame B to be decoded before decoding.

In this case, the decoding time of video frame A, B, C is the same, but the actual decoding time is different. Since video A is the first frame received by the second electronic device, it can be decoded without waiting. Therefore, the decoding time of video frame A is the same as the actual decoding time. However, video frame B needs to wait for video frame A to be decoded before starting to decode, so the actual decoding time of video frame B is longer than the actual decoding time of video frame A. Similarly, the actual decoding time of the video frame C also includes the waiting time for decoding. In other words, the actual decoding time refers to the time required by the electronic device from receiving a video frame to completing decoding.

It can be understood that when the network condition is not good, the display delay of the video frame will be increased (as shown in FIG. 3B ), so that the video frame cannot be displayed on the second electronic device in time.

It can be understood that the process of waiting for decoding will increase the decoding delay, thus causing the delayed display of the video frame. Overall, the delay of mirroring screen projection will increase. That is to say, the picture displayed on the first electronic device may not be synchronized with the picture displayed on the second electronic device. For example, when the 5th second (s) of the video is displayed on the first electronic device, the 3rd second (s) of the video is displayed on the second electronic device. From the user's point of view, the above phenomena greatly affect the user experience.

In order to solve the above problem, a frame loss threshold can be set, and by judging the relationship between the time difference between the second electronic device receiving video frames and the frame loss threshold, frames are selectively dropped, thereby reducing the waiting time for video frames to be decoded. For example, as shown in FIG. 3C , when the time difference between receiving video frame A and video frame B is less than a predetermined threshold, video frame B may be selected to be discarded. In this case, the video frames can also be displayed on the display screen of the second electronic device in time.

The above method can improve the situation of video frame blocking, and there is room for further optimization. On the one hand, since there is no feedback mechanism established, it is impossible to determine whether the set frame loss threshold can effectively solve the problem of video frame blocking, that is, it is impossible to quantify the benefits of decoding delay and display delay caused by frame loss; on the other hand, The preset frame loss threshold may not be able to effectively solve the video frame blocking problem under different network conditions and different device states.

Based on the above content, the present application provides a method and related equipment for dynamically adjusting the frame loss threshold. The initial frame loss threshold can be set, and the frame loss threshold can be dynamically adjusted according to the time difference when the second electronic device receives video frames, that is, according to the received video frame The time difference judges the current network status and adjusts the frame loss threshold accordingly, and then performs the frame loss operation, which can effectively solve the video frame blocking problem under different network conditions. In addition, after the frame loss is completed, judge whether the decoding delay and the display delay after adjusting the frame loss threshold are reduced, and determine whether the frame loss threshold needs to be adjusted again according to the judgment result, so that the effect of the frame loss threshold can be fed back in time , to ensure that the current frame loss threshold can effectively solve the problem of video frame blocking.

It can be understood that before performing the mirror projection as shown in FIG. 1 , the user needs to trigger the mirror projection.

Exemplarily, the user triggers a setting application program control on the first electronic device, and in response to the user operation, the first electronic device displays a setting interface, and the setting interface includes a wireless screen projection control. The first electronic device may detect a user operation acting on the wireless screen projection control, and in response to the user operation, the first electronic device may display a wireless screen projection interface. The wireless screen projection interface includes one or more controls, and these controls are used to represent devices that can perform mirror projection with the first electronic device. The first electronic device may detect a user operation acting on the first control, and in response to the user operation, the first electronic device may perform screen mirroring with the second electronic device. The first electronic device not only displays images on the display screen of the device, but also sends video frames to the second electronic device, so that the images displayed on the first electronic device can also be displayed on the second electronic device.

It should be noted that the second electronic device needs to undergo a series of processing before it can display the video frame sent by the first electronic device on the display screen. For the processing procedure of the second electronic device, reference may be made to the following embodiments.

Fig. 4 exemplarily shows a flow chart of a method for dynamically adjusting the frame loss threshold provided by the embodiment of the present application.

S401: Receive a video frame.

The second electronic device receives the video frame sent by the first electronic device, and marks the received video frame as A. In an embodiment of the present application, the TCP/VTP module in the second electronic device receives the video frame sent by the first electronic device.

It can be understood that the method of transmitting video frames between the first electronic device and the second electronic device includes but is not limited to sending through wireless communication methods such as wireless local area networks (Wireless Local Area Networks, WLAN), for example, through wireless fidelity (Wireless Fidelity, Wi-Fi) network transmission, and transmission through wired communication, for example, transmission using media such as coaxial cables, twisted pairs, and optical fibers.

S402: Determine whether a frame drop operation needs to be performed.

The second electronic device determines the difference between the time of the received video frame (A) and the reception time of the previously received video frame. When the difference is smaller than the frame loss threshold, a frame loss operation is performed, that is, video frame A is discarded. When the difference is not less than the frame loss threshold, it is judged whether to adjust the frame loss threshold.

It can be understood that the process of judging whether to perform the frame loss operation and the process of adjusting the frame loss threshold will be specifically described in subsequent embodiments, and will not be described here.

S403: decoding and sending to display.

If the received video frame is not discarded, the video frame is transmitted to the decoder for decoding, and then sent for display after the decoding is completed, so that the video frame is displayed on the display screen of the second electronic device.

Fig. 5 exemplarily shows a flowchart of another method for dynamically adjusting the frame loss threshold provided by the embodiment of the present application.

S501: Receive a video frame.

Specifically, the second electronic device receives the video frame sent by the first electronic device. In an embodiment of the present application, the TCP/VTP module in the second electronic device receives the video frame sent by the first electronic device.

S502: Record the time of receiving the video frame.

Specifically, the second electronic device may record the time when the video frame arrives, that is, the time when the second electronic device receives the video frame. In an embodiment of the present application, the second electronic device may also set a number for the received video frame, and record both the number and the arrival time of the video frame. For example, the time when the second electronic device receives the Nth video frame is T, and the second electronic device can find out that T is the arrival time of the video frame corresponding to the number N according to the number N.

It can be understood that the arrival time of the video frame may be expressed in numbers or other forms, and the arrival time of the video frame mentioned here is not necessarily the real time when the second electronic device receives the video frame.

Exemplarily, the second electronic device records the time at which a certain video frame is received as 20210511235643.

Exemplarily, the time at which the second electronic device receives the first video frame may be set as the 1 ms, and the time at which subsequent video frames are received is calculated based on the time at which the first frame is received.

In an embodiment of the present application, the second electronic device may also set a queue to store the arrival time of the video frame and/or the number of the video frame. That is, the queue element represents the time when the second electronic device receives the video frame. Understandably, this queue may be recorded as the first queue.

It can be understood that the set queue can be a first-in-first-out linear table, which only allows insertion at one end of the table and deletion of elements at the other end. The queue element refers to the data element in the queue or refers to the data element using the queue data structure to perform related operations. The data type of the queue data element can be an existing data type or a user-defined data type. For example, if the time when the second electronic device receives the first video frame is set as 1 ms, the corresponding queue element may be 1.

In addition, you can determine the length of the queue when setting the queue, that is, determine the maximum number of elements that the queue can accommodate, and when the number of elements in the queue reaches the upper limit that the queue can accommodate (the queue is full), you need to add the maximum number of elements in the queue. The elements written first are removed before new elements can be written into the queue.

In an embodiment of the present application, the length of the queue is 3, that is, the number of elements that can be accommodated in the queue is at most 3, that is, the queue can contain at most the receiving time of three video frames.

S503: Initialize the frame loss threshold.

Specifically, the second electronic device initializes a frame loss threshold, and the frame loss threshold is used to determine whether a video frame is discarded.

In an embodiment of the present application, the initial frame loss threshold may be set as mThreshold. Exemplarily, mThreshold=1*VsyncDuration, VsyncDuration refers to the time interval between transmitting two adjacent video frames at a preset frame rate. For example, when the preset frame rate is 60FPS, the interval between transmitting two adjacent video frames is 1000/60=16.67ms.

It can be understood that the preset frame rate refers to the rate at which the first electronic device sends video frames to the second electronic device, that is, the number of video frames that the first electronic device sends to the second electronic device per second. It is determined when the second electronic device establishes a communication connection.

In addition, an adjustment range of the frame loss threshold may be set, for example, the adjustment range of the frame loss threshold is set as: 1*VsyncDuration≤mThreshold≤2*VsyncDuration.

S504: Determine whether the queue is full.

Specifically, if the second electronic device sets up a queue to record the time when the second electronic device receives video frames (such as step S502), the second electronic device also needs to determine whether the queue is full, that is, determine the number of elements contained in the queue Whether the upper limit is reached. If the queue is full, continue to execute step S505, and if the queue is not full, write the receiving time of the recorded video frame into the queue (step S507).

In other words, the data in the queue is used to judge and adjust the frame loss threshold. In some embodiments, the frame loss threshold is adjusted only when the queue is full. In some other embodiments, the frame loss threshold may also be adjusted according to the elapsed time when the queue is not full.

S505: Determine whether the average frame rate is greater than the minimum frame rate.

Specifically, the average frame rate is compared with the minimum frame rate. If the average frame rate is less than the minimum frame rate, it means that the original frame rate is small or there are too many frames lost. In order to avoid affecting the continuity of the projected screen, the received video frames are not discarded, and the received video frames are directly sent to the decoder. The device waits for decoding, and removes the first written element in the queue, and then writes the receiving time of the video frame into the queue (step S507); if the average frame rate is not less than the minimum frame rate, calculate the time and the time of receiving the video frame The second electronic device receives the time difference of the Mth video frame, and denote the difference as a.

In some embodiments of the present application, the time of the Mth video frame may be the first written element in the first queue.

Generally speaking, the frame rate refers to the number of frames of pictures refreshed per second, which can be understood here as: when the video frame of the first electronic device is mirrored and projected to the second electronic device, the frame rate of the second electronic device The number of video frames refreshed on the screen per second, therefore, the frame rate refers to the number of video frames displayed on the screen of the second electronic device per second.

The average frame rate mentioned above refers to the average frame rate within a period of time, which is not necessarily 1s. For example, the average frame rate may be an average rate at which the second electronic device receives video frames sent by the first electronic device within 10s.

In an embodiment of the present application, the average frame rate may be from the second electronic device receiving the first video frame sent by the first electronic device to the second electronic device receiving the latest video frame sent by the first electronic device Average frame rate so far. In yet another embodiment of the present application, the average frame rate may also be an average frame rate at which the second electronic device receives N frames of video frames recently sent by the first electronic device.

It should be noted that if the second electronic device discards the video frames sent by the first electronic device, the discarded video frames may not be calculated during the calculation of the average frame rate.

In addition, the minimum frame rate is set to ensure the continuity of the screen projection, because when the frame rate is too low, the screen projection is not smooth, which greatly affects the user experience, so the minimum frame rate can be preset according to actual needs. Judge the relationship between the average frame rate and the minimum frame rate before frame loss, so as to avoid the problem of low frame rate that may be caused by continuous frame loss.

It should be noted that, in order to ensure the continuity of the screen projected to the second electronic device, the number of consecutive dropped frames should not be too much. After step S505 is performed, it is also possible to determine whether the frame dropping operation can be performed according to the number of consecutive dropped frames , if the number of consecutive lost frames exceeds the preset number of lost frames, the frame loss operation cannot be performed, and the frame loss threshold is not adjusted. At this time, the received video frames are directly sent to the decoder for decoding, and the continuous lost frames The number is cleared, the first written element in the queue is removed, and then the receiving time of the video frame is written into the queue (step S507).

Exemplarily, the preset number of dropped frames may be 2, that is, when the number of consecutive dropped frames exceeds 2, the second electronic device cannot drop the received video frame, but directly transmits the video frame to the decoder for waiting Decode, and clear the number of consecutive lost frames.

Optionally, you can also set a parameter to indicate the feasibility of the frame loss operation, for example, set the parameter Adjust to indicate the feasibility of the frame loss operation, that is, judge whether the frame loss operation can be performed and adjust the frame loss threshold according to the value of Adjust (Step S506), specifically, after step S505 is executed, check the value of Adjust, and judge whether the frame loss operation can be performed according to the value of Adjust. When Adjust=0, it means that the frame loss threshold cannot be adjusted at this time, and the frame loss cannot be performed. Frame operation, the video frame is directly sent to the decoder to wait for decoding, and the element written first in the queue is removed, and then the receiving time of the video frame is written into the queue (step S507); when Adjust=1, you can Execute the subsequent steps (such as step S506), and whether to perform the frame dropping operation needs to be specifically judged in the subsequent steps (step S506). For another example, it is set that when Adjust=false, the frame loss threshold cannot be adjusted, and the frame loss operation cannot be performed; when Adjust=true, subsequent steps can be performed (such as step S506). It can be understood that there may be other judgment methods, which are not limited in the present application.

S506: Adjust the frame loss threshold.

Specifically, judge the size relationship between a and the frame loss threshold mThreshold, and adjust the frame loss threshold according to the size relationship, there are two situations as follows:

1. a<mThreshold.

In this case, the second electronic device receives too many video frames in a short period of time, directly discards the received video frames, then counts the number of consecutive dropped frames, and calculates the average frame rate. It can be understood that when calculating the average frame rate here, the video frames received but discarded by the second electronic device may not be included in the calculation. If the number of consecutive lost frames exceeds the preset number of lost frames, the frame loss operation is not performed on the next received video frame, and the frame loss threshold is not adjusted, but the video frame is directly sent to the decoder for decoding.

In addition, when setting a parameter to indicate the feasibility of the frame loss operation, the value of the parameter can be updated after determining a<mThreshold and counting the number of consecutive frame loss. For example, when setting the parameter Adjust to indicate the feasibility of the frame loss operation , after determining a<mThreshold and counting the number of consecutive frame loss, if the number of consecutive frame loss exceeds the frame loss threshold, the value of Adjust is updated to zero, that is, Adjust=0, indicating that the second electronic device cannot discard the next received A video frame.

2. a≥mThreshold.

In conjunction with Figure 6, the adjustment of the frame loss threshold in the case of a≥mThreshod is specifically explained:

S601: Determine whether the second electronic device is performing a threshold test.

First, a brief description of the threshold test is given. The threshold test refers to: after threshold adjustment, the second electronic device judges whether the adjustment is valid according to the decoding delay and display delay of the next received N video frames, and when the adjustment is valid, maintain the adjusted The frame loss threshold, otherwise, adjust the threshold again.

The ongoing threshold test indicates that it is currently impossible to know whether the last threshold adjustment is effective, so the threshold will not be adjusted again at this time to avoid interfering with the judgment of the effect of the last threshold adjustment.

Therefore, if the second electronic device is performing a threshold test, it directly transmits the received video frame to the decoder to wait for decoding, and removes the first written element in the queue, and then writes the receiving time of the video frame into the queue (such as Step S507); if the second electronic device is not performing a threshold test, continue to execute step S602.

S602: Determine whether the frame loss threshold is not adjusted within a preset time.

If the frame loss threshold is not adjusted within the preset time, it may be because the frame loss threshold is too large, so a always satisfies a<mThreshold (case 1) within the preset time, and the threshold can be considered to be reduced. If the frame loss threshold is adjusted within the preset time, continue to execute step S603.

It should be noted that the preset time can be set according to actual needs, and in an embodiment of the present application, the preset time can be set to 1.5s.

S603: Determine whether the frame loss threshold is smaller than the upper threshold.

As shown in step S503, an adjustment range can be set for the frame loss threshold, that is, an upper limit and/or a lower limit for adjusting the frame loss threshold can be set. In one embodiment of the present application, the lower limit of the frame loss threshold is set to 1*VsyncDuration , the upper threshold is 2*VsyncDuration.

If the frame loss threshold is not less than the upper limit of the threshold, it means that the frame loss threshold has reached the adjustable upper limit (because the adjustment of the frame loss threshold cannot exceed the adjustment range), and the frame loss threshold can be considered to be reduced. If the frame loss threshold is less than the upper threshold, it may be considered to continue to increase the frame loss range to alleviate video frame congestion, that is, increase the frame loss threshold (as shown in step S604).

It should be noted that after the frame loss threshold is adjusted (increased or decreased by a certain step size), it can be understood that the threshold test has started.

S604: Increase the frame loss threshold.

A specific way to adjust the frame loss threshold may be: increase the frame loss threshold with a certain step size, and denote the step size as b, then the adjusted frame loss threshold value is: mThreshold=mThreshold+b.

It can be understood that the value of b can be set according to the actual situation. For example, b is set to 0.1*VsyncDuration. At this time, the frame loss threshold is: mThreshold=mThreshold-0.1*VsyncDuration. It should be noted that if the adjustment range of the frame loss threshold has been set (such as step S503), then the frame loss threshold can only be adjusted within this range.

S605: Decrease the frame loss threshold.

A specific way of adjusting the frame loss threshold may be: reduce the frame loss threshold with a certain step size, and denote the step size as b, then the adjusted frame loss threshold value is: mThreshold=mThreshold-b.

Exemplarily, the adjustment of the frame loss threshold is described with reference to FIG. 7A-FIG. 7C. In step S503, the frame loss threshold is initialized. The frame loss threshold after initialization is mThreshold=1*VsyncDuration. As shown in FIG. 7A, when a<mThreshold, the frame loss process is performed. When the method of adjusting the frame loss threshold is to increase the frame loss threshold with a certain step size (as shown in Figure 7B), if the step size is set to 0.1*VsyncDuration, the updated frame loss threshold is mThreshold=1.1*VsyncDuration, As shown in FIG. 7C , when a<mThreshold, frame dropping processing is performed, that is, when a<1.1*VsyncDuration, frame dropping processing is performed. It can be understood that, as shown in FIGS. 7A-7C , the adjustable upper limit of the frame loss threshold is 2*VsyncDuration, that is, mThreshold<2*VsyncDuration.

S507: Adjust the queue.

Specifically, if the second electronic device has set up a queue to record the time when the second electronic device receives the video frame (such as step S502), after performing step S506, the second electronic device can remove the first written element in the queue , write the time of receiving the video frame (the receiving time of the video frame recorded in step S502) into the queue.

Exemplarily, the queue is {1, 5, 10}, that is, the receiving times of the three video frames stored in the queue are the 1st, 5th, and 10th ms respectively, where 1 is the first element written into the queue, and 10 It is the last element written in the queue, remove the first element written in the queue - 1, write the time of receiving the video frame into the queue, if the time of receiving the video frame is the 16th ms, write 16 into the queue , the adjusted queue is {5, 10, 16}.

Exemplarily, the queue is {3, 4, 5}, that is, the queue stores the numbers of the 3rd, 4th, and 5th video frames received by the second electronic device. Through these numbers, the second electronic device can find out the receiving time when the second electronic device receives the 3rd, 4th, and 5th video frames.

Exemplarily, the queue is {20210511235643, 20210511235666, 20210511235733}, and the three strings of numbers stored in the queue indicate the time of the three video frames received by the second electronic device.

S508: decoding.

Specifically, the decoder in the second electronic device decodes the received video frame, and records the decoding delay. It can be understood that the decoding delay refers to a period of time from when a video frame is transmitted to a decoder to when decoding of the video frame is completed.

S509: audio and video synchronization processing and display.

Specifically, according to the frame loss situation in the previous step, the audio frame is adjusted accordingly, and it is judged whether to send the video frame for display.

As shown in Figure 8, the second electronic device can determine whether there is a frame dropping operation (step S801), if a video frame received before a video frame to be synchronized (received video frame) has been discarded, it also needs to be discarded Only the audio frames corresponding to the video frames (step S802) with the same number of discarded video frames can be obtained, otherwise the video frames and audio frames cannot be correspondingly displayed on the screen of the second electronic device The sound and picture are out of sync. If there is no frame dropping operation, or the same number of audio frames has been discarded (step S802), it is necessary to calculate the average PTS interval of all video frames sent for display, which is recorded as c, and then the sending time of the next video frame is estimated. Display time (step S803), estimated video frame PTS=actual PTS of the previous video frame+c. It can be understood that the actual PTS of the previous video frame refers to the actual display time of the previous video frame. If the estimated video frame PTS is different from the actual display time, when the difference between the two is less than the first preset threshold, the video frame is sent to display with the estimated video frame PTS, because the video frame is only displayed at the VSYNC time point The video frame can only be displayed after the frame is sent for display, so it is necessary to find the VSYNC time point closest to the time point of sending for display (estimated video frame PTS) (step S804). After finding the corresponding VSYNC time point, judge whether the difference between the VSYNC time point and the audio frame display time point is less than the second preset threshold (step S805), if the VSYNC time point and the audio frame display time point are between If the difference is not less than the second preset threshold, the video frame is not sent for display (step S806), otherwise, the video frame is sent for display (step S807), so that the video frame is finally displayed on the screen of the second electronic device.

It can be understood that both the first preset threshold and the second preset threshold can be set according to actual needs, which is not limited in the present application.

It should be noted that after the display sending is completed, the upper layer application of the second electronic device will receive the callback information of the completion of display sending from MediaCodec (as shown in Figure 2), update the decoding delay and display sending delay, and update the average frame Rate. It can be understood that the delay in sending and displaying refers to the time required from the decoding of the video frame to the actual display on the screen of the second electronic device.

S510: Test whether the frame loss threshold is valid.

If the frame loss threshold is adjusted in the above steps, the second electronic device needs to perform a threshold test to test whether the adjustment to the frame loss threshold is an effective adjustment. Specifically, the second electronic device monitors the decoding delay and display delay of the N video frames received subsequently. If the frame loss threshold is adjusted, the decoding delay and display delay of the N video frames received by the second electronic device , which are at least c% lower than the average decoding delay and average display delay in the current full time period, then it is judged that the adjustment to the frame loss threshold in the above steps is an effective adjustment, and the adjusted frame loss threshold is continued to be used; otherwise, It is judged that the adjustment effect of the frame loss threshold is not good, and the frame loss threshold mThreshold is reduced by a certain step size. For details, please refer to step S506, which will not be repeated here.

It can be understood that the current full time period refers to a period of time from when the second electronic device receives the first video frame to when the average decoding delay and the average display delay are calculated.

It should be noted that c can be adjusted according to actual needs. For example, c can be 10. Under this condition, if the frame loss threshold is adjusted, the decoding delay and display delay of the N video frames received by the second electronic device , which are respectively reduced by at least 10% compared with the average decoding delay and the average display delay of the current full time period, it is judged that the adjustment to the frame loss threshold is an effective adjustment.

FIG. 9 exemplarily shows a flowchart of another method for dynamically adjusting the frame loss threshold provided by the embodiment of the present application.

S901: Receive a video frame.

The second electronic device receives the video frame sent by the first electronic device. In an embodiment of the present application, the TCP/VTP module in the second electronic device receives the video frame sent by the first electronic device.

It can be understood that, for the specific implementation manner of step S901, reference may be made to step S501, which will not be repeated here.

S902: Record the time of receiving the video frame and initialize the frame loss threshold.

After receiving the video frame sent by the first electronic device, the second electronic device records the receiving time of the video frame, and initializes the frame loss threshold. It can be understood that, for the specific implementation manner of step S902, reference may be made to step S502 and step S503, which will not be repeated here.

S903: Determine whether the queue is full.

The second electronic device can set a queue for storing the receiving time of the video frame. Before writing the receiving time of the video frame into the queue, the second electronic device can judge whether the queue is full. The specific judgment process can refer to step S504, here No longer.

S904: Determine whether the average frame rate is greater than the minimum frame rate.

In order to display the video frames on the second electronic device at an appropriate frame rate and ensure the continuity of the images displayed on the second electronic device, the second electronic device may determine whether the average frame rate is greater than the minimum frame rate. It can be understood that, for a specific implementation manner of step S904, reference may be made to step S505, which will not be repeated here.

S905: Calculate the difference between the arrival time of the current video frame and the first element in the queue, and denote the difference as a.

The second electronic device calculates the difference between the received time of the current video frame and the first written element in the queue, and marks the difference as a.

S906: Determine whether the frame loss threshold can be adjusted.

Specifically, the second electronic device may set a parameter to indicate the feasibility of adjusting the frame loss threshold. For example, the second electronic device can determine whether the frame loss threshold can be adjusted according to the value of Adjust. When Adjust=0, it means that the frame loss operation cannot be performed at this time, and the video frame is directly sent to the decoder to wait for decoding, and the element written first in the queue is removed, and then the receiving time of the video frame is written into the queue (such as step S915); when Adjust=1, subsequent steps can be performed (such as step S506), and whether to perform the frame dropping operation needs to be specifically determined in subsequent steps (such as step S907).

It can be understood that, for a specific implementation manner of step S906, reference may be made to step S505, which will not be repeated here.

S907: Determine whether a is smaller than the frame loss threshold.

The second electronic device may determine whether a is smaller than the frame loss threshold. If a is less than the frame loss threshold, directly drop the frame and count the number of consecutive frame loss (such as step S908); if a is not less than the frame loss threshold, determine whether a threshold test is in progress (such as step S909).

S908: Drop frames directly and count the number of consecutive dropped frames.

It can be understood that, for the specific implementation manner of step S908, reference may be made to step S506, which will not be repeated here.

S909: Determine whether a threshold value test is being performed.

The second electronic device can determine whether a threshold test is in progress. If the threshold test is being performed, step S914 is directly performed; if the threshold test is not being performed, step S910 is continued.

It can be understood that, for the specific implementation manner of step S909, reference may be made to step S601, which will not be repeated here.

S910: Determine whether the frame loss threshold is not adjusted within a preset time.

The second electronic device may determine whether the frame loss threshold is not adjusted within a preset time. If the second electronic device adjusts the frame loss threshold within the preset time, directly perform step S911; if the second electronic device does not adjust the frame loss threshold within the preset time, continue to perform step S913.

It can be understood that, for a specific implementation manner of step S910, reference may be made to step S602, which will not be repeated here.

S911: Determine whether the frame loss threshold is smaller than the upper threshold.

The second electronic device may determine whether the frame loss threshold is smaller than the upper threshold. If the frame loss threshold is less than the upper threshold, continue to execute step S912; if the frame loss threshold is not less than the upper threshold, execute step S913.

It can be understood that, for the specific implementation manner of step S911, reference may be made to step S603, which will not be repeated here.

S912: Increase the frame loss threshold by one step and start threshold testing.

It can be understood that, for a specific implementation manner of step S912, reference may be made to step S604, which will not be repeated here.

S913: Decrease the frame loss threshold by one step and start threshold testing.

It can be understood that, for the specific implementation manner of step S913, reference may be made to step S605, which will not be repeated here.

S914: Remove the first element of the queue.

It can be understood that, for the specific implementation manner of step S914, reference may be made to step S507, which will not be repeated here.

S915: Write the time of the received video frame into a queue.

It can be understood that, for the specific implementation manner of step S915, reference may be made to step S507, which will not be repeated here.

S916: Decode and count the decoding delay.

It can be understood that, for the specific implementation manner of step S916, reference may be made to step S508, which will not be repeated here.

S917: Audio and video synchronization and display.

It can be understood that, for a specific implementation manner of step S917, reference may be made to step S509, which will not be repeated here.

S918: Sending to display callback and updating decoding delay, display delay and average frame rate.

The second electronic device can check the display status of the video frame on the second electronic device by sending and displaying the callback. For details, refer to step S509, which will not be repeated here.

S919: Determine whether a threshold test is being performed.

It can be understood that, for the specific content of step S919-step S923, reference may be made to step S510, which will not be repeated here.

S920: Count the decoding delay and display sending delay of video frames.

It can be understood that, after adjusting the frame loss threshold, the second electronic device may detect the decoding delay and the display delay of subsequent received video frames.

S921: Determine whether 60 video frames are reached.

It can be understood that, after receiving 60 video frames, the second electronic device may determine whether the adjustment to the frame loss threshold is an effective adjustment. Therefore, the second electronic device needs to determine whether the received video frames reach 60 frames after the frame loss threshold is adjusted.

S922: Determine whether the adjustment to the frame loss threshold is an effective adjustment.

It can be understood that, for a specific determination method, reference may be made to step S510, which will not be repeated here.

S923: Decrease the frame loss threshold by one step.

It can be understood that, for the specific implementation manner of step S923, reference may be made to step S605, which will not be repeated here.

The devices involved in the embodiments of the present application are introduced below.

FIG. 10 is a schematic diagram of a hardware structure of an electronic device 100 provided in an embodiment of the present application.

It can be understood that the electronic device 100 may implement the methods for dynamically adjusting the frame loss threshold shown in FIG. 4 , FIG. 5 and FIG. 9 . It can be understood that the above-mentioned first electronic device and the second electronic device may be the electronic device 100 .

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (Universal Serial Bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber Identification Module (Subscriber Identification Module, SIM) card interface 195 and so on. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (Application Processor, AP), a modem processor, a graphics processor (Graphics Processing unit, GPU), an image signal processor (Image Signal Processor, ISP), controller, memory, video codec, digital signal processor (Digital Signal Processor, DSP), baseband processor, and/or neural network processor (Neural-network Processing Unit, NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

In an embodiment of the present application, the first electronic device may be the electronic device 100, and the specific process of displaying the screen by the first electronic device is: the processor 110 completes the synthesis of multiple screen layers (WMS layer display, and SurfaceFlinger layer synthesis), and then sent to display screens 1 to N194 for display (HWC/DSS display, and main screen display). In addition, the processor 110 of the first electronic device has also completed the encoding and packaging of video frames (VirtualDisplay virtual display, and VTP/TCP packets), and finally these packaged video frames will be sent to the second video frame through the wireless communication module 160. Electronic equipment.

In yet another embodiment of the present application, the second electronic device, that is, the device that receives the video frame, may be the electronic device 100 . The wireless communication module 160 of the second electronic device receives the video frame data sent by the first electronic device. The video frame data will be processed by the processor 110 through a series of reverse unpacking (VTP/TCP unpacking, and RTP unpacking) and decoding (MediaCodec decoding) to obtain video frame data that can actually be displayed. These video frame data will also be sent to display (MediaCodec for display) and layer synthesis (SurfaceFlinger layer synthesis), and finally sent to display screen 194 for display (main screen for display).

It can be understood that the processing flow of audio data is similar to that of video data (video frame), and will not be repeated here.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface can include an integrated circuit (Inter-Integrated Circuit, I2C) interface, an integrated circuit built-in audio (Inter-Integrated Circuit Sound, I2S) interface, a pulse code modulation (Pulse Code Modulation, PCM) interface, a universal asynchronous transmitter (Universal Asynchronous Receiver/Transmitter, UART) interface, mobile industry processor interface (Mobile Industry Processor Interface, MIPI), general-purpose input and output (General-Purpose Input/Output, GPIO) interface, subscriber identity module (Subscriber Identity Module, SIM) interface, and /or Universal Serial Bus (Universal Serial Bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (Serial Data Line, SDA) and a serial clock line (Serial Clock Line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example: the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interface includes camera serial interface (Camera Serial Interface, CSI), display serial interface (Display Serial Interface, DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices 100, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device 100 through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (Low Noise Amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide wireless local area network (Wireless Local Area Networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (Bluetooth, BT), global navigation satellite System (Global Navigation Satellite System, GNSS), frequency modulation (Frequency Modulation, FM), near field communication technology (Near Field Communication, NFC), infrared technology (Infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), broadband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include Global Positioning System (Global Positioning System, GPS), Global Navigation Satellite System (Global Navigation Satellite System, GLONASS), Beidou Navigation Satellite System (Beidou Navigation Satellite System, BDS), Quasi Zenith Satellite System (Quasi - Zenith Satellite System (QZSS) and/or Satellite Based Augmentation Systems (SBAS).

In an embodiment of the present application, the communication between the first electronic device and the second electronic device can be realized through the wireless communication module 160 . It can be understood that a point-to-point communication manner may be adopted between the first electronic device and the second electronic device, or communication may be performed through a server.

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. GPU is a microprocessor for image processing, connected to display screen 194 and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (Liquid Crystal Display, LCD), organic light-emitting diode (Organic Light-Emitting Diode, OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (Active-Matrix Organic Light Emitting Diode, AMOLED), flexible light-emitting diode (Flex Light-Emitting Diode, FLED), Mini LED, Micro LED, Micro-OLED, quantum dot light-emitting diode (Quantum Dot Light Emitting Diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The electronic device 100 may realize the acquisition function through an ISP, a camera 193 , a video codec, a GPU, a display screen 194 , and an application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image or video visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element can be a charge coupled device (Charge Coupled Device, CCD) or a complementary metal oxide semiconductor (Complementary Metal-Oxide-Semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image or video signal. ISP outputs digital image or video signal to DSP for processing. DSP converts digital images or video signals into standard RGB, YUV and other formats of images or video signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1. For example, in some embodiments, the electronic device 100 can use N cameras 193 to acquire images with multiple exposure coefficients, and then, in video post-processing, the electronic device 100 can synthesize HDR images using the HDR technology based on the images with multiple exposure coefficients. image.

Digital signal processors are used to process digital signals. In addition to digital image or video signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: Moving Picture Experts Group (Moving Picture Experts Group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

NPU is a neural network (Neural-Network, NN) computing processor. By referring to the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 . The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image and video playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS) and the like.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the receiver 170B can be placed close to the human ear to receive the voice.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be a USB interface 130, or a 35mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

The sensor module 180 may include one or more sensors, and these sensors may be of the same type or different types. It can be understood that the sensor module 180 shown in FIG. 1 is only an exemplary division method, and there may be other division methods. This application is not limited to this.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.

The gyro sensor 180B can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y and z axes) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a clamshell machine, the electronic device 100 can detect opening and closing of the clamshell according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device 100, and can be applied to applications such as horizontal and vertical screen switching, pedometers, etc.

The distance sensor 180F is used to measure the distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes.

The ambient light sensor 180L is used for sensing ambient light brightness.

The fingerprint sensor 180H is used to acquire fingerprints. The electronic device 100 can use the acquired fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature treatment strategy.

Touch sensor 180K, also known as "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

The bone conduction sensor 180M can acquire vibration signals.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects.

The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calling and data communication. In some embodiments, the electronic device 100 adopts an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

FIG. 11 is a schematic diagram of a software structure of an electronic device 100 provided by an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the system is divided into four layers, which are application program layer, application program framework layer, runtime (Runtime) and system library, and kernel layer from top to bottom.

The application layer can consist of a series of application packages.

As shown in FIG. 11 , the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications (also called applications).

In an embodiment of the present application, the application program package may also include another application program, and the user can complete the screen mirroring after triggering the application program by touch, click, gesture, voice, etc., during the screen mirroring process The electronic device 100 can be used as a device for sending video frames and audio frames (for example, a first electronic device), or as a device for receiving video frames and audio frames (for example, a second electronic device). It can be understood that the name of the application program may be "wireless projection", which is not limited in this application.

The application framework layer provides an application programming interface (Application Programming Interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 11, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager and so on.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 100 . For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog interface. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

Runtime (Runtime) includes the core library and virtual machine. Runtime is responsible for the scheduling and management of the system.

The core library includes two parts: one part is the function function that the programming language (for example, java language) needs to call, and the other part is the core library of the system.

The application layer and the application framework layer run in virtual machines. The virtual machine executes programming files (for example, java files) of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: Surface Manager (Surface Manager), Media Library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem, and provides fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, and a virtual card driver.

The workflow of the software and hardware of the electronic device 100 will be exemplarily described below in combination with the screen mirroring scene.

If the electronic device 100 is a device (for example, the first electronic device) that sends video frames and audio frames during screen mirroring, when the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, and other information). Raw input events are stored at the kernel level. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch-click operation, and the control corresponding to the click operation is the control of the wireless projection icon as an example, the wireless projection application calls the interface of the application framework layer, starts the wireless projection application, and then calls the kernel layer to Start the drive, so as to transmit the video frame and the audio frame to another device (the device receiving the video frame and the audio frame during the screen mirroring process, for example, the second electronic device) through the wireless communication module 160 .

It can be understood that the device to be screen-cast (for example, the second electronic device) may start the wireless screen-casting application by default, or start the wireless screen-casting application when receiving a screen mirroring request sent by other devices. When the first electronic device starts the wireless screen projection application, it can select the second electronic device that has already started the wireless screen projection application, so when the first electronic device is selected and establishes a communication connection with the second electronic device, mirroring can begin. Cast screen.

It should be noted that a communication connection between the first electronic device and the second electronic device may be established through the wireless communication technology provided by the wireless communication module 160 in FIG. 10 .

If the electronic device 100 is a device (for example, a second electronic device) that receives video frames and audio frames in mirror projection, correspondingly, start the wireless screen projection application, receive the video frames and audio frames through the wireless communication module 160, and call the kernel layer to start the display driver and audio driver, display the received video frame through the display screen 194, and play the received audio frame through the speaker 170A.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

It should be understood that the first, second, third, fourth and various numbers mentioned herein are only for convenience of description, and are not intended to limit the scope of the present application.

It should be understood that the term "and/or" in this article is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and/or B may mean: A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

It should also be understood that in various embodiments of the present application, the serial numbers of the above-mentioned processes do not mean the order of execution, and the order of execution of the processes should be determined by their functions and internal logic, and should not be implemented in this application. The implementation of the examples constitutes no limitation.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The steps in the methods of the embodiments of the present application can be adjusted, combined and deleted according to actual needs.

The modules in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions described in each embodiment are modified, or some of the technical features are replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the various embodiments of the application.

Claims

A method for dynamically adjusting the frame loss threshold, which is applied to a second electronic device, and the method includes:

receiving video frames sent by the first electronic device;

Determine the frame receiving time difference; the frame receiving time difference is the difference between the time of receiving the video frame and the time of receiving the Mth video frame; the Mth video frame is the first electronic device sending the the video frame sent before the video frame;

In the case that the frame receiving time difference is not less than the frame loss threshold, if the first duration does not reach the preset time, reduce the frame loss threshold; the first duration is the duration from the last time the frame loss threshold was adjusted to the current moment; The frame loss threshold is used to judge whether to discard the video frame;

Alternatively, when the frame receiving time difference is not less than the frame loss threshold, if the first duration reaches the preset time, and the frame loss threshold is less than the upper threshold, increase the frame loss threshold; if the first duration reaches the preset time , and the frame loss threshold is not less than the upper threshold, reduce the frame loss threshold; the upper threshold is the maximum value of the frame loss threshold;

Record the decoding delay and the display delay of the N frames of video frames received after receiving the video frame; the decoding delay is the time when the video frame arrives at the decoder and completes the decoding; the display delay is The time from when a video frame is decoded to when it is displayed on the display;

judging whether the N is equal to the preset number of frames;

If the N is equal to the preset number of frames, determine whether the adjusted frame loss threshold is valid; if the adjusted frame loss threshold is invalid, reduce the adjusted frame loss threshold, and stop the threshold test ; The threshold test is used to determine whether the adjustment to the frame loss threshold is effective.
The method of claim 1, wherein the video frame is a video frame not used for the threshold test.
The method according to claim 1 or 2, wherein the judging whether the adjusted frame loss threshold is valid comprises:

Determine the average decoding delay and the average display delay in the current full time period; the current full time period is from receiving the first video frame to determining the average decoding delay and the average display delay for a period of time;

If the decoding delay and the display delay of the N frames of video frames are respectively reduced by at least c% compared with the average decoding delay and the average display delay, it is determined that the adjusted frame loss threshold is valid.
The method according to any one of claims 1-3, further comprising: discarding the video frame when the frame receiving time difference is less than a frame loss threshold.
The method according to any one of claims 1-4, wherein after receiving the video frame sent by the first electronic device, the method further comprises: recording the time of receiving the video frame; The frame threshold is initialized.
The method according to claim 5, wherein after said recording the time of receiving said video frame, said method further comprises: storing the time of receiving said video frame in a first queue; said first The time at which the second electronic device receives the Mth video frame is stored in the queue.
The method according to claim 6, wherein the recording is before the decoding delay and the display delay of the N frames of video frames received after receiving the video frames, and the method further comprises: removing the first The element written first in a queue writes the time of receiving the video frame into the first queue.
A method for dynamically adjusting the frame loss threshold, which is applied to a second electronic device, and the method includes:

receiving video frames sent by the first electronic device;

Determine the frame receiving time difference; the frame receiving time difference is the difference between the time of receiving the video frame and the time of receiving the Mth video frame; the Mth video frame is the first electronic device sending the the video frame sent before the video frame;

In the case that the frame receiving time difference is not less than the frame loss threshold, if the first duration does not reach the preset time, reduce the frame loss threshold; the first duration is the duration from the last time the frame loss threshold was adjusted to the current moment; The frame loss threshold is used to judge whether to discard the video frame;

Alternatively, when the frame receiving time difference is not less than the frame loss threshold, if the first duration reaches the preset time, and the frame loss threshold is less than the upper threshold, increase the frame loss threshold; if the first duration reaches the preset time , and the frame loss threshold is not less than the upper threshold, reduce the frame loss threshold; the upper threshold is the maximum value of the frame loss threshold;

Record the decoding delay and the display delay of the N frames of video frames received after receiving the video frame; the decoding delay is the time when the video frame arrives at the decoder and completes the decoding; the display delay is The time from when a video frame is decoded to when it is displayed on the display;

judging whether the N is equal to the preset number of frames;

If the N is equal to the preset number of frames, determine the average decoding delay and the average display delay in the current full time period; the current full time period is from receiving the first video frame to determining the A period of time until the average decoding delay and the average display delay;

If the decoding delay and the display delay of the N frames of video frames are respectively at least c% lower than the average decoding delay and the average display delay, it is determined that the adjusted frame loss threshold is valid; if The decoding delay and the display delay of the N frames of video frames are not reduced by at least c% than the average decoding delay and the average display delay respectively, and it is determined that the adjusted frame loss threshold is invalid;

If the adjusted frame loss threshold is invalid, reduce the adjusted frame loss threshold and stop the threshold test; the threshold test is used to determine whether the adjustment to the frame loss threshold is valid.
The method of claim 8, wherein the video frame is a video frame not used for the threshold test.
The method according to claim 9, further comprising: discarding the video frame when the frame receiving time difference is less than a frame loss threshold.
The method according to any one of claims 8-10, wherein after receiving the video frame sent by the first electronic device, the method further comprises: recording the time of receiving the video frame; The frame threshold is initialized.
The method according to claim 11, wherein after said recording the time of receiving said video frame, said method further comprises: storing the time of receiving said video frame in a first queue; said first The time at which the second electronic device receives the Mth video frame is stored in the queue.
The method according to claim 12, characterized in that, before the decoding delay and the display delay of the N frames of video frames received after the video frame is received, the method further comprises: removing the first video frame The element written first in a queue writes the time of receiving the video frame into the first queue.
An electronic device, comprising a display screen, a memory, and one or more processors, wherein the memory is used to store a computer program; the processor is used to call the computer program, so that the electronic device executes the claims The method of any one of 1-13.
A computer storage medium, characterized by comprising: computer instructions; when the computer instructions are run on an electronic device, the electronic device is made to execute the method described in any one of claims 1-13.
A computer program product, the computer program comprising instructions, when the computer program is executed by a computer, the computer can execute the method according to any one of claims 1-13.