CN111818383B

CN111818383B - Video data generation method, system, device, electronic equipment and storage medium

Info

Publication number: CN111818383B
Application number: CN202010675531.6A
Authority: CN
Inventors: 李貌; 赵田; 杨晶生; 戴真理
Original assignee: Beijing ByteDance Network Technology Co Ltd
Current assignee: Beijing ByteDance Network Technology Co Ltd
Priority date: 2020-07-14
Filing date: 2020-07-14
Publication date: 2022-03-04
Anticipated expiration: 2040-07-14
Also published as: CN111818383A

Abstract

The embodiment of the disclosure discloses a method, a system, a device, an electronic device and a storage medium for generating video data, wherein the method comprises the following steps: responding to the sharing instruction, and presenting the sharing file corresponding to the sharing instruction on a window interface; processing the shared file according to the demonstration operation of the client-side triggered by the sharing instruction, and carrying out screenshot on a window interface in the processing process; video data is generated based on the screenshot image. The video data generation method, the video data generation system, the video data generation device, the electronic equipment and the storage medium disclosed by the embodiment of the disclosure can generate video data capable of presenting an information sharing process, and improve the watching experience of a user.

Description

Video data generation method, system, device, electronic equipment and storage medium

Technical Field

The embodiment of the disclosure relates to the technical field of computers, and in particular, to a method, a system, a device, an electronic device, and a storage medium for generating video data.

Background

With the continuous development of the internet and communication technology, information communication through communication applications has become one of the important ways for users to communicate information. Currently, a backend server for communication applications can pull multimedia data streams within a communication range from a streaming media server and perform streaming. When information sharing is performed based on communication applications, due to the fact that a barrier exists between a streaming media server and a server for information sharing, multimedia data synthesized by a back-end server of the communication applications cannot present an information sharing process, and therefore when a user watches recorded or live multimedia data, the user cannot watch content of shared information and operate the content, and the watching experience of the user is affected.

Disclosure of Invention

The embodiment of the disclosure provides a method, a system, a device, an electronic device and a storage medium for generating video data, which can generate video data capable of presenting an information sharing process and improve the watching experience of a user.

In a first aspect, an embodiment of the present disclosure provides a method for generating video data, including:

responding to a sharing instruction, and presenting a sharing file corresponding to the sharing instruction on a window interface;

processing the shared file according to the demonstration operation of the trigger client of the sharing instruction, and capturing a screenshot of the window interface in the processing process;

video data is generated based on the screenshot image.

In a second aspect, an embodiment of the present disclosure further provides a system for generating video data, including: the system comprises a real-time communication server, an information sharing server and a streaming media server;

the real-time communication server establishes communication connection with the information sharing server and the streaming media server, and is used for responding to a sharing instruction, accessing the information sharing server according to the sharing instruction and displaying a shared file obtained by accessing to a window interface; processing the shared file according to the demonstration operation of the trigger client of the sharing instruction, and capturing a screenshot of the window interface in the processing process; and according to the identity of the trigger client, pulling the multimedia data stream in the communication range corresponding to the identity from the streaming media server, and generating video data by using the screenshot image and the multimedia data stream.

In a third aspect, an embodiment of the present disclosure further provides a device for generating video data, including:

the file presentation module is used for responding to a sharing instruction and presenting a sharing file corresponding to the sharing instruction on a window interface;

the screenshot processing module is used for processing the shared file according to the demonstration operation of the client-side triggered by the sharing instruction and screenshot the window interface in the processing process;

and the data generation module is used for generating video data based on the screenshot image.

In a fourth aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement the video data generation method according to any one of the embodiments of the present disclosure.

In a fifth aspect, the embodiments of the present disclosure further provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are used for executing the video data generation method according to any one of the embodiments of the present disclosure.

According to the technical scheme of the embodiment of the disclosure, in response to a sharing instruction, a sharing file corresponding to the sharing instruction is presented on a window interface; processing the shared file according to the demonstration operation of the client-side triggered by the sharing instruction, and carrying out screenshot on a window interface in the processing process; and generating video data based on the screenshot image, so that the video data capable of presenting the information sharing process can be generated, and the watching experience of a user is improved.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.

Fig. 1 is a schematic flowchart of a method for generating video data according to a first embodiment of the disclosure;

fig. 2 is a schematic flowchart of a method for generating video data according to a second embodiment of the disclosure;

fig. 3 is a flowchart of a method for generating video data according to a second embodiment of the disclosure;

fig. 4 is a schematic diagram of a video data generation system according to a third embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of a video data generation apparatus according to a fourth embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of an electronic device according to a fifth embodiment of the disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

Example one

Fig. 1 is a schematic flow chart of a method for generating video data according to a first embodiment of the present disclosure, which is applicable to a situation of generating video data representing an information sharing process, and is particularly applicable to a situation of generating video data representing an information sharing process when a client performs information sharing in a live broadcast or recorded broadcast process. The method may be performed by a video data generating apparatus, which may be implemented in software and/or hardware, and may be configured in an electronic device, for example, in a backend server of a communication-type application.

As shown in fig. 1, the method for generating video data according to this embodiment includes:

and S110, responding to the sharing instruction, and displaying the sharing file corresponding to the sharing instruction on a window interface.

In the method for sharing information based on communication applications according to the embodiments of the present disclosure, multimedia data communication may be implemented as follows:

firstly, the multimedia data stream of each client in the communication range can be pulled; then, the multimedia data streams can be pushed to each client in the communication range after being converged, so that the multimedia data communication in the communication range is realized. The communication range is a range formed by clients participating in communication, for example, in a multimedia conference scene, each participating client of the multimedia conference forms a communication range. In a specific embodiment, the multimedia data communication mode may be performed by a backend server of the communication application, and the multimedia data stream may be pulled from the streaming server.

In the process of realizing multimedia data communication, sharing files can be further shared in the following way:

firstly, receiving a communication range provided by a back-end server of a communication application; then, monitoring a sharing instruction triggered by any client in the communication range, and for convenience of description, in the embodiment of the present disclosure, a client that triggers the sharing instruction may be referred to as a triggering client; then, a Uniform Resource Locator (URL) and a sharing range within the communication range of the shared file can be determined according to the sharing instruction; and finally, the URL of the shared file can be issued to other clients in the sharing range, so that the other clients in the sharing range can access the URL to the information sharing server, and can obtain the file content of the shared file by accessing the URL, thereby realizing that the other clients and the triggering client can simultaneously display the file content of the shared file. In a specific embodiment, the sharing manner of the shared file may be performed by an information sharing server.

By the method for sharing information based on the communication application, information can be shared simultaneously on the basis of the existing multimedia data communication. In addition, although the information sharing can be simultaneously performed on the basis of multimedia data communication through the commonly used screen sharing technology, the screen sharing technology has high requirements on network and video compression, various problems such as network delay, video decompression distortion and the like are easily caused in the transmission process of intercepted screen data, and the situation that a shared picture is not smooth and clear is easily caused. The information sharing method related to the embodiment of the disclosure can realize high-fidelity and smooth information sharing based on the URL of the shared information on the basis of multimedia data communication.

However, the method for sharing information based on communication applications according to the embodiments of the present disclosure has a drawback that, due to a barrier between the streaming media server and the information sharing server, when the backend server of the communication applications records or broadcasts the merged multimedia data, the recorded or broadcast multimedia data cannot present the information sharing process, so that a user of a viewer client outside the communication range cannot view the shared file and perform operations on the shared file when viewing the recorded or broadcast multimedia data, which affects the viewing experience of the user. The video data generation method provided by the embodiment of the disclosure can overcome the defects, can generate video data capable of presenting an information sharing process, and improves the user watching experience.

In this embodiment, the backend server of the communication application may receive a sharing instruction that triggers triggering of the client within the communication range while implementing multimedia data communication within the communication range. And the back-end server of the communication application can also respond to the sharing instruction to acquire the corresponding shared file and present the acquired shared file in a window interface opened at the back end of the server. It can be considered that the backend server of the communication application after receiving the sharing instruction can perform the same operation as other clients within the sharing scope in the communication scope, i.e. the shared file is presented in the window interface.

In a specific implementation manner, when monitoring a sharing instruction in a certain communication range, a backend server of a communication application may further determine whether a task of generating multimedia data, such as recording or live broadcasting, is currently executed for the communication range. If so, the corresponding shared file can be acquired in response to the sharing instruction, and the acquired shared file is presented in a window interface opened at the back end of the server. If not, the sharing instruction is not responded, so that the sharing instruction is responded only when the video data generation requirement is met, and the processing resource of the server can be saved to a certain extent.

By presenting the shared file corresponding to the sharing instruction on the window interface of the back-end server of the communication application, a foundation can be laid for processing the shared file and performing screenshot on the window interface in the step S120, that is, a foundation is laid for obtaining image data in the information sharing process.

And S120, processing the shared file according to the demonstration operation of the trigger client of the sharing instruction, and capturing the screen of the window interface in the processing process.

In the embodiment of the disclosure, the triggering of the demonstration operation of the client includes triggering the client to perform operations supported by the shared file, such as scrolling, page turning, editing, annotating, deleting, selecting, amplifying, shrinking, partially highlighting and the like on the shared file. After the back-end server of the communication application presents the shared file in the window interface, the back-end server of the communication application can also acquire demonstration operation from the information sharing server and/or the trigger client, and process the presented shared file according to the demonstration operation, so that the back-end server of the communication application can synchronize the shared file presented in the window interface of the trigger client. The back-end server of the communication application can also capture the window interface of the server in the process of processing the shared file, namely in the process of sharing the file in the window interface of the synchronous presentation trigger client, so that the acquisition of the image data including the information sharing process is realized.

In a specific embodiment, the triggering client may hand over the presentation authority to any other client within the sharing scope, and the client with the handed over presentation authority may perform the presentation operation on the shared file. For convenience of description, a client currently having a presentation right may be referred to as a presentation client. Correspondingly, the method for generating video data further comprises the following steps: when the demonstration permission instruction in the communication range is monitored, the shared file can be processed according to the demonstration operation of the demonstration client represented by the demonstration permission instruction, and screenshot is carried out on the window interface in the processing process. Moreover, the technical details of "processing the shared file according to the presentation operation of the trigger client" related to the present embodiment may be applied to "processing the shared file according to the presentation operation of the presentation client", and are not described herein again.

Optionally, the method for generating video data may further include: storing window demonstration operation of a trigger client of an acquired sharing instruction to a message queue, and acquiring current window demonstration operation to be processed from the message queue according to an acquisition timestamp of the window demonstration operation; correspondingly, the processing of the shared file according to the demonstration operation of the trigger client of the sharing instruction comprises the following steps: and processing the shared file according to the current window demonstration operation to be processed.

In this embodiment, the window presentation operation of the trigger client may include operations embodied in a window interface of the trigger client, for example, operations including cursor movement, scroll bar scrolling, and zooming on the content of the shared file. Under the condition that the window demonstration operation is not acquired by the information sharing server, the back-end server of the communication application can directly acquire the window demonstration operation of the trigger client, so that the operation of the window interface of the trigger client can be synchronously displayed in the window interface of the back-end server of the communication application. The back-end server of the communication application can store the acquired window demonstration operation in a message queue, for example, a Kafka message queue, so that the acquired window demonstration operation can be buffered, and the condition that the window demonstration operation is lost because a shared file is not processed according to the window demonstration operation can be effectively avoided.

Due to the influence of various factors such as a network in the transmission process, the acquisition sequence of the window demonstration operation may be different from the storage sequence stored in the message queue, and in order to ensure that the window interface of the back-end server of the communication application follows the operation of the window interface of the trigger client, the back-end server of the communication application may acquire the current window demonstration operation to be processed from the message queue according to the acquisition timestamp of the window demonstration operation, and process the shared file according to the current window demonstration operation to be processed.

Optionally, screenshot is performed on the window interface in the processing process, including: in the processing process, screenshot is carried out on the window interface according to a preset frequency; or, in the processing process, when the display change of the shared file is monitored, screenshot is carried out on the window interface.

In this embodiment, the backend server of the communication application performs screenshot on the window interface, which may be to uninterruptedly screenshot the window interface according to a preset frequency until the file sharing is finished, for example, when the shared file is presented on the window interface, the backend server starts to uninterruptedly screenshot the window interface according to a frequency of 3 times per second, and may stop screenshot on the window interface until receiving a sharing stop instruction that triggers the client. The preset frequency is not limited to 3 screenshots per second, and can be specifically adjusted according to the video smoothness of the information sharing process of the service demand or according to factors such as residual computing resources and storage resources of a back-end server of the communication application.

In addition, the backend server of the communication application captures the window interface, or captures the window interface when it is monitored that the display of the shared file changes, for example, when the backend server of the communication application processes the shared file according to the window demonstration operation and/or other demonstration operations, it may be considered that the display of the shared file changes, and at this time, captures the window interface. And when the back-end server of the communication application does not receive the window demonstration operation and/or other demonstration operations, the display of the shared file is not monitored to be changed, and at the moment, the screenshot of the window interface can be stopped, so that the processing resources of the back-end server of the communication application can be saved to a certain extent.

And S130, generating video data based on the screenshot image.

In the embodiment of the present disclosure, video data is generated based on the screenshot image, for example, in order to convert the screenshot image into a video frame format, an information sharing video stream is generated according to the screenshot image in the video frame format, and video data is generated according to the information sharing video stream.

When the backend server of the communication application obtains the screenshot image, the screenshot image can be converted from a picture format to a video frame format, for example, a jpg format is converted into an avi format, and the format conversion operation can be completed based on a third-party format conversion tool or based on a format conversion program local to the server. After obtaining the screenshot image sequence in the video frame format, the screenshot image sequence can be converted into an information sharing video stream based on program sentences such as python, opencv, ffmpeg and the like, and the information sharing video stream can be merged with one or more pulled video streams to obtain video data.

Further, generating video data according to the information sharing video stream includes: according to the identity of the trigger client, pulling the multimedia data stream in the communication range corresponding to the identity; and converging the information sharing video stream and the multimedia data stream to obtain video data.

In the embodiment of the present disclosure, the identity of the trigger client may be carried in the sharing instruction, and the identity may include at least one of an internet protocol address of the trigger client, account information of the client, a serial number of a device where the client is located, and an identifier of a real-time interaction room such as a conference room and a live broadcast room where the client is located.

The method comprises the following steps that a back-end server of the communication application can firstly pull multimedia data streams uploaded by a client and/or other clients in a communication range for interactive communication with the client from a streaming media server according to an identity mark of a trigger client, wherein the multimedia data streams include but are not limited to audio streams and video streams; and then, the pulled multimedia data stream and the information sharing video stream can be merged in real time according to the layout information of each data stream. When the synthesized video data is played at the audience client, the multimedia data stream and the information sharing stream can be spliced and presented at the playing window interface, so that a user at the audience client can watch the video data presenting the information sharing process, and the watching experience is improved.

In a specific embodiment, pulling the multimedia data stream within the communication range corresponding to the identity includes pulling the multimedia data stream of at least one of the following clients: a trigger client, a current presentation client, an active client, and a moderator client.

The current demonstration client is a client with demonstration permission currently in a communication range corresponding to the identity; the active client determining method comprises the steps of ranking clients in a communication range corresponding to the identity identification according to the speaking activity degree in a preset time period, and determining a preset number of clients ranked ahead as active clients; the host client is a client with communication management authority in a communication range corresponding to the identity, wherein the communication management authority includes but is not limited to closing a microphone and a camera, requesting to open the microphone and the camera, muting a whole person, canceling the muting of the whole person, locking a conference and the like.

Further, if the back-end server of the communication application is executing a task of generating video data such as recording or live broadcasting aiming at the communication range corresponding to the identity of the trigger client, the back-end server of the communication application can store the video data as a recording file and send the recording file to the client requesting the video data; or pushing the video data to a live broadcast server so that the live broadcast server sends the video data to a client requesting the video data.

When the audience client requests to watch the recorded file from the back-end server of the communication application or requests to watch the live video data from the live broadcast server, the video data in the information sharing process fed back by the back-end server or the live broadcast server of the communication application can be received and played, so that the user of the audience client can watch the multimedia data stream of each client in the communication range, can watch the content of the shared file in the communication process, can watch the operation on the content of the shared file and the like, and the watching experience of the user is improved.

Optionally, the method for generating video data provided in this embodiment is applied to a Real-time communication (RTC) server, where the RTC server includes at least one of an instant messaging server, a multimedia conference server, a video live broadcast server, and a group chat interaction server.

In this embodiment, the backend server of the communication application may be a real-time communication server, such as an instant messaging server, a multimedia conference server, a live video server, or a group chat server. When the real-time communication server receives a sharing instruction triggered in the communication process of a client, firstly, a sharing file corresponding to the sharing instruction can be obtained by accessing an information sharing server which establishes communication connection with the real-time communication server, and the sharing file is presented in a window interface of the real-time communication server; then, processing the shared file according to the demonstration operation of the trigger client, and carrying out screenshot on the window interface in the processing process; and finally, the multimedia data stream of the client in the communication range can be pulled from the streaming media server which establishes communication connection with the real-time communication server, and video data is generated according to the pulled multimedia data stream and the screenshot image. The information sharing server may be, for example, a DOCS server.

The real-time communication server can generate video data presenting an information sharing process when a task of generating video data such as recording or live broadcasting needs to be executed in the process of supporting real-time communication such as video conference or live broadcasting, and the like, so that the watching experience of a watching user of the video data is greatly improved.

Example two

The embodiments of the present disclosure and various alternatives in the generation method of video data provided in the above embodiments may be combined. The method for generating video data provided by this embodiment elaborates the process of cooperatively executing the task of generating video data by each execution resource in the backend server of the communication application, and can realize generation of video data in the process of sharing presentation information.

Fig. 2 is a flowchart illustrating a method for generating video data according to a second embodiment of the disclosure. As shown in fig. 2, the method for generating video data according to this embodiment includes:

and S210, responding to the sharing instruction to generate an information sharing task, and allocating execution resources for the information sharing task.

In this embodiment, a plurality of execution resources exist in the backend server of the communication application, each execution resource may include an execution module of a plurality of service tasks, and each execution resource may select a corresponding execution module to execute according to a specific task assigned thereto. The execution module comprises an execution module which is not present in a video recording task, an execution module of a video live broadcast task and an execution module of an information sharing task.

The back-end server of the communication application can respond to the sharing instruction to generate a new information sharing task, can select an execution resource capable of executing the information sharing task from the registered execution resources through the task scheduling service, and can distribute the newly generated information sharing task to the selected execution resource so that the execution resource selects an execution module corresponding to the information sharing task to execute the information sharing task.

And S221, starting a back-end browser through the execution resources of the information sharing task, and acquiring and accessing the uniform resource locator of the shared file carried by the sharing instruction.

In this embodiment, the execution resource of the information sharing task may start a back-end browser (e.g., a browser) through an execution module corresponding to the information sharing task, obtain a Uniform Resource Locator (URL) carried in the sharing instruction, and access the URL to obtain the shared file.

And S231, presenting the accessed shared file on a window interface of a back-end browser, and processing the shared file according to the editing demonstration operation of the client to the shared file, which is triggered by the accessed shared instruction.

In this embodiment, the editing and presentation operation may be regarded as an editing operation that is performed by the client on the content of the shared file, for example, the editing operation supported by the shared file includes adding, deleting, modifying, and annotating the content of the shared file. And the editing demonstration operation can be transmitted to a back-end server of the communication application through the information sharing server, specifically, the execution resource of the information sharing task obtains the shared file by accessing the URL carried by the sharing instruction, and simultaneously obtains the editing demonstration operation of the triggering client to the shared file. After the shared file obtained by access is presented in a window interface of a back-end browser, the execution resource of the information sharing task can further process the shared file according to the editing and demonstration operation obtained by access.

S222, storing window demonstration operation of the acquired sharing instruction triggering client to a message queue.

In this embodiment, the step and the step S221 are not strictly limited in time sequence, and the backend server of the communication application may collect the window demonstration operation of the trigger client in the process of executing the step S221 by the execution resource of the information sharing task, and store the collected window demonstration operation in the message queue.

S232, according to the acquisition time stamp of the window demonstration operation, the current window demonstration operation to be processed is obtained from the message queue, and the shared file is processed according to the current window demonstration operation to be processed.

In this embodiment, the execution resource of the information sharing task may obtain the current window demonstration operation to be processed from the message queue according to the acquisition timestamp of the window demonstration operation, and process the shared file presented by the back-end browser according to the current window demonstration operation to be processed.

And S240, screenshot is carried out on the window interface in the processing process.

In this embodiment, the execution resource of the information sharing task may perform screenshot on the window interface of the back-end browser in the process of processing the shared file according to the editing demonstration operation and/or the window demonstration operation.

And S250, pulling the multimedia data stream according to the identity of the triggering client through the execution resource of the video recording task or the video live broadcast task.

In this embodiment, the backend server of the communication application may pull the multimedia data stream from the streaming media server according to the identity of the triggering client based on the execution resource of the video recording task or the video live broadcast task. It should be noted that, when there is no client-side triggering sharing instruction in the communication range, but there is a client-side triggering recording instruction or live broadcasting instruction, the backend server of the communication application can still pull the multimedia data stream according to the execution resource of the video recording task or video live broadcasting task and the identity of the triggering client-side of the recording instruction or live broadcasting instruction. It can be considered that, in step S250, the backend server of the communication application executes in response to the recording instruction or the live broadcast instruction, and in the embodiment of the present disclosure, the receiving time of the recording instruction or the live broadcast instruction may be prior to the sharing instruction, may be subsequent to the sharing instruction, or may be received simultaneously with the sharing instruction.

And S260, based on a local socket communication mechanism, transmitting the screenshot image from the execution resource of the information sharing task to the execution resource of the video recording task or the video live broadcast task.

In this embodiment, after the execution resource of the information sharing task intercepts each screenshot image, the screenshot image may be transmitted to the execution resource that executes the video recording task or the video live broadcast task through a local socket communication mechanism (e.g., IPC socket mechanism).

And S270, generating video data based on the screenshot image and the multimedia data stream through the execution resources of the video recording task or the video live broadcast task.

In this embodiment, the execution resource executing the video recording task or the video live broadcast task may convert the received screenshot image from an image format to a video frame format, generate an information sharing video stream according to the screenshot image in the video frame format, and merge the information sharing video stream with the multimedia data stream to obtain video data.

Fig. 3 is a flowchart of a method for generating video data according to a second embodiment of the disclosure.

As shown in fig. 3, the backend server of the communication application is an RTC server, and the vchat is a backend service core, which can be used to process related communication services; scheduler is scheduling service and can be used for resource allocation of service tasks; the register is a resource registration service and can be used for managing available resources and distributing service tasks; the worker is a service task execution resource, each worker can be regarded as an execution resource, each worker can comprise an execution module binary for executing various service tasks, and corresponding binary can be started according to the type of the received task to execute the corresponding task, such as a record binary for executing a video recording task and a header binary for executing an information sharing task; the method is characterized in that the method is a window demonstration operation tool of an acquisition client, and can be used for acquiring and triggering window demonstration operation of the client.

The core generates a new information sharing task in response to the sharing instruction, and may select, through rtc.

The method comprises the steps of starting a back-end thread browser through a thread binding, obtaining a URL carried in a sharing instruction, accessing the URL from a DOCS server, obtaining a shared file, triggering an editing demonstration operation of a client on the shared file, presenting the accessed shared file on a window interface of the thread browser, and processing the shared file according to the accessed editing demonstration operation. And meanwhile, the RTC server can trigger window demonstration operation of the client through vchat.

In the process that the head library processes the shared file according to the editing demonstration operation and/or the window demonstration operation, screenshot can be carried out on a window interface of a head browser, and the screenshot picture can be transmitted to the record library through the IPC socket, so that the record library generates video data by utilizing the screenshot image and the multimedia data stream pulled from the multimedia server, and generates a recording file. If the screenshot picture is transmitted to or executed by the bind of the video live broadcast task through the IPC socket, the bind of the video live broadcast task can also push the video data to the live broadcast server after the video data is generated, so that the live broadcast server sends the video data to the client requesting the video data.

According to the technical scheme, the back-end browser can be started through execution resources of an information sharing task, a sharing file URL carried by a sharing instruction is accessed, the accessed sharing file is presented on a window interface of the back-end browser, the sharing file is processed according to editing and demonstrating operations obtained through access, the sharing file is processed according to the current window demonstrating operations to be processed, screenshot is conducted on the window interface of the back-end browser, and a screenshot image is transmitted to the execution resources of a video recording task or a video live broadcast task. Through the execution resources of the video recording task or the video live broadcast task, the multimedia data stream can be pulled according to the identity of the trigger client, and the video data is generated by utilizing the screenshot image and the multimedia data stream. Therefore, the back-end server of the communication application realizes the generation of video data in the information sharing process, and improves the watching experience of users. The method for generating video data provided by the embodiment of the present disclosure and the method for generating video data provided by the above embodiment belong to the same disclosure concept, and the technical details that are not described in detail in the embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in the embodiment and the above embodiment.

EXAMPLE III

Fig. 4 is a schematic diagram of a video data generation system according to a third embodiment of the present disclosure. The embodiment of the disclosure is suitable for generating video data presenting an information sharing process, and is particularly suitable for generating video data presenting an information sharing process when a client performs information sharing in a live broadcast or recorded broadcast process.

As shown in fig. 4, the video data generation system provided in this embodiment includes: a real-time communication server 410, an information sharing server 420 and a streaming media server 430;

the real-time communication server 410 establishes communication connection with the information sharing server 420 and the streaming media server 430, and is used for responding to a sharing instruction, accessing the information sharing server 420 according to the sharing instruction, and presenting an accessed shared file on a window interface; processing the shared file according to the demonstration operation of the client-side triggered by the sharing instruction, and carrying out screenshot on a window interface in the processing process; and according to the identity of the trigger client, pulling the multimedia data stream in the communication range corresponding to the identity from the streaming media server 430, and generating video data by using the screenshot image and the multimedia data stream.

Referring to fig. 4, the video data generation system may further include a live server 440;

the real-time communication server 410 establishes communication connection with the live broadcast server 440, and is further configured to push video data to the live broadcast server 440; correspondingly, the live broadcast server 440 is configured to send the video data to the client requesting the video data.

The video data generation system provided by the embodiment of the present disclosure and the video data generation method provided by the above embodiment belong to the same disclosure concept, and the technical details that are not described in detail in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

Example four

Fig. 5 is a schematic structural diagram of a video data generation apparatus according to a fourth embodiment of the disclosure. The video data generation apparatus provided in this embodiment generates video data representing an information sharing process, and is particularly suitable for generating video data representing an information sharing process when a client performs information sharing in a live broadcast or a recorded broadcast process.

As shown in fig. 5, the video data generation apparatus includes:

the file presenting module 510 is configured to, in response to the sharing instruction, present the shared file corresponding to the sharing instruction on the window interface;

a screenshot processing module 520, configured to process the shared file according to a demonstration operation of the shared instruction that triggers the client, and perform screenshot on a window interface in the process of processing;

a data generating module 530 for generating video data based on the screenshot image.

On the basis of the above technical solutions, the file presenting module includes:

the resource allocation submodule is used for responding to the sharing instruction to generate an information sharing task and allocating execution resources for the information sharing task;

and the file presentation sub-module is used for starting the back-end browser through the execution resources of the information sharing task, acquiring and accessing the uniform resource locator of the shared file carried by the sharing instruction, and presenting the accessed shared file on a window interface of the back-end browser.

On the basis of the technical schemes, the screenshot processing module comprises the following steps:

the first processing submodule is used for accessing the uniform resource locator to obtain a sharing instruction and triggering a client to edit and demonstrate the shared file; and processing the shared file according to the accessed editing demonstration operation.

On the basis of the above technical solutions, the apparatus for generating video data further includes:

the window demonstration operation processing module is used for storing the window demonstration operation of the acquired sharing instruction triggering client to the message queue and acquiring the current window demonstration operation to be processed from the message queue according to the acquisition timestamp of the window demonstration operation;

correspondingly, the screenshot processing module comprises:

and the second processing submodule is used for processing the shared file according to the current window demonstration operation to be processed.

the screenshot submodule is used for screenshot the window interface according to the preset frequency in the processing process; or, in the processing process, when the display change of the shared file is monitored, screenshot is carried out on the window interface.

On the basis of the above technical solutions, the data generation module is specifically configured to:

and converting the screenshot image into a video frame format, generating an information sharing video stream according to the screenshot image of the video frame format, and generating video data according to the information sharing video stream.

On the basis of the above technical solutions, the data generation module includes:

the stream pulling submodule is used for pulling the multimedia data stream in the communication range corresponding to the identity mark according to the identity mark of the trigger client;

and the confluence submodule is used for conflating the information sharing video stream and the multimedia data stream to obtain video data.

pulling the multimedia data stream according to the identity of the triggering client by executing resources of the video recording task or the video live broadcast task; based on a local socket communication mechanism, transferring the screenshot image from the execution resource of the information sharing task to the execution resource of a video recording task or a video live broadcast task; and generating video data based on the screenshot image and the multimedia data stream through the execution resources of the video recording task or the video live broadcast task.

On the basis of the above technical solutions, the video data generation device is applied to a real-time communication server, and the real-time communication server includes at least one of an instant messaging server, a multimedia conference server, a live video server and a group chat interaction server.

On the basis of the technical scheme, the shared file is obtained by accessing an information sharing server which establishes communication connection with a real-time communication server.

and the data pushing module is used for pushing the video data to the live broadcast server after the video data is generated by utilizing the screenshot image and the video stream, so that the live broadcast server sends the video data to the client requesting the video data.

The video data generation device provided by the embodiment of the disclosure can execute the video data generation method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

It should be noted that, the units and modules included in the apparatus are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the embodiments of the present disclosure.

EXAMPLE five

Referring now to fig. 6, a schematic diagram of an electronic device (e.g., the terminal device or the server in fig. 6) 600 suitable for implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 6, the electronic device 600 may include a processing means (e.g., central processor, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read-Only Memory (ROM) 602 or a program loaded from a storage means 606 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network through the communication device 609, or installed from the storage device 606, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the generation method of video data of the embodiment of the present disclosure.

The electronic device provided by the embodiment of the present disclosure is similar to the video data generating method provided by the above embodiment, and the technical details that are not described in detail in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

EXAMPLE six

The disclosed embodiments provide a computer storage medium having stored thereon a computer program that, when executed by a processor, implements the video data generation method provided by the above-described embodiments.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM) or FLASH Memory (FLASH), an optical fiber, a portable compact disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (Hyper Text Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

responding to the sharing instruction, and presenting the sharing file corresponding to the sharing instruction on a window interface; processing the shared file according to the demonstration operation of the client-side triggered by the sharing instruction, and carrying out screenshot on a window interface in the processing process; video data is generated based on the screenshot image.

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or hardware. The names of the units and modules do not limit the units and modules in some cases, and for example, the data generation module may be described as a "video data generation module".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Part (ASSP), a System On Chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, [ example one ] there is provided a method of generating video data, the method including:

video data is generated based on the screenshot image.

According to one or more embodiments of the present disclosure, [ example two ] there is provided a video data generation method, further comprising:

optionally, the presenting, in response to the sharing instruction, the shared file corresponding to the sharing instruction on the window interface includes:

responding to a sharing instruction to generate an information sharing task, and allocating execution resources for the information sharing task;

and starting a back-end browser through the execution resources of the information sharing task, acquiring and accessing the uniform resource locator of the shared file carried by the sharing instruction, and displaying the accessed shared file on a window interface of the back-end browser.

According to one or more embodiments of the present disclosure, [ example three ] there is provided a video data generation method, further comprising:

optionally, the processing the shared file according to the demonstration operation of the trigger client of the sharing instruction includes:

accessing the uniform resource locator to obtain the editing demonstration operation of the sharing instruction on the shared file by the triggering client;

and processing the shared file according to the accessed editing demonstration operation.

According to one or more embodiments of the present disclosure, [ example four ] there is provided a video data generation method, further comprising:

storing the acquired window demonstration operation of the trigger client of the sharing instruction to a message queue, and acquiring the current window demonstration operation to be processed from the message queue according to the acquisition timestamp of the window demonstration operation;

correspondingly, the processing the shared file according to the demonstration operation of the trigger client of the sharing instruction includes: and processing the shared file according to the current window demonstration operation to be processed.

According to one or more embodiments of the present disclosure, [ example five ] there is provided a video data generation method, further comprising:

optionally, the screenshot of the window interface in the processing process includes:

in the processing process, screenshot is carried out on the window interface according to a preset frequency; or,

and in the processing process, when the display change of the shared file is monitored, screenshot is carried out on the window interface.

According to one or more embodiments of the present disclosure, [ example six ] there is provided a video data generation method, further comprising:

optionally, the generating video data based on the screenshot image includes:

converting the screenshot image into a video frame format, generating an information sharing video stream according to the screenshot image of the video frame format, and generating video data according to the information sharing video stream.

According to one or more embodiments of the present disclosure, [ example seven ] there is provided a video data generation method, further comprising:

optionally, the generating video data according to the information sharing video stream includes:

according to the identity of the trigger client, pulling the multimedia data stream in the communication range corresponding to the identity;

and converging the information sharing video stream and the multimedia data stream to obtain video data.

According to one or more embodiments of the present disclosure, [ example eight ] there is provided a video data generation method, further comprising:

optionally, the generating video data based on the screenshot graph includes:

through the execution resources of the video recording task or the video live broadcast task, pulling the multimedia data stream according to the identity of the trigger client;

based on a local socket communication mechanism, transferring the screenshot image from the execution resource of the information sharing task to the execution resource of the video recording task or the video live broadcast task;

and generating video data based on the screenshot image and the multimedia data stream through the execution resources of the video recording task or the video live broadcast task.

According to one or more embodiments of the present disclosure, [ example nine ] there is provided a video data generation method, further comprising:

optionally, the method is applied to a real-time communication server, and the real-time communication server includes at least one of an instant messaging server, a multimedia conference server, a live video server, and a group chat interaction server.

According to one or more embodiments of the present disclosure, [ example ten ] there is provided a video data generation method, further comprising:

optionally, the shared file is obtained by accessing an information sharing server that establishes a communication connection with the real-time communication server

According to one or more embodiments of the present disclosure, [ example eleven ] there is provided a video data generation method, further comprising:

optionally, after generating the video data based on the screenshot image, the method further includes: and pushing the video data to a live broadcast server so that the live broadcast server sends the video data to a client requesting the video data.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for generating video data, which is applied to a real-time communication server, comprises the following steps:

responding to a sharing instruction, acquiring and accessing a uniform resource locator of a shared file carried by the sharing instruction, and displaying the accessed shared file on a window interface;

processing the shared file according to the demonstration operation of the trigger client of the sharing instruction, and capturing a screenshot of the window interface in the processing process; the triggering client is a client triggering the sharing instruction in a communication range of the real-time communication server;

generating video data based on the screenshot image;

wherein the generating video data based on the screenshot image comprises:

and pulling the multimedia data stream in the communication range corresponding to the identity identification according to the identity identification of the trigger client, and generating video data by using the screenshot image and the multimedia data stream.

2. The method according to claim 1, wherein the obtaining and accessing a uniform resource locator of a shared file carried by the sharing instruction in response to the sharing instruction, and presenting the accessed shared file in a window interface, comprises:

3. The method according to claim 2, wherein the processing the shared file according to the demonstration operation of the trigger client of the sharing instruction comprises:

4. The method of claim 1, further comprising:

5. The method of claim 1, wherein the screenshot of the window interface during processing comprises:

6. The method of claim 1, wherein generating video data using the captured image and the multimedia data stream comprises:

converting the screenshot image into a video frame format, generating an information sharing video stream according to the screenshot image of the video frame format, and generating video data according to the information sharing video stream;

7. The method of claim 2, wherein generating video data based on the screenshot image comprises:

8. The method of any one of claims 1-7, wherein the real-time communication server comprises at least one of an instant messaging server, a multimedia conferencing server, a video live server, and a group chat server.

9. The method of claim 1, wherein the shared file is obtained by accessing an information sharing server that establishes a communication connection with the real-time communication server.

10. The method of claim 1, further comprising, after the generating video data based on the screenshot image: and pushing the video data to a live broadcast server so that the live broadcast server sends the video data to a client requesting the video data.

11. A system for generating video data, comprising: the system comprises a real-time communication server, an information sharing server and a streaming media server;

the real-time communication server establishes communication connection with the information sharing server and the streaming media server, and is used for responding to a sharing instruction, acquiring a uniform resource locator of a shared file carried by the sharing instruction, accessing the information sharing server according to the uniform resource locator, and presenting the accessed shared file on a window interface; processing the shared file according to the demonstration operation of the trigger client of the sharing instruction, and capturing a screenshot of the window interface in the processing process; the triggering client is a client triggering the sharing instruction in a communication range of the real-time communication server; and according to the identity of the trigger client, pulling the multimedia data stream in the communication range corresponding to the identity from the streaming media server, and generating video data by using the screenshot image and the multimedia data stream.

12. The system of claim 11, further comprising a live server;

the real-time communication server establishes communication connection with the live broadcast server and is also used for pushing the video data to the live broadcast server;

correspondingly, the live broadcast server is used for sending the video data to the client side requesting the video data.

13. An apparatus for generating video data, applied to a real-time communication server, comprising:

the file presentation module is used for responding to a sharing instruction, acquiring and accessing a uniform resource locator of a shared file carried by the sharing instruction, and presenting the accessed shared file on a window interface;

the screenshot processing module is used for processing the shared file according to the demonstration operation of the client-side triggered by the sharing instruction and screenshot the window interface in the processing process; the triggering client is a client triggering the sharing instruction in a communication range of the real-time communication server;

the data generation module is used for generating video data based on the screenshot image;

the data generation module is specifically configured to:

14. An electronic device, characterized in that the electronic device comprises:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of generating video data according to any one of claims 1-10.

15. A storage medium containing computer-executable instructions for performing the method of generating video data according to any one of claims 1-10 when executed by a computer processor.