WO2012079510A1

WO2012079510A1 - Mute indication method and device applied to video conferencing

Info

Publication number: WO2012079510A1
Application number: PCT/CN2011/084000
Authority: WO
Inventors: 吴永明
Original assignee: 中兴通讯股份有限公司
Priority date: 2010-12-16
Filing date: 2011-12-14
Publication date: 2012-06-21
Also published as: CN102025972A

Abstract

Disclosed in the invention are a mute indication method and device applicable to video conferencing. The method comprises: a MCU carries out sound activation detection for a voice frequency media flow sent by a terminal which participates in the video conference and is muted; the MCU obtains the detection result of the terminal, wherein the detection result comprises one of the following states: sound activation state and sound inactivation state; and when the detection result is the sound activation state, the MCU superposes the mute video indication to the video signal of the terminal. Using the invention can solve the problem in the relevant technology that mute indication by using prompt tone is not intuitive and to some extent disturbs listening to the normal conference voice.

Description

TECHNICAL FIELD The present invention relates to the field of communications, and in particular to a mute indication method and apparatus for video conferencing. BACKGROUND OF THE INVENTION A video conferencing system is a multimedia communication system that supports remote bidirectional transmission of sound and video. It is used to help users in different places to perform real-time two-way face-to-face visual communication. Standards organizations such as the International Telecommunication Union (ITU), the Internet Engineering Task Force (IETF), and the Third Generation Partnership Project (3GPP) are each engaged in the development of multimedia standardization. The ITU currently develops multiple multimedia communication standards such as ITU-T H.320, ITU-T H.323, and ITU-T H.324. Among them, ITU-T H.320 is a multimedia communication application for narrowband circuit switched networks. ITU-T H.323 is a multimedia communication application for IP networks, and ITU-T H.324 is a multimedia communication application for very low-speed networks, such as PSTN (Public Switched Telephone Network) networks and mobile networks. . The IETF is responsible for the development of the Session Initiation Protocol SIP and multimedia conferencing standards based on this protocol. 3GPP is responsible for the development of the IP Multimedia Subsystem IMS standard. It also develops an IMS network-based multimedia conferencing standard based on the IETF standard. This standard is very close to the SIP-based standard established by the IETF. Figure 1 depicts the basic principles of video conferencing communication. The terminal 101 is a device used by the user, including terminals l~n. Each terminal contains a codec, which is responsible for compression encoding and decoding of sound, video and other media; the terminal is also connected to a microphone, a camera, a display, a sound playing subsystem for completing sound and video input and output; A user input interface is also included, and the user inputs commands and information to the terminal through the input interface. When a video conference is held, the terminal 101 establishes a connection with an MCU (Multipoint Control Unit) 102, including two-way communication of control signaling, audio, and video. To save network bandwidth, audio and video are generally in a compression-encoded format. Transfer on the network.

The MCU 102 is used to perform multi-party conference communication. The terminal 101 participating in the multi-party conference communication establishes a connection with the MCU 102 to perform two-way communication of control signaling, audio, and video. The MCU 102 is responsible for completing the exchange and mixing of the media streams. For the sound media stream, the MCU 102 generally outputs a sound-mixed sound media stream for each terminal 101, and the sound synthesis is generally superimposed by a plurality of sound media streams having the highest volume input. For video, the MCU 102 can send a single-picture video stream of another terminal for a certain terminal, if the MCU 102 supports multiple pictures. The face function can also combine the video from multiple terminals into one multi-picture image and then send it to one or some terminals. In video conferencing, in order to meet the needs of users for conference management, conference control functions are generally provided. The conference control software 103 of Figure 1 is used to complete the conference control function. An important function of the conference control software 103 is to perform mute control on the terminal. In order to achieve a good voice communication effect, the terminal that does not need to speak at present is usually muted, and after one terminal is muted, the other terminals participating in the same conference are engaged. The terminal's speech could not be heard. If the muted terminal is not notified that it is muted, the user of the terminal will attempt to perform the speaking operation, but the user on the other terminal side may not hear his speech, which may be misunderstood as a system failure, causing a decrease in ease of use. In a conventional audio conference system, a special tone is generally played by a mute terminal, for example, a "beep" tone is played at intervals. The disadvantage of this approach is that the prompts are not intuitive enough, and to some extent interfere with the listening of normal conference sounds. In view of the fact that the mute prompt in the related art adopts the prompt sound mode, it is not intuitive enough, and to some extent interferes with the problem of listening to the normal conference sound, and no effective solution has been proposed yet. SUMMARY OF THE INVENTION The present invention is directed to a method and apparatus for mute indication applied to a video conference, so as to solve the problem that the mute prompt adopts a prompt tone mode in the related art, which is not intuitive enough, and interferes to the listening of the normal conference sound to some extent. According to an aspect of the present invention, a mute indication method applied to a video conference is provided, including: a multipoint conference unit (MCU) performs sound activation detection on an audio media stream sent by a terminal that has participated in a video conference and has been muted; The MCU obtains the detection result of the terminal, where the detection result includes any one of the following: a sound activation state and a sound inactivity state; when the detection result is a sound activation state, the MCU is sent to the The mute video indication is superimposed in the video signal of the terminal. Preferably, the MCU performs sound activation detection on the audio media stream sent by the terminal participating in the video conference, and the method includes: the MCU periodically performing sound activation detection on the audio media stream. Preferably, the acquiring, by the MCU, the detection result of the terminal, includes: if the sound parameter of the audio media stream is higher than a threshold value of the sound activation detection, the MCU determines that the detection result is a sound activation state. And if the sound parameter of the audio media stream is not higher than a threshold value of the sound activation detection, the MCU determines that the detection result is a sound inactive state. Preferably, the MCU superimposes the mute video indication in a video signal sent to the terminal, where: the MCU superimposes a text or an icon in a video signal sent to the terminal, where the text or icon is used The terminal is instructed to be muted. Preferably, the superimposing the mute video indication in the video signal sent by the MCU to the terminal includes: repeating, by the MCU, superimposing the mute video indication on each video frame sent to the terminal, Until the mute video indication is cancelled. According to another aspect of the present invention, a mute indication device for a video conference is provided, which is disposed in an MCU, and includes: a detection module configured to perform sound on an audio media stream sent by a terminal that has participated in a video conference and has been muted. The detection module is configured to obtain the detection result of the terminal, where the detection result includes any one of the following: a sound activation state and a sound inactivity state; and an overlay module configured to activate when the detection result is sound In the state, the mute video indication is superimposed in the video signal transmitted to the terminal. Preferably, the detecting module is further configured to periodically perform sound activation detection on the audio media stream. Preferably, the acquiring module includes: a first determining submodule, configured to determine that the detection result is a sound activated state if a sound parameter of the audio media stream is higher than a threshold value of the sound activation detection; And determining, by the second determining submodule, if the sound parameter of the audio media stream is not higher than a threshold value of the sound activation detection, determining that the detection result is a sound inactive state. Preferably, the superimposing module is further configured to superimpose a text or an icon in the video signal sent to the terminal, where the text or icon is used to indicate that the terminal is muted. Preferably, the superposition module is further configured to perform a repetitive process of superimposing the mute video indication on each video frame sent to the terminal until the mute video indication is cancelled. In the embodiment of the present invention, the MCU performs sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted. When the detection result is the sound activation state, the MCU superimposes the silence video indication in the video signal sent to the terminal. . In the embodiment of the present invention, when a terminal is muted, if the user of the terminal attempts to speak, a mute video indication message is displayed in the received video signal, for example, "You are currently prohibited from speaking, please first Conduct an application to speak." The purpose of the embodiment of the present invention is to improve the communication experience of the video conference, and to make the video conference use simple and efficient. The advantage of the embodiment of the present invention is that the prompt information is intuitive, the information content of the prompt information can be rich and accurate, and the prompt information is dynamically appeared, and there is no prompt under normal circumstances to ensure minimal interference to the user. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are set to illustrate,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, In the drawings: FIG. 1 is a schematic diagram of a basic principle of a video conference communication according to the related art; FIG. 2 is a flowchart of a process of a mute indication method applied to a video conference according to an embodiment of the present invention; FIG. 3 is a flowchart according to an embodiment of the present invention. FIG. 4 is a schematic diagram of another MCU device supporting video overlay muting prompt information and corresponding processing flow according to an embodiment of the present invention; FIG. 5 is a schematic diagram of a corresponding processing flow according to an embodiment of the present invention; FIG. 6 is a display effect diagram of a mute prompt using a video superimposition manner according to an embodiment of the present invention; FIG. 7 is a display effect diagram of a mute prompt using a video insertion mode according to an embodiment of the present invention; 8 is a schematic structural diagram of a mute indication device applied to a video conference according to an embodiment of the present invention; and FIG. 9 is a schematic structural diagram of an acquisition module according to an embodiment of the present invention. BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in detail with reference to the accompanying drawings in conjunction with the embodiments. The invention will be described in detail below with reference to the drawings in conjunction with the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. In a conventional audio conference system, a special tone is generally played by a mute terminal, for example, a "beep" tone is played at intervals. The disadvantage of this approach is that the prompts are not intuitive enough, and to some extent interfere with the listening of normal conference sounds. To solve the above technical problem, the embodiment of the present invention provides a mute indication method applied to a video conference. The processing flow is as shown in FIG. 2, and includes: Step 202: The multi-point conference unit MCU performs sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted. Step 204: The MCU acquires the detection result of the terminal, where the detection result includes any one of the following: The state and the sound are not activated; Step 206: When the detection result is the sound activation state, the MCU superimposes the mute video indication in the video signal sent to the terminal. In the embodiment of the present invention, the MCU performs sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted. When the detection result is the sound activation state, the MCU superimposes the silence video indication in the video signal sent to the terminal. . In the embodiment of the present invention, when a terminal is muted, if the user of the terminal attempts to speak, a mute video indication message is displayed in the received video signal, for example, "You are currently prohibited from speaking, please first Conduct an application to speak." The purpose of the embodiment of the present invention is to improve the communication experience of the video conference, and to make the video conference use simple and efficient. The advantage of the embodiment of the present invention is that the prompt information is intuitive, the information content of the prompt information can be rich and accurate, and the prompt information is dynamically appeared, and there is no prompt under normal circumstances to ensure minimal interference to the user. Preferably, the MCU performs sound activation detection (VAD) on the audio media stream sent by the terminal participating in the video conference, and the method includes: the MCU periodically performs sound activation detection on the audio media stream. The MCU continuously performs sound activation detection on the audio media stream, and outputs a detection result of the sound activation state every time period τι. The detection result is two states, one is the sound activated state, and the other is the sound inactive state. T1 can be used as an adjustable MCU configuration item. Preferably, the MCU obtains the detection result of the terminal, including: if the sound parameter of the audio media stream is higher than the threshold of the sound activation detection, the MCU determines that the detection result is a sound activation state; if the sound parameter of the audio media stream is not higher than the sound When the detected threshold is activated, the MCU determines that the detection result is a sound inactive state. The threshold value of the VAD detection can be adjusted according to the specific situation. In implementation, according to the determination result of step 204, the MCU may select to superimpose (or insert) the mute video indication or cancel the superimposition (or insertion) video mute indication in the video signal sent to the terminal. The MCU checks whether the terminal is muted. If it is muted, it further determines whether the audio media stream sent by the current terminal is in an active state. If it is a voice activated state, it needs to send a mute video indication to the terminal, and other conditions are to stop sending the mute video. Instructions. The muting refers to the sound processing inside the MCU, and can prevent other terminals participating in the video conference from listening to the sound of the terminal in the video conference. Preferably, in step 206, the MCU superimposes the mute video indication in the video signal sent to the terminal, including: the MCU superimposes the text or the icon in the video signal sent to the terminal, and the text or icon is used to indicate that the terminal is Mute. Properties such as text, icon, font size, color, display position, etc. can be used as adjustable configuration items. In implementation, the MCU performs the iterative process of superimposing the mute video indication in each video frame sent to the terminal until the mute video indication is cancelled. Unmute the video indicator without superimposing the video frame. It can be known from the above description that after inserting the mute video indication, the MCU replaces the normal conference video stream with the mute cue video stream. The mute prompt video stream contains text or icon information to indicate that the terminal is muted. Unmute the video indicator to resume sending the normal conference video stream. FIG. 3 depicts an MCU device and corresponding processing flow for supporting video overlay mute prompt information based on an embodiment of the present invention. The network interface module 301 is responsible for communication with the terminal and is responsible for transmitting and receiving sound and video media streams. The network interface module 301 sends the received audio stream (1) to the audio decoding module 302, and the audio decoding module 302 decodes the compressed audio format into an original format audio stream, and then sends the original format audio stream (2) separately. The sound module 303 and the sound activation detecting module 304, the mixing module 303 is responsible for mixing the audio streams from the multiple terminals to achieve the effect of the multi-party call, and the mixing module 303 sends the mixed audio stream (4). The audio encoding module 305 is configured to compress and encode the original audio, and send the encoded audio stream (3) to the network interface module 301. The network interface module 301 sends the received video stream (5) to the video decoding module 306. The sound activation detection module 304 is responsible for performing sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted. In this embodiment, the value of T1 is 1000 ms, and the voice activation detection module 304 activates the voice every 1000 ms. (7) Reported to the main control module 307. The main control module 307 is responsible for determining whether a video mute indication is needed. If the terminal is muted, if the status indication of the sound activation is received, the mute video indication needs to be sent to the terminal, and in other cases, the mute video indication is stopped. The main control module 307 sends a command to send the mute video indication (8) to the graphics overlay module 308, and the video decoding module 306 sends the video code stream (6) of the original format sent to the terminal to the graphics overlay module 308. The graphics overlay module 308 is responsible for superimposing the muting prompt information into the video code stream of the original format sent to the terminal, and then sending the superimposed original format video stream (9) to the video encoder 309, and the video encoder 309 will use the original format. After the video stream is compressed and encoded, it is sent to the network interface module 301 and sent by the network interface module 301 to the terminal. The user can pre-set the volume comparison threshold, the volume calculation sample number or the corresponding time interval, the prompt text content, the text color, the font size, the font type, and the prompt text in the video frame by the device configuration mode to the MCU. In the device. FIG. 4 depicts another MCU apparatus and processing flow for supporting mute prompt information in a video insertion mode based on an embodiment of the present invention. The network interface module 401 is responsible for communication with the terminal, and is responsible for transmitting and receiving sound and video media streams. The network interface module 401 sends the received audio stream (1) to the audio decoding module 402, and the audio decoding module 402 decodes the compressed audio format into the original format audio stream, and then the original format audio stream (2) respectively. The mixing module 403 and the sound activation detecting module 404 are responsible for mixing the audio streams from the multiple terminals to achieve the effect of the multi-party call, and the mixing module 403 will mix the audio stream (4). The audio encoding module 405 is responsible for compressing and encoding the original audio, and sending the encoded audio stream (3) to the network interface module 401. The video mixing and switching module 406 receives the video stream (5) sent by the terminal, combines the video of the plurality of terminals into one multi-picture video, or selects the video input of a certain terminal to be exchanged to other terminals, and the video mixing and switching module 406 The output video stream (6) is sent to the video switching module 407. The sound activation detection module 404 is responsible for performing sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted. In this embodiment, the value of T1 is 1000 ms, and the voice activation detection module 404 activates the voice every 1000 ms. (8) is reported to the main control module 408. The main control module 408 is responsible for determining whether a video mute indication is needed. If the terminal is muted, if the status indication of the sound activation is received, the mute video indication needs to be sent to the terminal, and in other cases, the mute video indication is stopped. The main control module 408 sends a command to send the mute video indication (9) to the video switching module 407. The video switching module 407 selects a normal conference video stream (6) or a muting prompt video stream (7) to send to the terminal according to the command of the main control module 408. The video prompting module 409 is used to output a mute prompt video stream (7). The advantage of using a mute video prompt is to save media computing resources. Usually video overlay operations consume CPU resources. FIG. 5 is a flowchart of processing according to an embodiment of the present invention, which is illustrated based on the MCU embodiment of FIG. 3. The specific steps of the process are as follows: Step 501: Accept audio stream data in an original format input by the terminal, for example, receive audio data corresponding to a duration of 100 ms; Step 502: Perform sound activation detection by using the latest received audio stream data, depending on the VAD Algorithm, the calculation may need to use the saved historical audio stream data and the previous calculation result; the VAD decision threshold may be configured by the user, the decision sensitivity may be adjusted; Step 503, output the sound activation state; the execution body of steps 501 to 503 may be set to VAD After the execution of step 503, the process returns to 501 to repeat the execution; subsequently, the sound activation state is output to the main control module, and the main control module performs the subsequent steps 511 to 515; Step 511, receiving the input and updating the sound activation state; And determining whether it is a sound activated state, if it is a sound activated state, performing step 513, if it is a non-sound active state, performing step 515; Step 513, determining whether the terminal is muted, if it is muted, performing step 514, otherwise performing steps

Step 514: Send a request overlay prompt message, notify the video overlay module to perform video overlay, and return to step 511 to repeat the execution; Step 515, send a cancel overlay prompt message, notify the video overlay module to cancel the video overlay, and return to step 512 to repeat the execution; Subsequently, the request superimposed prompt message or the cancel superimposed prompt message is output to the video superimposing module, and the subsequent steps 521 to 524 are performed by the video superimposing module; Step 521, the video superimposing module updates the video superimposed state according to the input of the main control module; The video overlay module determines whether to perform video overlay, if step 523 is performed, otherwise step 524 is performed; step 523, the video overlay module superimposes the prompt information into the video signal sent to the terminal, and the prompt information may be an icon or descriptive expression that is silent. The text string; the content of the prompt text, font, text size, color, display position and other attributes can be used as an adjustable configuration item; Step 524, the video overlay module does not perform overlay processing. The mute indication method provided by the embodiment of the present invention can generate a mute prompt in the video. For example, FIG. 6 is a display effect of a mute prompt using a video overlay mode, and an outer rectangular box represents a television screen, and a character icon is used. Indicates the video signal watched by the terminal. The text side at the bottom is the superimposed mute prompt message. For example, if you are currently prohibited from speaking, please apply for a speech first. For another example, FIG. 7 shows the display effect of the mute prompt using the video insertion mode, and the outer rectangular box represents the television screen. Based on the same inventive concept, the embodiment of the present invention further provides a mute indication device applied to a video conference, and the structure thereof is as shown in FIG. 8 , and is disposed in the multi-point conference unit MCU, and includes: a detection module 801, configured to participate in The audio media stream sent by the terminal that has been muted by the video conference is subjected to sound activation detection. The acquisition module 802 is coupled to the detection module 801 and configured to acquire the detection result of the terminal, wherein the detection result includes any one of the following: a sound activation state and The sound is not activated; The superimposing module 803 is coupled to the acquisition module 802, and is configured to superimpose the mute video indication in a video signal sent to the terminal when the detection result is a sound activation state. In one embodiment, the detection module 801 can also be configured to periodically perform sound activation detection on the audio media stream. In an embodiment, as shown in FIG. 9, the obtaining module 802 may include: a first determining submodule 901, configured to determine that the detection result is sound if the sound parameter of the audio media stream is higher than a threshold value of the sound activation detection. The second determination sub-module 902 is configured to determine that the detection result is a sound inactive state if the sound parameter of the audio media stream is not higher than the threshold value of the sound activation detection. The first determining submodule 901 and the second determining submodule 902 are two parallel functional modules, which are respectively coupled to the detecting module 801. In one embodiment, the overlay module 803 can also be configured to superimpose text or icons in the video signal sent to the terminal, the text or icon being used to indicate that the terminal is muted. In one embodiment, the overlay module 803 can also be configured to perform a repetitive process of superimposing the mute video indication on each video frame sent to the terminal until the mute video indication is cancelled. From the above description, it can be seen that the present invention achieves the following technical effects: In the embodiment of the present invention, the MCU performs sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted, when the detection result is When the sound is activated, the MCU superimposes the mute video indication in the video signal sent to the terminal. In the embodiment of the present invention, when a terminal is muted, if the user of the terminal attempts to speak, a mute video indication message is displayed in the received video signal, for example, "You are currently prohibited from speaking, please first Conduct an application to speak." The purpose of the embodiment of the present invention is to improve the communication experience of the video conference, and to make the video conference use simple and efficient. The advantage of the embodiment of the present invention is that the information is improved intuitively, and the information content of the prompt information can be rich and accurate, and the prompt information is dynamically appeared, and there is no prompt under normal circumstances to ensure minimal interference to the user. Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device, which can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device so that they may be stored in the storage device by the computing device, or they may be separately fabricated into respective integrated circuit modules. Alternatively, multiple modules or steps of them can be implemented as a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software. The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Claims

Claim

1. A mute indication method applied to a video conference, comprising:

Multi-point conference unit The MCU performs sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted;

The MCU acquires a detection result of the terminal, where the detection result includes any one of the following: a sound activation state and a sound inactivation state;

When the detection result is a sound activated state, the MCU superimposes the silent video indication in a video signal transmitted to the terminal.

The method according to claim 1, wherein the MCU performs sound activation detection on the audio media stream sent by the terminal participating in the video conference, and the method includes: the MCU periodically performing sound activation detection on the audio media stream.

The method according to claim 1 or 2, wherein the acquiring, by the MCU, the detection result of the terminal includes:

And if the sound parameter of the audio media stream is higher than a threshold value of the sound activation detection, the MCU determines that the detection result is a sound activation state;

And if the sound parameter of the audio media stream is not higher than a threshold value of the sound activation detection, the MCU determines that the detection result is a sound inactive state.

4. The method according to claim 3, wherein the MCU superimposes the mute video indication in a video signal sent to the terminal, comprising: the MCU superimposing text in a video signal sent to the terminal Or an icon, the text or icon is used to indicate that the terminal is muted.

5. The method according to claim 4, wherein the MCU superimposes the mute video indication in a video signal sent to the terminal, comprising: the MCU performing each video frame sent to the terminal The iterative process of the mute video indication is superimposed until the mute video indication is cancelled.

A mute indicating device applied to a video conference, which is disposed in the MCU of the multipoint conference unit, and includes: a detecting module, configured to perform sound activation detection on the audio media stream sent by the terminal that has participated in the video conference and has been muted; An acquiring module, configured to obtain a detection result of the terminal, where the detection result includes any one of the following: a sound activation state and a sound inactivation state;

The superimposing module is configured to superimpose the mute video indication in a video signal transmitted to the terminal when the detection result is a sound activated state.

7. The apparatus according to claim 6, wherein the detecting module is further configured to periodically perform sound activation detection on the audio media stream.

The device according to claim 6 or 7, wherein the obtaining module comprises: a first determining submodule, configured to: if a sound parameter of the audio media stream is higher than a threshold value of the sound activation detection Determining that the detection result is a sound activation state;

And a second determining submodule configured to determine that the detection result is a sound inactive state if the sound parameter of the audio media stream is not higher than a threshold value of the sound activation detection.

9. The apparatus according to claim 8, wherein the superimposing module is further configured to superimpose a text or an icon in a video signal sent to the terminal, the text or icon being used to indicate that the terminal is muted.

10. The apparatus of claim 9, wherein the superimposing module is further configured to perform a repetitive process of superimposing the mute video indication on each video frame transmitted to the terminal until the mute video indication is cancelled.