CN103236259A - Voice recognition processing and feedback system, voice response method - Google Patents

Voice recognition processing and feedback system, voice response method Download PDF

Info

Publication number
CN103236259A
CN103236259A CN2013100956595A CN201310095659A CN103236259A CN 103236259 A CN103236259 A CN 103236259A CN 2013100956595 A CN2013100956595 A CN 2013100956595A CN 201310095659 A CN201310095659 A CN 201310095659A CN 103236259 A CN103236259 A CN 103236259A
Authority
CN
China
Prior art keywords
voice
user
speech
feedback
storehouse
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN2013100956595A
Other languages
Chinese (zh)
Other versions
CN103236259B (en
Inventor
毛惟谦
王珏
彭培源
郑园蓁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LG Electronics China Research and Development Center Co Ltd
Shanghai LG Electronics Co Ltd
Original Assignee
Shanghai LG Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai LG Electronics Co Ltd filed Critical Shanghai LG Electronics Co Ltd
Priority to CN201310095659.5A priority Critical patent/CN103236259B/en
Publication of CN103236259A publication Critical patent/CN103236259A/en
Application granted granted Critical
Publication of CN103236259B publication Critical patent/CN103236259B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Electrically Operated Instructional Devices (AREA)

Abstract

The invention provides a voice recognition processing and feedback system and a voice response method. The method includes the steps of collecting user's voice information, identifying the user's voice information and transforming the identified voice information into text information, generating a text feedback corresponding to the text information, detecting the user's gender and age, acquiring and storing a male speech database, a female voice database and a children speech database, selecting the female speech database to perform the speech synthesis of the text feedback so as to generate a female voice file to reply to the user when the detected user is a male but not a child, selecting the male speech database to perform the speech synthesis of the text feedback so as to generate a male voice file to reply to the user when the detected user is a female but not a child, and selecting the children database to perform the speech synthesis of the text feedback so as to generate a cartoon voice file to reply to the user when the detected user is a child. With the method, different voice responses with different tones can be generated based on a speaker's age and gender.

Description

Voice recognition processing and feedback system, voice replying method
Technical field
The present invention relates to speech recognition and phonetic synthesis, relate to a kind of voice recognition processing and feedback system, voice replying method particularly.
Background technology
In existing voice identification processing and feedback system, generally include three important ingredients: speech recognition, speech understanding and speech answering.In this module of third part speech answering, what adopt is that single speech intonation and indifference are replied, and can only be divided into male voice and female voice substantially, and the user also can only select by arranging in advance for the demand of different phonetic.Can't satisfy the user for the demand of diversity speech answering, more can't automatically be converted to the speech answering of user preferences at the current user that speaks.
Traditional voice identification is handled and feedback system, and operating platform can be various multimedia platforms such as the mobile phone that has network function, perhaps portable terminal etc., and this sentences TV platform is example.Traditional voice identification handles and feedback system mainly is divided into following three modules: sound identification module, speech understanding and processing module, speech answering module.System sets in advance fixedly speech intonation answer, generally is divided into male voice and female voice, need be selected reply content male and female students unanimity in system's setting voluntarily by the user.
Fig. 1 is the voice replying method synoptic diagram in traditional voice identification processing and the feedback system.As shown in Figure 1, at first system is carried out phonetic entry, clear and definite instruction separately as any class user (as the man, woman or children).System identifies user's voice input, and voice messaging is converted into Word message.Enter speech understanding and processing module then, Word message is further understood and analysis, draw corresponding literal feedback.Finally change the speech answering module over to, the literal feedback can be transfused to the phonetic synthesis storehouse, and sound bank generally is divided into male voice and female voice, and the user need arrange selection in advance, and selecteed sound bank can generate voice document, and voice feedback is given the user the most at last.This speech answering and feedback service are: 1, and same pronunciation; 2, same feedback.
The inventor finds that in realizing process of the present invention the problem that prior art exists comprises at least:
1, speech answering is single
Integrated 1 to 2 sound bank of the general only meeting of traditional system needs the user to select voluntarily during use.So the voice feedback which kind of user hears all has only a kind of speech intonation.The user can produce auditory fatigue in the long-term process of using.Even added identity recognition function in the voice system that has again, but also just increase personalized service in final speech answering module, voice and intonation had not been carried out corresponding the adjustment.
2, can't attract children population
For children population, the attractive force of voice system more is embodied on the interactive experience of hearing, for current traditional voice system, can't satisfy children population for diversity and interesting requirement.Therefore can't transfer the use interest of this colony.Hindered in a sense the voice system expansion has been a kind of potential function of educational mode.
Summary of the invention
The objective of the invention is to, overcome the deficiencies in the prior art, a kind of voice recognition processing and feedback system, voice replying method are provided, so that in voice system, provide speech answering function multiple personalization, differentiation to the user, can carry out speech answerings differentiated, different intonation according to speaker's age and sex.
On the one hand, for reaching above-mentioned purpose, the embodiment of the invention provides a kind of voice recognition processing and feedback system, and described system comprises:
The voice collecting unit is used for gathering user's voice information;
Voice recognition unit is used for the user's voice information of gathering is identified, and the voice messaging that identifies is converted into Word message;
Speech understanding and processing unit, described speech understanding and processing unit comprise: the literal feedback module, being used for generating ground should be in the literal feedback of described Word message; The sex detection module is for detection of user's sex; And the age detection module is for detection of user's age;
Storage unit is used for the storaged voice storehouse, and described sound bank comprises: male voice storehouse, female voice storehouse and children speech storehouse;
The speech answering unit, comprise phonetic synthesis module and speech play module, described phonetic synthesis module be used for when detected user for the male sex and when being not children, select described female voice storehouse that described literal feedback is carried out phonetic synthesis, generation female voice voice document; When detected user for the women and when being not children, select described male voice storehouse that described literal feedback is carried out phonetic synthesis, generate the male voice voice document; When detected user is children, select described children speech storehouse that described literal feedback is carried out phonetic synthesis, generate the cartoon voice document; Described speech play unit is used for playing the corresponding voice document that described phonetic synthesis module generates.
On the other hand, for reaching above-mentioned purpose, the embodiment of the invention provides the voice replying method of a kind of voice recognition processing and feedback system, and described method comprises:
Gather user's voice information;
The user's voice information of gathering is identified, the voice messaging that identifies is converted into Word message;
Generating ground should be in the literal feedback of described Word message;
Detect user's sex;
Detect user's age;
Obtain and the storaged voice storehouse, described sound bank comprises: male voice storehouse, female voice storehouse and children speech storehouse;
When detected user for the male sex and when being not children, select the female voice storehouse that described literal feedback is carried out phonetic synthesis, generate the female voice voice document and reply described user; When detected user for the women and when being not children, select the male voice storehouse that described literal feedback is carried out phonetic synthesis, generate the male voice voice document and reply described user; When detected user is children, select the children speech storehouse that described literal feedback is carried out phonetic synthesis, generate the cartoon voice document and reply described user.
The useful technique effect of the technique scheme of the embodiment of the invention is:
1, in voice system, the invention provides user multiple personalization, the speech answering function of differentiation, to have only the single selection of 2 kinds of speech answerings originally, rise to the n kind and select, the likability and the user that strengthen the user experience, improve the interest that the user uses this product, increase the probability of buying this product.
2, the present invention is for children population, and the use interest of this colony transferred in diversity and interesting voice.It is a kind of potential educational system that voice system is expanded.On the basis of identification children identity information, can also monitor children's audio-visual content simultaneously.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, to do one to the accompanying drawing of required use in embodiment or the description of the Prior Art below introduces simply, apparently, accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the synoptic diagram of the speech answering mode in traditional voice identification processing and the feedback system;
Fig. 2 is the voice recognition processing of the embodiment of the invention and the system functional block diagram of feedback system;
Fig. 3 is the concrete function block diagram of voice recognition processing and feedback system in another embodiment of the present invention;
Fig. 4 is voice recognition processing and feedback system measures user sex and the automatic synoptic diagram of selecting reply voice of age of the embodiment of the invention;
Fig. 5 is that the sound characteristic of the voice recognition processing of the embodiment of the invention and feedback system collection hobby object is by the synthetic synoptic diagram as reply voice of system;
Fig. 6 is the process flow diagram of the voice replying method of the embodiment of the invention.
Embodiment
For the purpose, technical scheme and the advantage that make the embodiment of the invention clearer, below in conjunction with the accompanying drawing in the embodiment of the invention, technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills belong to the scope of protection of the invention not making the every other embodiment that obtains under the creative work prerequisite.
The embodiment of the invention provides a kind of voice recognition processing and feedback system.Fig. 2 is the system functional block diagram of the embodiment of the invention, and as shown in Figure 2, this system 100 comprises:
Voice collecting unit 110 is used for gathering user's voice information;
Speech recognition (Speech Recognition) unit 120 is used for the user's voice of gathering is identified, and the voice messaging that identifies is converted into Word message;
Speech understanding and processing (Language Understanding and Processing) unit 130, it comprises: literal feedback module 131, being used for generating ground should be in the literal feedback of described Word message; Sex detects (gender detection) module 132, for detection of user's sex; And the age is detected (age detection, children or adult) module 133, for detection of user's age;
Storage unit 140 is used for the storaged voice storehouse, and described sound bank comprises: male voice storehouse, female voice storehouse and children speech storehouse;
Speech answering unit 150, comprise phonetic synthesis (Text to Speech) module 151 and speech play module 152, described phonetic synthesis module 151 be used for when detected user be the male sex and when being not children, select described female voice storehouse that described literal feedback is carried out phonetic synthesis, generate the female voice voice document; When detected user for the women and when being not children, select described male voice storehouse that described literal feedback is carried out phonetic synthesis, generate the male voice voice document; When detected user is children, select described children speech storehouse that described literal feedback is carried out phonetic synthesis, generate the cartoon voice document; Described speech play unit 152 is used for playing the corresponding voice document that described phonetic synthesis module generates.
The course of work of this system 100 is as follows: system receives the user's voice input by voice collecting unit 110, convert literal to through voice recognition unit 120, enter speech understanding and processing unit 130 is handled, determine user identity and obtain replying literal, which determined with sound bank (male voice/female voice/cartoon sound/User Defined sound/famous person's sound etc.) by the speech answering unit then, call corresponding sound bank from storage unit 140, to reply literal through phonetic synthesis module 151 and synthesize voice, play out sound by speech play module 152.
Particularly, the Word message that voice recognition unit generates, enter the literal feedback module after, generate and reply literal; The sound characteristic information that obtains is judged user's sex and age bracket through sex detection module and age detection module, and testing result is outputed to the phonetic synthesis module, determines to adopt which sound bank by this phonetic synthesis module.
In a preferred embodiment, described speech answering unit 150 can also be used for replying described user after the voice document that generates adds the appellation and/or user's interest reply content that is associated with user's identity.
In another preferred embodiment, another kind of voice recognition processing and feedback system 200 are provided, see also Fig. 3, described system 200 also comprises: mode setting unit 160, be used for receiving external command, the mode of operation that described system is set is multi-lingual accent speech answering pattern, hobby object speech answering pattern or characteristic speech answering pattern; Under described multi-lingual accent speech answering pattern, sex detection module 132 and age detection module 133 are in running order, and under other patterns, sex detection module 132 and age detection module 133 are in off position.
Described speech understanding and processing unit 130 can further include: sound characteristic extracts and synthetic (speech feature extraction and synthesis) module 134, be used under hobby object speech answering pattern, gather the phonetic feature of the hobby object of designated user, and generate the sound bank of hobby object according to the phonetic feature of gathering.Described phonetic synthesis module 151, can also be used for when described user is designated user, select the sound bank of the hobby object of described designated user that described literal feedback is carried out phonetic synthesis, generate hobby object voice document, and reply described designated user by this speech play unit 152.
In another preferred embodiment, described storage unit 140 also stores the characteristic sound bank.Described phonetic synthesis module 151 can also be used under characteristic speech answering pattern, selects described characteristic sound bank that described literal feedback is carried out phonetic synthesis, generates the characteristic voice document, and replys described user by this speech play module 152.
Wherein, this multi-lingual accent speech answering pattern be before mode of operation described in Fig. 2, carry out corresponding multi-lingual accent speech answering according to user's sex, age.
Particularly, selecting male/female/children speech to reply in the pattern (multi-lingual accent speech answering pattern) automatically, module 131,132,133 work, module 134 is not worked; In hobby object speech answering pattern, module 131,134 work, module 132,133 is not worked; In specific human voices (as the famous person) answer pattern, module 131 work, module 132,133,134 is not worked.
The embodiment of the invention can strengthen experiencing for the likability of whole speech recognition system and user of user, make total system the sensory experience of when service is provided, having enriched the user more and use enjoyment, and the new service mode of having derived.
Below voice recognition processing and the feedback system of the embodiment of the invention made a more detailed description:
The embodiment of the invention realizes multi-lingual accent speech answering at user's sex, age, identity, hobby by following multiple mode.
1, detects user's sex and age and select reply voice automatically.By in speech understanding and processing unit, adding sex identification, age identification (adult or children), specific identity is identified this type of mature gordian technique, obtain speaker information, and shift to an earlier date integrated multiple voice packet, automatically switch to the relative speech answering that meets its hobby at the people of different identity (referring to age and sex), for example reply men's family with female voice, male voice is replied lady's family, and cartoon sound is replied child user.
In first kind of implementation, as shown in Figure 4, the embodiment of the invention adds functional modules such as sex detection, age detection in speech understanding and processing unit, adds one simultaneously again at the cartoon sound bank of children's hobby in system.System can pass to the speech answering module to speaker's identity information in identification voice and processing, the speech answering unit selects corresponding sound bank to carry out phonetic synthesis according to the identity information that provides automatically.So just can realize three kinds of answer effects at different groups.
Wherein, sex detection module, age detection module, identity detection module can detect speaker's age, sex, identity respectively.In this speech recognition and Audio Processing Unit, be written into three kinds of voice packets, be respectively male voice bag, female voice bag and at children's cartoon voice packet.Below all belong to mature technology.
In this specific implementation, system can provide voice, appellation and service feedback targetedly automatically.For example:
At women user, provide male voice to reply automatically.Reply to women's honorific title and at its hobby.
At male user, provide female voice to reply automatically.Reply to the male sex's honorific title and at its hobby.
At child user, provide the cartoon speech answering automatically.Reply to children's address and at its hobby.
In addition, in this specific implementation, the personalized service that can provide comprises: 1) at classification recommendation of different groups etc.; 2) watching and listening to and monitor and protect children.
2, the sound characteristic of gathering the hobby object is synthesized as reply voice by system.Like the sound of object to substitute by integrated, in system, add the sound synthesis module, utilize people that this system likes oneself or at one's side relatives' sound carry out the feature sampling, utilize the sample collection pronunciation character then, can imitate sampling people's sounding to greatest extent, thereby replace single speech answering.
In second kind of implementation, as shown in Figure 5, at the certain user prefer hearing oneself be familiar with or hobby person's sound as speech answering, the present invention can provide the user to select to like object speech answering option.The embodiment of the invention adds sound characteristic and extracts and synthesis module, the hobby object can come system is carried out sampled voice by 2 kinds of modes: 1), the sampling of nonsystematic duty is namely under the situation of not using voice system, open input port, under the guidance of system, carry out voice and collect.2), the working state of system sampling, namely working as pass-through mode 1 has adopted to such an extent that like under the prerequisite of object samples, if the hobby object is also using voice system, system can confirm that this sound derives from the hobby object by the function that identity detects in the process of handling so, collects for it carries out voice automatically, because the more collection of more voice information, can so that the final voice that generate more near picker's intonation, mode 2) can be used as a kind of function of study.System is by the phonetic feature of above 2 kinds of mode acquisition targets and generate the sound bank of liking object automatically.In the module of final speech answering, system just adopts this hobby people's sound bank to become voice next life automatically, will like the speech answering of object to feed back to the spokesman.
In this specific implementation, speech recognition and Audio Processing Unit comprise: sex detection module, age detection module, identity detection module are respectively applied to detect speaker's sex, age, identity; And phonetic feature is gathered synthesis module, is used for gathering the phonetic feature of hobby object, and the voice that generate the hobby object automatically comprise, and are kept in the system.
3, provide famous person or the download service of characteristic voice.Famous person or characteristic voice (for example robot, outman) download service is provided, enriches the speech answering of system self.
In the third implementation, the embodiment of the invention provides the option of famous person or characteristic speech answering again on the basis of mode 1,2.Can be with famous person or some distinctive voice (for example robot sounding, outman's sounding etc.) the synthetic high in the clouds that is placed in advance.Voluntarily download sound bank file under the condition of network is being arranged by the user, and be installed to assigned address.Equally, in the module of final speech answering, system adopts the library file of downloading to generate speech answering and gives the spokesman.More than three kinds of modes can exist simultaneously as option, do not interact, select wherein a kind of for the user.
The third mode can be triggered by the user, and namely system provides famous person or characteristic sound bank option, the voice that the user can select a certain characteristic voice to reply as system.With feedback text synthetic speech, answer allows the user hear according to user-selected sound bank in system.
Fig. 6 is the process flow diagram of the voice replying method of a kind of voice recognition processing of the embodiment of the invention and feedback system.As shown in Figure 6, described method comprises the steps:
610, gather user's voice information;
620, the user's voice information of gathering is identified, the voice messaging that identifies is converted into Word message;
630, generating ground should be in the literal feedback of described Word message;
640, detect user's sex;
650, detect user's age;
660, obtain also storaged voice storehouse, described sound bank comprises: male voice storehouse, female voice storehouse and children speech storehouse;
670, when detected user for the male sex and when being not children, select the female voice storehouse that described literal feedback is carried out phonetic synthesis, generate the female voice voice document and reply described user; When detected user for the women and when being not children, select the male voice storehouse that described literal feedback is carried out phonetic synthesis, generate the male voice voice document and reply described user; When detected user is children, select the children speech storehouse that described literal feedback is carried out phonetic synthesis, generate the cartoon voice document and reply described user.
In a preferred embodiment, described method is replied described user after can also comprising the steps: to add the appellation and/or user's interest reply content that is associated with user's identity in the voice document that generates.
In another preferred embodiment, described method can also comprise the steps:
Receive external command, the mode of operation that described system is set is multi-lingual accent speech answering pattern, hobby object speech answering pattern or characteristic speech answering pattern;
Under hobby object speech answering pattern, gather the phonetic feature of the hobby object of designated user, and generate the sound bank of hobby object according to the phonetic feature of gathering; When described user is designated user, select the sound bank of the hobby object of described designated user that described literal feedback is carried out phonetic synthesis, generate hobby object voice document and reply described designated user.
In another preferred embodiment, described method can also comprise the steps: to obtain the characteristic sound bank under characteristic speech answering pattern; Select described characteristic sound bank that described literal feedback is carried out phonetic synthesis according to user instruction, generate the characteristic voice document and reply described user.
The useful technique effect of the embodiment of the invention:
1, in voice system, the invention provides user multiple personalization, the speech answering function of differentiation, to have only the single selection of 2 kinds of speech answerings originally, rise to the n kind and select, the likability and the user that strengthen the user experience, improve the interest that the user uses this product, increase the probability of buying this product.
2, in voice system, the invention provides the multiple option that the user selects different phonetic to reply, with respect to the traditional mode that can only in option, select voice, the user can select the voice wanted according to self-interest, can be user's selection automatically by the identification identity information by system also.The space that provides more choices, the automatic selection of simultaneity factor is also convenient for users to use, makes operation more convenient.
3, the present invention is for children population, and the use interest of this colony transferred in diversity and interesting voice.It is a kind of potential educational system that voice system is expanded.On the basis of identification children identity information, can also monitor children's audio-visual content simultaneously.
4, the present invention is directed at traditional voice and reply the mode kind, can't carry out differentiated answer shortcoming at speaker's identity, also improve, identify according to identity information, system not only can provide the speech answering of different intonation, and can also add different appellations and user simultaneously may interested reply content.
Those skilled in the art can also recognize the various illustrative components, blocks (illustrative logical block) that the embodiment of the invention is listed, and unit and step can be passed through electronic hardware, computer software, or both combinations realize.Be the clear replaceability (interchangeability) of showing hardware and software, above-mentioned various illustrative components (illustrative components), unit and step have been described their function generally.Such function is to realize depending on the designing requirement of specific application and total system by hardware or software.Those skilled in the art can be for every kind of specific application, and can make ins all sorts of ways realizes described function, but this realization should not be understood that to exceed the scope of embodiment of the invention protection.
Various illustrative logical block described in the embodiment of the invention, or the unit can pass through general processor, digital signal processor, special IC (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the design of above-mentioned any combination realizes or operates described function.General processor can be microprocessor, and alternatively, this general processor also can be any traditional processor, controller, microcontroller or state machine.Processor also can realize by the combination of calculation element, for example digital signal processor and microprocessor, a plurality of microprocessors, Digital Signal Processor Core of one or more microprocessors associatings, or any other similarly configuration realize.
Method described in the embodiment of the invention or the step of algorithm can directly embed hardware, the software module of processor execution or the two combination.Software module can be stored in the storage medium of other arbitrary form in RAM storer, flash memory, ROM storer, eprom memory, eeprom memory, register, hard disk, moveable magnetic disc, CD-ROM or this area.Exemplarily, storage medium can be connected with processor, so that processor can read information from storage medium, and can deposit write information to storage medium.Alternatively, storage medium can also be integrated in the processor.Processor and storage medium can be arranged among the ASIC, and ASIC can be arranged in the user terminal.Alternatively, processor and storage medium also can be arranged in the different parts in the user terminal.
In one or more exemplary designs, the described above-mentioned functions of the embodiment of the invention can realize in hardware, software, firmware or this three's combination in any.If realize in software, these functions can be stored on the media with computer-readable, or are transmitted on the media of computer-readable with one or more instructions or code form.The computer-readable media comprises the computer storage medium and is convenient to make allows computer program transfer to other local telecommunication media from a place.Storage medium can be the useable medium that any general or special computer can access.For example, such computer readable media can include but not limited to RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage device, or other anyly can be used for carrying or storage can be read the program code of form by general or special computer or general or special processor with instruction or data structure and other media.In addition, any connection can suitably be defined as the computer-readable media, for example, if software is by a concentric cable, optical fiber computer, twisted-pair feeder, Digital Subscriber Line (DSL) or also being comprised in the defined computer-readable media with wireless mode transmission such as for example infrared, wireless and microwaves from a web-site, server or other remote resource.Described video disc (disk) and disk (disc) comprise Zip disk, radium-shine dish, CD, DVD, floppy disk and Blu-ray Disc, and disk is usually with the magnetic duplication data, and video disc carries out the optical reproduction data with laser usually.Above-mentioned combination also can be included in the computer-readable media.
Above-described embodiment; purpose of the present invention, technical scheme and beneficial effect are further described; institute is understood that; the above only is the specific embodiment of the present invention; and be not intended to limit the scope of the invention; within the spirit and principles in the present invention all, any modification of making, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.

Claims (8)

1. a voice recognition processing and feedback system is characterized in that described system comprises:
The voice collecting unit is used for gathering user's voice information;
Voice recognition unit is used for the user's voice information of gathering is identified, and the voice messaging that identifies is converted into Word message;
Speech understanding and processing unit, described speech understanding and processing unit comprise: the literal feedback module, being used for generating ground should be in the literal feedback of described Word message; The sex detection module is for detection of user's sex; And the age detection module is for detection of user's age;
Storage unit is used for the storaged voice storehouse, and described sound bank comprises: male voice storehouse, female voice storehouse and children speech storehouse;
The speech answering unit, comprise phonetic synthesis module and speech play module, described phonetic synthesis module be used for when detected user for the male sex and when being not children, select described female voice storehouse that described literal feedback is carried out phonetic synthesis, generation female voice voice document; When detected user for the women and when being not children, select described male voice storehouse that described literal feedback is carried out phonetic synthesis, generate the male voice voice document; When detected user is children, select described children speech storehouse that described literal feedback is carried out phonetic synthesis, generate the cartoon voice document; Described speech play unit is used for playing the corresponding voice document that described phonetic synthesis module generates.
2. system according to claim 1, it is characterized in that, described phonetic synthesis module also is used for replying described user by described speech play unit after the voice document that generates adds the appellation and/or user's interest reply content that is associated with user's identity.
3. system according to claim 1 and 2, it is characterized in that, described system also comprises: mode setting unit, be used for receiving external command, and the mode of operation that described system is set is multi-lingual accent speech answering pattern, hobby object speech answering pattern or characteristic speech answering pattern;
Under described multi-lingual accent speech answering pattern, described sex detection module and described age detection module are in running order, and under other patterns, described sex detection module and described age detection module are in off position;
Described speech understanding and processing unit also comprise:
Sound characteristic extracts and synthesis module, is used for gathering the phonetic feature of the hobby object of designated user, and generating the sound bank of hobby object according to the phonetic feature of gathering under hobby object speech answering pattern;
Described phonetic synthesis module, also be used for when described user is designated user, select the sound bank of the hobby object of described designated user that described literal feedback is carried out phonetic synthesis, generate hobby object voice document, and reply described designated user by described speech play unit.
4. system according to claim 1 and 2 is characterized in that, described storage unit also stores the characteristic sound bank;
Described phonetic synthesis module also is used under characteristic speech answering pattern, selects described characteristic sound bank that described literal feedback is carried out phonetic synthesis, generates the characteristic voice document, and replys described user by described speech play unit.
5. the voice replying method of a voice recognition processing and feedback system is characterized in that described method comprises:
Gather user's voice information;
The user's voice information of gathering is identified, the voice messaging that identifies is converted into Word message;
Generating ground should be in the literal feedback of described Word message;
Detect user's sex;
Detect user's age;
Obtain and the storaged voice storehouse, described sound bank comprises: male voice storehouse, female voice storehouse and children speech storehouse;
When detected user for the male sex and when being not children, select the female voice storehouse that described literal feedback is carried out phonetic synthesis, generate the female voice voice document and reply described user; When detected user for the women and when being not children, select the male voice storehouse that described literal feedback is carried out phonetic synthesis, generate the male voice voice document and reply described user; When detected user is children, select the children speech storehouse that described literal feedback is carried out phonetic synthesis, generate the cartoon voice document and reply described user.
6. method according to claim 5 is characterized in that, described method also comprises:
After in the voice document that generates, adding the appellation and/or user's interest reply content that is associated with user's identity, reply described user.
7. according to claim 5 or 6 described methods, it is characterized in that described method also comprises:
Receive external command, the mode of operation that described system is set is multi-lingual accent speech answering pattern, hobby object speech answering pattern or characteristic speech answering pattern;
Under hobby object speech answering pattern, gather the phonetic feature of the hobby object of designated user, and generate the sound bank of hobby object according to the phonetic feature of gathering;
When described user is designated user, select the sound bank of the hobby object of described designated user that described literal feedback is carried out phonetic synthesis, generate hobby object voice document and reply described designated user.
8. according to claim 5 or 6 described methods, it is characterized in that described method also comprises:
Under characteristic speech answering pattern, obtain the characteristic sound bank;
Select described characteristic sound bank that described literal feedback is carried out phonetic synthesis according to user instruction, generate the characteristic voice document and reply described user.
CN201310095659.5A 2013-03-22 2013-03-22 Voice recognition processing and feedback system, voice replying method Expired - Fee Related CN103236259B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310095659.5A CN103236259B (en) 2013-03-22 2013-03-22 Voice recognition processing and feedback system, voice replying method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310095659.5A CN103236259B (en) 2013-03-22 2013-03-22 Voice recognition processing and feedback system, voice replying method

Publications (2)

Publication Number Publication Date
CN103236259A true CN103236259A (en) 2013-08-07
CN103236259B CN103236259B (en) 2016-06-29

Family

ID=48884295

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310095659.5A Expired - Fee Related CN103236259B (en) 2013-03-22 2013-03-22 Voice recognition processing and feedback system, voice replying method

Country Status (1)

Country Link
CN (1) CN103236259B (en)

Cited By (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104681023A (en) * 2015-02-15 2015-06-03 联想(北京)有限公司 Information processing method and electronic equipment
CN104700843A (en) * 2015-02-05 2015-06-10 海信集团有限公司 Method and device for identifying ages
CN104795067A (en) * 2014-01-20 2015-07-22 华为技术有限公司 Voice interaction method and device
CN105498168A (en) * 2015-12-07 2016-04-20 永康市玛马吉智能科技有限公司 Method and device for controlling treadmill through voices
CN105575383A (en) * 2014-10-28 2016-05-11 现代摩比斯株式会社 Apparatus and method for controlling target information voice output through using voice characteristics of user
CN105843382A (en) * 2016-03-18 2016-08-10 北京光年无限科技有限公司 Man-machine interaction method and device
CN105895105A (en) * 2016-06-06 2016-08-24 北京云知声信息技术有限公司 Speech processing method and device
WO2016167992A1 (en) * 2015-04-14 2016-10-20 Google Inc. A method and system for speech synthesis for voice queries
CN106328128A (en) * 2016-08-16 2017-01-11 成都市和平科技有限责任公司 Teaching system and method based on speech recognition technology
CN106412263A (en) * 2016-09-19 2017-02-15 合肥视尔信息科技有限公司 Human-computer interaction voice system
CN106548773A (en) * 2016-11-04 2017-03-29 百度在线网络技术(北京)有限公司 Child user searching method and device based on artificial intelligence
CN106649643A (en) * 2016-12-08 2017-05-10 腾讯音乐娱乐(深圳)有限公司 Audio data processing method and device
CN106686267A (en) * 2015-11-10 2017-05-17 中国移动通信集团公司 Method and system for implementing personalized voice service
CN107015781A (en) * 2017-03-28 2017-08-04 联想(北京)有限公司 Audio recognition method and system
CN107077315A (en) * 2014-11-11 2017-08-18 瑞典爱立信有限公司 For select will the voice used with user's communication period system and method
CN107170456A (en) * 2017-06-28 2017-09-15 北京云知声信息技术有限公司 Method of speech processing and device
CN107340991A (en) * 2017-07-18 2017-11-10 百度在线网络技术(北京)有限公司 Switching method, device, equipment and the storage medium of speech roles
CN107507620A (en) * 2017-09-25 2017-12-22 广东小天才科技有限公司 Voice broadcast sound setting method and device, mobile terminal and storage medium
CN107564532A (en) * 2017-07-05 2018-01-09 百度在线网络技术(北京)有限公司 Awakening method, device, equipment and the computer-readable recording medium of electronic equipment
CN107832720A (en) * 2017-11-16 2018-03-23 北京百度网讯科技有限公司 information processing method and device based on artificial intelligence
CN107886938A (en) * 2016-09-29 2018-04-06 中国科学院深圳先进技术研究院 Virtual reality guides hypnosis method of speech processing and device
CN108053820A (en) * 2017-12-13 2018-05-18 广东美的制冷设备有限公司 The voice broadcast method and device of air regulator
CN108154877A (en) * 2017-12-13 2018-06-12 广东美的制冷设备有限公司 The voice broadcast method and device of air regulator
CN108366302A (en) * 2018-02-06 2018-08-03 南京创维信息技术研究院有限公司 TTS broadcast commands optimization method, smart television, system and storage device
CN108364638A (en) * 2018-01-12 2018-08-03 咪咕音乐有限公司 A kind of voice data processing method, device, electronic equipment and storage medium
CN108429953A (en) * 2018-04-11 2018-08-21 四川斐讯信息技术有限公司 A kind of Foreigh-language oral-speech practice intelligent earphone and its man-machine interaction method
CN108564943A (en) * 2018-04-27 2018-09-21 京东方科技集团股份有限公司 voice interactive method and system
CN108639057A (en) * 2018-05-15 2018-10-12 上海博泰悦臻网络技术服务有限公司 Vehicle-mounted control platform and its control method
CN109002274A (en) * 2017-06-07 2018-12-14 塞舌尔商元鼎音讯股份有限公司 The method of the electronic device and adjustment output sound of adjustable output sound
CN109036377A (en) * 2018-07-26 2018-12-18 中国银联股份有限公司 A kind of phoneme synthesizing method and device
CN109147800A (en) * 2018-08-30 2019-01-04 百度在线网络技术(北京)有限公司 Answer method and device
CN109346083A (en) * 2018-11-28 2019-02-15 北京猎户星空科技有限公司 A kind of intelligent sound exchange method and device, relevant device and storage medium
CN109377979A (en) * 2018-12-07 2019-02-22 苏州思必驰信息科技有限公司 Update the method and system of welcome words
WO2019075829A1 (en) * 2017-10-17 2019-04-25 深圳市沃特沃德股份有限公司 Voice translation method and apparatus, and translation device
CN109725798A (en) * 2017-10-25 2019-05-07 腾讯科技(北京)有限公司 The switching method and relevant apparatus of Autonomous role
CN110390938A (en) * 2018-04-20 2019-10-29 比亚迪股份有限公司 Method of speech processing, device and terminal device based on vocal print
JP2020016875A (en) * 2018-07-24 2020-01-30 バイドゥ オンライン ネットワーク テクノロジー (ベイジン) カンパニー リミテッド Voice interaction method, device, equipment, computer storage medium, and computer program
CN110930999A (en) * 2018-09-19 2020-03-27 上海博泰悦臻电子设备制造有限公司 Voice interaction method and device and vehicle
CN110930998A (en) * 2018-09-19 2020-03-27 上海博泰悦臻电子设备制造有限公司 Voice interaction method and device and vehicle
CN112185344A (en) * 2020-09-27 2021-01-05 北京捷通华声科技股份有限公司 Voice interaction method and device, computer readable storage medium and processor
CN112417181A (en) * 2016-02-26 2021-02-26 原相科技股份有限公司 Interactive service platform with active response and operation method thereof
CN112489636A (en) * 2020-10-15 2021-03-12 南京创维信息技术研究院有限公司 Intelligent voice broadcast assistant selection method and system
CN113091221A (en) * 2021-03-08 2021-07-09 珠海格力电器股份有限公司 Air conditioner and control method thereof
CN113220913A (en) * 2020-02-04 2021-08-06 阿里巴巴集团控股有限公司 Multimedia file processing method, device and interactive system
WO2021218136A1 (en) * 2020-04-27 2021-11-04 平安科技(深圳)有限公司 Voice-based user gender and age recognition method and apparatus, computer device, and storage medium
CN113819602A (en) * 2021-09-06 2021-12-21 青岛海尔空调器有限总公司 Air conditioner control method and system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1409280A (en) * 2002-08-14 2003-04-09 冯鲁民 Municipal intelligent traffic and vehicle navigation system
CN1716273A (en) * 2004-06-28 2006-01-04 李剑华 Outer shape structure of commercial guest greeting robot and identifying method
CN101390155A (en) * 2006-02-21 2009-03-18 索尼电脑娱乐公司 Voice recognition with speaker adaptation and registration with pitch
JP2010078763A (en) * 2008-09-25 2010-04-08 Brother Ind Ltd Voice processing device, voice processing program, and intercom system
CN102117614A (en) * 2010-01-05 2011-07-06 索尼爱立信移动通讯有限公司 Personalized text-to-speech synthesis and personalized speech feature extraction
CN102655001A (en) * 2011-03-04 2012-09-05 东芝泰格有限公司 Information processing apparatus and method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1409280A (en) * 2002-08-14 2003-04-09 冯鲁民 Municipal intelligent traffic and vehicle navigation system
CN1716273A (en) * 2004-06-28 2006-01-04 李剑华 Outer shape structure of commercial guest greeting robot and identifying method
CN101390155A (en) * 2006-02-21 2009-03-18 索尼电脑娱乐公司 Voice recognition with speaker adaptation and registration with pitch
JP2010078763A (en) * 2008-09-25 2010-04-08 Brother Ind Ltd Voice processing device, voice processing program, and intercom system
CN102117614A (en) * 2010-01-05 2011-07-06 索尼爱立信移动通讯有限公司 Personalized text-to-speech synthesis and personalized speech feature extraction
CN102655001A (en) * 2011-03-04 2012-09-05 东芝泰格有限公司 Information processing apparatus and method

Cited By (69)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110459214A (en) * 2014-01-20 2019-11-15 华为技术有限公司 Voice interactive method and device
CN104795067A (en) * 2014-01-20 2015-07-22 华为技术有限公司 Voice interaction method and device
CN110459214B (en) * 2014-01-20 2022-05-13 华为技术有限公司 Voice interaction method and device
US11380316B2 (en) 2014-01-20 2022-07-05 Huawei Technologies Co., Ltd. Speech interaction method and apparatus
CN104795067B (en) * 2014-01-20 2019-08-06 华为技术有限公司 Voice interactive method and device
US9990924B2 (en) 2014-01-20 2018-06-05 Huawei Technologies Co., Ltd. Speech interaction method and apparatus
US10468025B2 (en) 2014-01-20 2019-11-05 Huawei Technologies Co., Ltd. Speech interaction method and apparatus
CN105575383A (en) * 2014-10-28 2016-05-11 现代摩比斯株式会社 Apparatus and method for controlling target information voice output through using voice characteristics of user
CN107077315B (en) * 2014-11-11 2020-05-12 瑞典爱立信有限公司 System and method for selecting speech to be used during communication with a user
CN107077315A (en) * 2014-11-11 2017-08-18 瑞典爱立信有限公司 For select will the voice used with user's communication period system and method
CN104700843A (en) * 2015-02-05 2015-06-10 海信集团有限公司 Method and device for identifying ages
CN104681023A (en) * 2015-02-15 2015-06-03 联想(北京)有限公司 Information processing method and electronic equipment
US10102852B2 (en) 2015-04-14 2018-10-16 Google Llc Personalized speech synthesis for acknowledging voice actions
WO2016167992A1 (en) * 2015-04-14 2016-10-20 Google Inc. A method and system for speech synthesis for voice queries
CN106686267A (en) * 2015-11-10 2017-05-17 中国移动通信集团公司 Method and system for implementing personalized voice service
CN105498168A (en) * 2015-12-07 2016-04-20 永康市玛马吉智能科技有限公司 Method and device for controlling treadmill through voices
CN112417181A (en) * 2016-02-26 2021-02-26 原相科技股份有限公司 Interactive service platform with active response and operation method thereof
CN105843382B (en) * 2016-03-18 2018-10-26 北京光年无限科技有限公司 A kind of man-machine interaction method and device
CN105843382A (en) * 2016-03-18 2016-08-10 北京光年无限科技有限公司 Man-machine interaction method and device
CN105895105A (en) * 2016-06-06 2016-08-24 北京云知声信息技术有限公司 Speech processing method and device
CN106328128A (en) * 2016-08-16 2017-01-11 成都市和平科技有限责任公司 Teaching system and method based on speech recognition technology
CN106412263A (en) * 2016-09-19 2017-02-15 合肥视尔信息科技有限公司 Human-computer interaction voice system
CN107886938B (en) * 2016-09-29 2020-11-17 中国科学院深圳先进技术研究院 Virtual reality guidance hypnosis voice processing method and device
CN107886938A (en) * 2016-09-29 2018-04-06 中国科学院深圳先进技术研究院 Virtual reality guides hypnosis method of speech processing and device
CN106548773B (en) * 2016-11-04 2020-06-23 百度在线网络技术(北京)有限公司 Child user searching method and device based on artificial intelligence
CN106548773A (en) * 2016-11-04 2017-03-29 百度在线网络技术(北京)有限公司 Child user searching method and device based on artificial intelligence
CN106649643A (en) * 2016-12-08 2017-05-10 腾讯音乐娱乐(深圳)有限公司 Audio data processing method and device
CN106649643B (en) * 2016-12-08 2019-10-22 腾讯音乐娱乐(深圳)有限公司 A kind of audio data processing method and its device
CN107015781A (en) * 2017-03-28 2017-08-04 联想(北京)有限公司 Audio recognition method and system
CN109002274A (en) * 2017-06-07 2018-12-14 塞舌尔商元鼎音讯股份有限公司 The method of the electronic device and adjustment output sound of adjustable output sound
CN107170456A (en) * 2017-06-28 2017-09-15 北京云知声信息技术有限公司 Method of speech processing and device
CN107564532A (en) * 2017-07-05 2018-01-09 百度在线网络技术(北京)有限公司 Awakening method, device, equipment and the computer-readable recording medium of electronic equipment
CN107340991B (en) * 2017-07-18 2020-08-25 百度在线网络技术(北京)有限公司 Voice role switching method, device, equipment and storage medium
CN107340991A (en) * 2017-07-18 2017-11-10 百度在线网络技术(北京)有限公司 Switching method, device, equipment and the storage medium of speech roles
US11302302B2 (en) 2017-07-18 2022-04-12 Baidu Online Network Technology (Beijing) Co., Ltd. Method, apparatus, device and storage medium for switching voice role
CN107507620A (en) * 2017-09-25 2017-12-22 广东小天才科技有限公司 Voice broadcast sound setting method and device, mobile terminal and storage medium
WO2019075829A1 (en) * 2017-10-17 2019-04-25 深圳市沃特沃德股份有限公司 Voice translation method and apparatus, and translation device
CN109725798B (en) * 2017-10-25 2021-07-27 腾讯科技(北京)有限公司 Intelligent role switching method and related device
CN109725798A (en) * 2017-10-25 2019-05-07 腾讯科技(北京)有限公司 The switching method and relevant apparatus of Autonomous role
CN107832720A (en) * 2017-11-16 2018-03-23 北京百度网讯科技有限公司 information processing method and device based on artificial intelligence
CN107832720B (en) * 2017-11-16 2022-07-08 北京百度网讯科技有限公司 Information processing method and device based on artificial intelligence
WO2019114127A1 (en) * 2017-12-13 2019-06-20 广东美的制冷设备有限公司 Voice output method and device for air conditioner
CN108154877B (en) * 2017-12-13 2021-12-17 广东美的制冷设备有限公司 Voice broadcasting method and device of air conditioner
CN108154877A (en) * 2017-12-13 2018-06-12 广东美的制冷设备有限公司 The voice broadcast method and device of air regulator
CN108053820A (en) * 2017-12-13 2018-05-18 广东美的制冷设备有限公司 The voice broadcast method and device of air regulator
CN108364638A (en) * 2018-01-12 2018-08-03 咪咕音乐有限公司 A kind of voice data processing method, device, electronic equipment and storage medium
CN108366302A (en) * 2018-02-06 2018-08-03 南京创维信息技术研究院有限公司 TTS broadcast commands optimization method, smart television, system and storage device
CN108366302B (en) * 2018-02-06 2020-06-30 南京创维信息技术研究院有限公司 TTS (text to speech) broadcast instruction optimization method, smart television, system and storage device
CN108429953A (en) * 2018-04-11 2018-08-21 四川斐讯信息技术有限公司 A kind of Foreigh-language oral-speech practice intelligent earphone and its man-machine interaction method
CN110390938A (en) * 2018-04-20 2019-10-29 比亚迪股份有限公司 Method of speech processing, device and terminal device based on vocal print
CN108564943A (en) * 2018-04-27 2018-09-21 京东方科技集团股份有限公司 voice interactive method and system
CN108564943B (en) * 2018-04-27 2021-02-12 京东方科技集团股份有限公司 Voice interaction method and system
CN108639057A (en) * 2018-05-15 2018-10-12 上海博泰悦臻网络技术服务有限公司 Vehicle-mounted control platform and its control method
JP2020016875A (en) * 2018-07-24 2020-01-30 バイドゥ オンライン ネットワーク テクノロジー (ベイジン) カンパニー リミテッド Voice interaction method, device, equipment, computer storage medium, and computer program
CN109036377A (en) * 2018-07-26 2018-12-18 中国银联股份有限公司 A kind of phoneme synthesizing method and device
CN109147800A (en) * 2018-08-30 2019-01-04 百度在线网络技术(北京)有限公司 Answer method and device
US11475897B2 (en) 2018-08-30 2022-10-18 Baidu Online Network Technology (Beijing) Co., Ltd. Method and apparatus for response using voice matching user category
CN110930999A (en) * 2018-09-19 2020-03-27 上海博泰悦臻电子设备制造有限公司 Voice interaction method and device and vehicle
CN110930998A (en) * 2018-09-19 2020-03-27 上海博泰悦臻电子设备制造有限公司 Voice interaction method and device and vehicle
CN109346083A (en) * 2018-11-28 2019-02-15 北京猎户星空科技有限公司 A kind of intelligent sound exchange method and device, relevant device and storage medium
CN109377979B (en) * 2018-12-07 2021-09-24 思必驰科技股份有限公司 Method and system for updating welcome language
CN109377979A (en) * 2018-12-07 2019-02-22 苏州思必驰信息科技有限公司 Update the method and system of welcome words
CN113220913A (en) * 2020-02-04 2021-08-06 阿里巴巴集团控股有限公司 Multimedia file processing method, device and interactive system
CN113220913B (en) * 2020-02-04 2024-07-30 阿里巴巴集团控股有限公司 Multimedia file processing method, device and interactive system
WO2021218136A1 (en) * 2020-04-27 2021-11-04 平安科技(深圳)有限公司 Voice-based user gender and age recognition method and apparatus, computer device, and storage medium
CN112185344A (en) * 2020-09-27 2021-01-05 北京捷通华声科技股份有限公司 Voice interaction method and device, computer readable storage medium and processor
CN112489636A (en) * 2020-10-15 2021-03-12 南京创维信息技术研究院有限公司 Intelligent voice broadcast assistant selection method and system
CN113091221A (en) * 2021-03-08 2021-07-09 珠海格力电器股份有限公司 Air conditioner and control method thereof
CN113819602A (en) * 2021-09-06 2021-12-21 青岛海尔空调器有限总公司 Air conditioner control method and system

Also Published As

Publication number Publication date
CN103236259B (en) 2016-06-29

Similar Documents

Publication Publication Date Title
CN103236259A (en) Voice recognition processing and feedback system, voice response method
CN110782900B (en) Collaborative AI storytelling
US11475897B2 (en) Method and apparatus for response using voice matching user category
JP6505117B2 (en) Interaction of digital personal digital assistant by replication and rich multimedia at response
US11646026B2 (en) Information processing system, and information processing method
AU2014331209B2 (en) Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method
CN108228132A (en) Promote the establishment and playback of audio that user records
KR20120038000A (en) Method and system for determining the topic of a conversation and obtaining and presenting related content
CN106599256B (en) Structure information generation method and device and interaction method and device
CN103533519A (en) Short message broadcasting method and system
CN108140030A (en) Conversational system, terminal, the method for control dialogue and the program for making computer performance conversational system function
CN110442867A (en) Image processing method, device, terminal and computer storage medium
EP3989083A1 (en) Information processing system, information processing method, and recording medium
CN109710799B (en) Voice interaction method, medium, device and computing equipment
CN111413877A (en) Method and device for controlling household appliance
US20120185417A1 (en) Apparatus and method for generating activity history
KR101791979B1 (en) Method for providing a messenger service through item representing metadata or supplemental item and server using the same
CN109478405A (en) Information processing equipment, information processing method and program
CN110660388A (en) Voice interaction point-reading device
CN109725798A (en) The switching method and relevant apparatus of Autonomous role
KR102226427B1 (en) Apparatus for determining title of user, system including the same, terminal and method for the same
KR102388465B1 (en) Virtual contents creation method
Kimura et al. Virtualizing human conversation: two case studies
Júdice et al. Elderly speech collection for speech recognition based on crowd sourcing
JP7262142B2 (en) Method for Implementing Online Media Service Equipped with Multiple Audio Systems

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20160629

Termination date: 20210322