Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20040083104 A1
Publication typeApplication
Application numberUS 10/685,478
Publication date29 Apr 2004
Filing date16 Oct 2003
Priority date17 Oct 2002
Also published asUS7292977, US7389229, US7424427, US20040083090, US20040138894, US20040163034, US20040172250, US20040176946, US20040204939, US20040230432, US20050038649
Publication number10685478, 685478, US 2004/0083104 A1, US 2004/083104 A1, US 20040083104 A1, US 20040083104A1, US 2004083104 A1, US 2004083104A1, US-A1-20040083104, US-A1-2004083104, US2004/0083104A1, US2004/083104A1, US20040083104 A1, US20040083104A1, US2004083104 A1, US2004083104A1
InventorsDaben Liu, Francis Kubala
Original AssigneeDaben Liu, Kubala Francis G.
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Systems and methods for providing interactive speaker identification training
US 20040083104 A1
Abstract
A system (100) provides speaker identification training. The system (100) generates speaker models and receives audio segments. The system (100) identifies speakers corresponding to the audio segments based on the speaker models. At least one of the audio segments has an unidentified or misidentified speaker (i.e., an audio segment whose speaker cannot be accurately identified). The system (100) presents, to a user, audio segments that include an audio segment whose speaker is unidentified or misidentified and receives, from the user, the name of the unidentified or misidentified speaker. The system (100) may use this information to subsequently identify the unidentified or misidentified speaker by name for future audio segments.
Images(9)
Previous page
Next page
Claims(26)
What is claimed is:
1. A speaker identification system, comprising:
an indexer configured to:
generate a plurality of speaker models,
receive a plurality of audio segments, and
identify speakers corresponding to the audio segments based on the speaker models, the indexer being unable to correctly identify at least one of the speakers, as an unidentified speaker, corresponding to the audio segments; and
a server configured to:
receive, from a user, the name of the unidentified speaker, and
provide the name of the unidentified speaker to the indexer for identification of the unidentified speaker in subsequent audio segments.
2. The system of claim 1, wherein the indexer is further configured to:
generate a new speaker model for the unidentified speaker based on an audio segment corresponding to the unidentified speaker.
3. The system of claim 1, wherein the indexer is further configured to:
generate labels for the audio segments, the labels being based on names of speakers that can be identified and gender of speakers that cannot be identified.
4. The system of claim 3, wherein when receiving the name of an unidentified speaker, the server is configured to:
present a document to the user, the document including a transcription of a plurality of the audio segments and the labels for the plurality of the audio segments, and
receive, from the user, the name of one of the speakers that cannot be identified.
5. The system of claim 4, wherein when presenting a document, the server is further configured to:
provide audio data corresponding to at least one of the plurality of the audio segments to the user.
6. The system of claim 1, wherein the server is further configured to:
locate one or more additional audio segments from the unidentified speaker, and
present the one or more additional audio segments to the user for confirmation that the one or more additional audio segments were produced by the unidentified speaker.
7. The system of claim 6, wherein when presenting the one or more additional audio segments, the server is configured to continue to present audio segments to the user for confirmation until at least four minutes of audio data is obtained.
8. The system of claim 6, wherein the unidentified speaker corresponds to at least one of the audio segments; and
wherein when locating one or more additional audio segments, the server is configured to:
find one or more additional audio segments similar to the at least one of the audio segments.
9. The system of claim 1, wherein the indexer is configured to:
fit audio data from the unidentified speaker to a speaker independent Gaussian mixture model using an expectation and maximization process, and
generate a new speaker model for the unidentified speaker using a maximum a posteriori adaptation process.
10. The system of claim 1, wherein the unidentified speaker is a misidentified speaker of one of the audio segments; and
wherein when receiving the name, the server is configured to:
receive, from the user, a correct name of a speaker of the one of the audio segments.
11. The system of claim 10, wherein the indexer is further configured to:
identify one of the speaker models, as an identified speaker model, that corresponds to the one of the audio segments, and
update a label associated with the identified speaker model to include the correct name of the speaker of the one of the audio segments.
12. The system of claim 10, wherein the server is further configured to:
locate one or more additional audio segments similar to the one of the audio segments, and
present the one or more additional audio segments to the user for confirmation that the one or more additional audio segments were produced by the speaker of the one of the audio segments.
13. A speaker identification system, comprising:
means for generating a plurality of speaker models;
means for receiving a plurality of audio segments;
means for identifying speakers corresponding to the audio segments based on the speaker models, at least one of the audio segments being associated with an unidentified or misidentified speaker;
means for labeling the audio segments with names of the speakers that can be identified;
means for presenting a plurality of the audio segments, including the at least one of the audio segments, with the labels to a user;
means for receiving, from the user, the name of the unidentified or misidentified speaker; and
means for identifying the unidentified or misidentified speaker by name in future audio segments.
14. A method for providing speaker identification training, comprising:
generating a plurality of speaker models;
receiving a plurality of audio segments;
identifying speakers corresponding to the audio segments based on the speaker models, at least one of the audio segments being associated with an unidentified or misidentified speaker;
presenting a plurality of the audio segments, including the at least one of the audio segments, to a user;
receiving, from the user, the name of the unidentified or misidentified speaker; and
identifying the unidentified or misidentified speaker by name for future audio segments.
15. The method of claim 14, wherein the unidentified or misidentified speaker is an unidentified speaker; and
wherein the method further comprises:
generating a new speaker model for the unidentified speaker based on the at least one of the audio segments.
16. The method of claim 14, further comprising:
generating labels for the audio segments, the labels being based on names of speakers that can be identified and gender of speakers that cannot be identified.
17. The method of claim 16, wherein the presenting a plurality of the audio segments includes:
providing a document to the user, the document including a transcription of the plurality of the audio segments and the labels for the plurality of the audio segments.
18. The method of claim 17, wherein the providing a document includes:
providing audio data corresponding to one or more of the plurality of the audio segments to the user.
19. The method of claim 14, further comprising:
locating one or more additional audio segments from the unidentified or misidentified speaker, and
presenting the one or more additional audio segments to the user for confirmation that the one or more additional audio segments were produced by the unidentified or misidentified speaker.
20. The method of claim 19, wherein the presenting the one or more additional audio segments includes:
presenting audio segments to the user for confirmation until at least four minutes of audio data is obtained.
21. The method of claim 19, wherein the locating one or more additional audio segments includes:
finding one or more additional audio segments similar to the at least one of the audio segments.
22. The method of claim 14, wherein the unidentified or misidentified speaker is an unidentified speaker; and
wherein the method further comprises:
fitting audio data from the unidentified speaker to a speaker independent Gaussian mixture model using an expectation and maximization process; and
generating a new speaker model for the unidentified speaker using a maximum a posteriori adaptation process.
23. The method of claim 14, wherein the unidentified or misidentified speaker is a misidentified speaker; and
wherein the receiving the name includes:
receiving, from the user, a correct name of a speaker of the at least one of the audio segments.
24. The method of claim 23, further comprising:
identifying one of the speaker models, as an identified speaker model, that corresponds to the at least one of the audio segments; and
updating a label associated with the identified speaker model to include the correct name of the speaker of the at least one of the audio segments.
25. A computer-readable medium that stores instructions executable by one or more processors for speaker identification training by a speaker identification system, comprising:
instructions for generating a plurality of speaker models based on training data;
instructions for presenting, to a user, audio segments for which no speakers can be identified from the speaker models;
instructions for obtaining, from the user, a name of a speaker for at least one of the audio segments;
instructions for generating a new speaker model for the speaker based on the at least one of the audio segments; and
instructions for associating the name of the speaker with the new speaker model.
26. A speaker identification system, comprising:
an indexer configured to:
receive a plurality of speech segments, each of the speech segments being associated with a corresponding speaker,
create a plurality of documents by transcribing the speech segments,
identify names of the speakers corresponding to the speech segments, the indexer being unable to correctly identify names of at least one of the speakers corresponding to the speech segments, the speakers for which the indexer can correctly identify names being identified speakers and the speakers for which the indexer cannot correctly identify names being unidentified speakers;
a database configured to store the documents; and
a server configured to:
retrieve one or more of the documents from the database,
present the one or more of the documents to a user,
receive, from the user, a name for one of the unidentified speakers, and
provide the name for the one of the unidentified speakers to the indexer for subsequent identification of speech segments from the one of the unidentified speakers.
Description
    RELATED APPLICATION
  • [0001]
    This application claims priority under 35 U.S.C. 119 based on U.S. Provisional Application No. 60/419,214, filed Oct. 17, 2002, the disclosure of which is incorporated herein by reference.
  • [0002]
    This application is related to U.S. patent application, Ser. No. 10/______ (Docket No. 02-4042), entitled “Continuous Learning for Speech Recognition Systems,” filed concurrently herewith, and U.S. patent application, Ser. No. 10/610,533 (Docket No. 02-4046), entitled “Systems and Methods for Improving Recognition Results via User-Augmentation of a Database,” filed Jul. 2, 2003, the disclosures of which are incorporated herein by reference.
  • BACKGROUND OF THE INVENTION
  • [0003]
    1. Field of the Invention
  • [0004]
    The present invention relates generally to multimedia environments and, more particularly, to systems and methods for providing interactive speech identification training in a multimedia environment.
  • [0005]
    2. Description of Related Art
  • [0006]
    Conventional speaker identification systems require a huge amount of training data to identify speakers based on audio data received from the speakers. These systems typically train a set of models based on the training data. When audio data is received by one of these systems, the system segments the audio data into speaker turns. The system then analyzes each of the speaker turns to determine whether it matches one of the models. If a speaker turn matches one of the models, the system labels the speaker turn with the name of the speaker.
  • [0007]
    For proper speaker identification, the system requires a huge amount of training data. For each speaker, for example, the system requires several minutes of speech to generate a Gaussian mixture model that can later be used to locate segments of speech from the same speaker. In some situations, it is difficult to obtain a sufficient amount of training data to generate an accurate Gaussian mixture model. For example, it is difficult to obtain sufficient training data from speakers who do not routinely speak or who speak in only small bursts (i.e., one to two sentences at a time).
  • [0008]
    As a result, there is a need for mechanisms to improve speaker identification results even for speakers where there is insufficient training data.
  • SUMMARY OF THE INVENTION
  • [0009]
    Systems and methods consistent with the present invention collect data for training a speaker identification system in an interactive manner. Users may be prompted to identify speaker turns for unidentified or incorrectly identified speakers. The data from the users may be used to generate new speaker models and/or update links to existing speaker models. The speaker models may then be used to correctly label speaker turns.
  • [0010]
    In one aspect consistent with the principles of the invention, a system provides speaker identification training. The system generates speaker models and receives audio segments. The system identifies speakers corresponding to the audio segments based on the speaker models. At least one of the audio segments has an unidentified or misidentified speaker (i.e., an audio segment whose speaker cannot be accurately identified). The system presents, to a user, audio segments that include an audio segment whose speaker is unidentified or misidentified and receives, from the user, the name of the unidentified or misidentified speaker. The system may use this information to subsequently identify the unidentified or misidentified speaker by name for future audio segments.
  • [0011]
    In another aspect consistent with the principles of the invention, a speaker identification system is provided. The system includes an indexer and a server. The indexer is configured to generate speaker models, receive audio segments, and identify speakers corresponding to the audio segments based on the speaker models. The indexer is unable to correctly identify at least one of the speakers corresponding to the audio segments. The server is configured to receive, from a user, the name of an unidentified speaker of the speakers corresponding to the audio segments, and provide the name of the unidentified speaker to the indexer for identification of the unidentified speaker in subsequent audio segments.
  • [0012]
    In a further aspect consistent with the principles of the invention, a computer-readable medium stores instructions executable by one or more processors for speaker identification training by a speaker identification system. The computer-readable medium includes instructions for generating speaker models based on training data; instructions for presenting, to a user, audio segments for which no speakers can be identified from the speaker models; instructions for obtaining, from the user, a name of a speaker for at least one of the audio segments; instructions for generating a new speaker model for the speaker based on the at least one audio segment; and instructions for associating the name of the speaker with the new speaker model.
  • [0013]
    In another aspect consistent with the principles of the invention, a speaker identification system includes an indexer, a database, and a server. The indexer receives speech segments, where each of the speech segments has a corresponding speaker. The indexer also creates documents by transcribing the speech segments and identifies the names of the speakers corresponding to the speech segments. The indexer is unable to correctly identify the names of at least one of the speakers corresponding to the speech segments. As a result, some of the speakers are identified speakers and others of the speakers are unidentified speakers. The database stores the documents. The server retrieves one or more of the documents from the database and presents the one or more of the documents to a user. The server receives, from the user, the name of one of the unidentified speakers and provides the name of the unidentified speaker to the indexer for subsequent identification of speech segments from the unidentified speaker.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0014]
    The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the invention and, together with the description, explain the invention. In the drawings,
  • [0015]
    [0015]FIG. 1 is a diagram of a system in which systems and methods consistent with the present invention may be implemented;
  • [0016]
    [0016]FIG. 2 is an exemplary diagram of the indexer of FIG. 1 according to an implementation consistent with the principles of the invention;
  • [0017]
    [0017]FIG. 3 is an exemplary diagram of a portion of the recognition system of FIG. 2 according to an implementation consistent with the present invention;
  • [0018]
    [0018]FIG. 4 is an exemplary diagram of the memory system of FIG. 1 according to an implementation consistent with the principles of the invention;
  • [0019]
    FIGS. 5A-5C are flowcharts of exemplary processing for speaker identification training according to an implementation consistent with the principles of the invention; and
  • [0020]
    [0020]FIG. 6 is a diagram of an exemplary graphical user interface by which a document may be presented to a user according to an implementation consistent with the principles of the invention.
  • DETAILED DESCRIPTION
  • [0021]
    The following detailed description of the invention refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Also, the following detailed description does not limit the invention. Instead, the scope of the invention is defined by the appended claims and equivalents.
  • [0022]
    Systems and methods consistent with the present invention permit users to aid in the training of a speaker identification system by, for example, correctly identifying unidentified and misidentified speakers. The systems and methods may use the user-identification of unidentified speakers to train new speaker models for improved recognition results. The systems and methods may use the user-identification of misidentified speakers to update links to existing speaker models.
  • EXEMPLARY SYSTEM
  • [0023]
    [0023]FIG. 1 is a diagram of an exemplary system 100 in which systems and methods consistent with the present invention may be implemented. System 100 may include multimedia (MM) sources 110, indexer 120, memory system 130, and server 140 connected to clients 150 via network 160. Network 160 may include any type of network, such as a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a public telephone network (e.g., the Public Switched Telephone Network (PSTN)), a virtual private network (VPN), or a combination of networks. The various connections shown in FIG. 1 may be made via wired, wireless, and/or optical connections.
  • [0024]
    Multimedia sources 110 may include one or more audio sources and/or one or more video sources. An audio source may include any source of audio data, such as radio, telephone, and conversations in any language. A video source may include any source of video data with integrated audio data in any language, such as television, satellite, and a camcorder. The audio and/or video data may be provided to indexer 120 as a stream or file.
  • [0025]
    Indexer 120 may include mechanisms for processing audio and/or video data. Indexer 120 may include mechanisms that receive data from multimedia sources 110, process the data, perform feature extraction, and output analyzed, marked-up, and enhanced language metadata. In one implementation consistent with the principles of the invention, indexer 120 includes mechanisms, such as the ones described in John Makhoul et al., “Speech and Language Technologies for Audio Indexing and Retrieval,” Proceedings of the IEEE, Vol. 88, No. 8, August 2000, pp. 1338-1353, which is incorporated herein by reference.
  • [0026]
    [0026]FIG. 2 is an exemplary diagram of indexer 120 according to an implementation consistent with the principles of the invention. Indexer 120 may include training system 210, statistical model 220, and recognition system 230. Training system 210 may include logic that estimates parameters of statistical model 220 from a corpus of training data. The training data may initially include human-produced data. For example, the training data might include one hundred hours of audio data that has been meticulously and accurately transcribed by a human. Training system 210 may use the training data to generate parameters for statistical model 220 that recognition system 230 may later use to recognize future data that it receives (i.e., new audio that it has not heard before).
  • [0027]
    The training data might also include audio data for which the speaker has been identified. Training system 210 may use the training data to generate parameters for statistical speaker models that recognition system 230 may later use to recognize speakers from future data that it receives. To build a speaker model for a new speaker, training system 210 may use a speaker independent Gaussian mixture model with approximately 2,048 Gaussians. This Gaussian mixture model may be trained based on a lot of diversified speakers (e.g., speakers of different ages, different genders, etc.). Training system 210 may use a conventional expectation and maximization process to fit audio from a speaker to the model. Training system 210 may then use a conventional maximum a posteriori adaptation process to generate a final model for the speaker.
  • [0028]
    Statistical model 220 may include acoustic models, language models, and speaker models. The acoustic models may describe the time-varying evolution of feature vectors for each sound or phoneme. The acoustic models may employ continuous hidden Markov models (HMMs) to model each of the phonemes in the various phonetic contexts.
  • [0029]
    The language models may include n-gram language models, where the probability of each word is a function of the previous word (for a bi-gram language model) and the previous two words (for a tri-gram language model). Typically, the higher the order of the language model, the higher the recognition accuracy at the cost of slower recognition speeds.
  • [0030]
    The speaker models may include a pool of models that are used to identify speakers from their speech. Speech from a particular speaker may be compared to the speaker models to determine the likelihood that the speech was produced from each of the models. The model with the highest likelihood may be determined to correspond to that speech. The particular speaker corresponding to the model may be determined based, for example, on a mapping function that links speaker names to speaker models.
  • [0031]
    Recognition system 230 may use statistical model 220 to process input audio data. FIG. 3 is an exemplary diagram of a portion of recognition system 230 according to an implementation consistent with the principles of the invention. Recognition system 230 may include speaker segmentation logic 310, speech recognition logic 320, speaker clustering logic 330, and speaker identification logic 340. In one implementation consistent with the principles of the invention, the functions performed by speaker segmentation logic 310, speech recognition logic 320, speaker clustering logic 330, and speaker identification logic 340 are similar to the functions described in John Makhoul et al., “Speech and Language Technologies for Audio Indexing and Retrieval,” Proceedings of the IEEE, Vol. 88, No. 8, August 2000, pp. 1338-1353, which was previously incorporated herein by reference.
  • [0032]
    Generally, speaker segmentation logic 310 may distinguish speech from silence, noise, and other audio signals in input audio data. For example, speaker segmentation logic 310 may analyze each thirty second window of the input data to determine whether it contains speech. Speaker segmentation logic 310 may also identify boundaries between speakers in the input stream. Speaker segmentation logic 310 may group speech segments from the same speaker and send the segments to speech recognition logic 320.
  • [0033]
    Speech recognition logic 320 may perform continuous speech recognition to recognize the words spoken in the segments that it receives from speaker segmentation logic 310. Speech recognition logic 320 may generate a transcription of the speech using statistical model 220. Speaker clustering logic 330 may identify all of the segments from the same speaker in a single document (i.e., a body of media that is contiguous in time (from beginning to end or from time A to time B)) and group them into speaker clusters (or speaker turns). Speaker clustering logic 330 may then assign each of the speaker turns a unique label.
  • [0034]
    Speaker identification logic 340 may identify the speaker in each speaker turn by name or, when the name cannot be determined, by gender. Speaker identification logic 340 may use the speaker models within statistical model 220 to identify speakers from their speech. For example, speaker identification logic 340 may compare speech from a particular speaker within a speaker turn to the speaker models to determine the likelihood that the speech was produced from each of the models. Speaker identification logic 340 may identify the model with the highest likelihood (above some threshold) as the model corresponding to that speech. Speaker identification logic 340 may include a mapping function that links speaker names to the speaker models so that speaker identification logic 340 can identify the speaker by name once a model has been identified.
  • [0035]
    Returning to FIG. 1, memory system 130 may store documents from indexer 120 and, possibly, documents from clients 150. FIG. 4 is an exemplary diagram of memory system 130 according to an implementation consistent with the principles of the invention. Memory system 130 may include loader 410, trainer 420, one or more databases 430, and interface 440. Loader 410 may include logic that receives documents from indexer 120 and stores them in database 430. Trainer 420 may include logic that sends speaker identification information, such as audio data with identification of the corresponding speaker, in the form of training data to indexer 120.
  • [0036]
    Database 430 may include a conventional database, such as a relational database, that stores documents from indexer 120. Database 430 may also store documents received from clients 150 via server 140. Interface 440 may include logic that interacts with server 140 to store documents in database 130, query or search database 130, and retrieve documents from database 130.
  • [0037]
    Returning to FIG. 1, server 140 may include a computer (e.g., a processor and memory) or another device that is capable of interacting with memory system 130 and clients 150 via network 170. Server 140 may receive queries from clients 150 and use the queries to retrieve relevant documents from memory system 130.
  • [0038]
    Clients 150 may include personal computers, laptops, personal digital assistants, or other types of devices that are capable of interacting with server 140 to retrieve documents from memory system 130. Clients 150 may present information to users via a graphical user interface, such as a web browser window.
  • EXEMPLARY PROCESSING
  • [0039]
    Systems and methods consistent with the present invention permit users to assist in speaker identification training to improve recognition results of system 100. For example, the user may supply the name of an unidentified or misidentified speaker that may be used to retrain indexer 120.
  • [0040]
    FIGS. 5A-5C are flowcharts of exemplary processing for speaker identification training according to an implementation consistent with the principles of the invention. Processing may begin with a user desiring to retrieve one or more documents from memory system 130. The user may use a conventional web browser of client 150 to access server 140 in a conventional manner. To obtain documents of interest, the user may generate a search query and send the query to server 140 via client 150. Server 140 may use the query to search memory system 130 and retrieve relevant documents.
  • [0041]
    Server 140 may present the relevant documents to the user. For example, the user may be presented with a list of relevant documents. The documents may include any combination of audio documents and video documents. The user may select one or more documents on the list to view. When this happens, the user may be presented with a transcription corresponding to the audio data from the document. The user may also be presented with the audio and/or video data corresponding to the transcription.
  • [0042]
    [0042]FIG. 6 is a diagram of an exemplary graphical user interface (GUI) 600 by which a document may be presented to a user according to an implementation consistent with the principles of the invention. In one implementation, GUI 600 is part of an interface of a standard Internet browser, such as Internet Explorer or Netscape Navigator, or any browser that follows World Wide Web Consortium (W3C) specifications for HTML.
  • [0043]
    GUI 600 may include a speaker section 610 and a transcription section 620. Speaker section 610 may identify boundaries between speakers, the gender of a speaker, and the name of a speaker (when known). In this way, speaker segments are clustered together over the entire document to group together segments from the same speaker under the same label. In the example of FIG. 6, one speaker, Elizabeth Vargas, has been identified by name. Other speakers have been identified by gender and number. For example, if “male 1” speaks again within the same document, his speech will also be labeled “male 1.” Transcription section 620 may include a transcription of the document. In the example of FIG. 6, the document corresponds to video data from a television broadcast of ABC's World News Tonight.
  • [0044]
    GUI 600 may also include playback button 630. The user may select playback button 630 to hear the audio (or possibly see the video) corresponding to the transcription in transcription section 620. To facilitate this, the user may select a portion of the transcription in transcription section 620 by, for example, using a mouse to highlight the portion. The user may then select playback button 630 to hear the audio (or see the video) corresponding to the selected portion of the transcription.
  • [0045]
    GUI 600 may further include a correction button 640. The user may select correction button 640 when the user desires to supply the name of an unidentified speaker or correct the name of a misidentified speaker within speaker section 610. Sometimes, a speaker may be identified only by gender in speaker section 610 because indexer 120 (FIG. 1) can identify the speaker's gender but cannot determine the speaker's name. This may occur when indexer 120 has not yet generated a speaker model corresponding to the speaker due, for example, to a lack of audio data for the speaker. Sometimes, a speaker may be misidentified (i.e., labeled with the incorrect speaker name) in speaker section 610. This may occur when indexer 120 incorrectly matches the audio data from the speaker to a speaker model of another speaker. If the user desires, the user may provide the name of an unidentified speaker or correct the name of a misidentified speaker by selecting correction button 640 and providing the correct information.
  • [0046]
    GUI 600 may receive the information provided by the user and modify the document onscreen. This way, the user may determine whether the information was correctly provided. GUI 600 may also send the speaker identification information to server 140.
  • [0047]
    Returning to FIG. 5A, the user may express a desire to provide the name of an unidentified speaker or correct the name of a misidentified speaker. In doing so, the user may identify a segment, such as a speaker turn, within transcription section 620 (FIG. 6) (act 505). The user may identify the speaker turn by selecting the corresponding segment using, for example, a mouse or by selecting one of the labels within speaker section 610. The user may then select correction button 640 to initiate the correction.
  • [0048]
    Client 150 may determine whether the user has selected to identify the name of an unidentified speaker or correct the name of a misidentified speaker (act 510). Client 150 may make this determination based on the current speaker label provided in speaker section 610 for the segment identified by the user. For example, if the current speaker label includes a gender identification, then client 150 may determine that the user has selected to identify the name of an unidentified speaker. If, on the other hand, the current speaker label includes something other than a gender identification, then client 150 may determine that the user has selected to correct the name of a misidentified speaker.
  • [0049]
    If the user desires to provide the name of an unidentified speaker, the user may input the name in any conventional manner (act 515) (FIG. 5B). Client 150 may provide the name and, possibly, other identifying information, as speaker identification information, to server 140. The other identifying information may include, for example, an indication of the audio segment to which the name corresponds, the audio (or video) data corresponding to the segment, and/or other information that may be useful for training indexer 120 to later identify audio data from this same speaker.
  • [0050]
    Server 140 may determine whether additional audio data is necessary or desired (act 520). For example, server 140 may determine whether the amount of audio data associated with the audio segment labeled by the user is less than approximately four minutes of audio. As explained above, the minimum amount of audio data needed to build a good Gaussian mixture model (with over 2,000 Gaussians) is approximately four minutes of audio.
  • [0051]
    If the audio data associated with the audio segment is less than four minutes in length, then server 140 may locate additional audio segments that may be from the same speaker (act 525). Server 140 may identify these additional audio segments based on the speaker clustering performed by indexer 120. Using the speaker clustering, more, similar audio segments may be identified in the same document and in other documents within memory system 130.
  • [0052]
    For each audio segment located, sever 140 may confirm with the user that the audio segment is from the same speaker identified by the user (act 530). Server 140 may continue to obtain additional audio segments and confirm them with the user until it obtains around four minutes or more of audio data or the user expresses a desire to cease the confirmations.
  • [0053]
    Once server 140 collects a sufficient amount of audio data or the user stops the confirmations, server 140 may provide the audio data, along with the name of the speaker, to indexer 120 as training data. Indexer 120 may use the audio data to build a speaker model for the new speaker (act 535). As described above, indexer 120 may use a speaker independent Gaussian mixture model with approximately 2,048 Gaussians. This Gaussian mixture model may be trained based on a lot of diversified speakers (e.g., speakers of different ages, different genders, etc.). Indexer 120 may use a conventional expectation and maximization process to fit the audio data from the speaker to the model. Indexer 120 may then use a conventional maximum a posteriori adaptation process to generate a final model for the speaker.
  • [0054]
    Indexer 120 may add the speaker model to the pool of speaker models within statistical model 220 (FIG. 2) (act 540). Thereafter, indexer 120 may correctly identify new audio data that it receives corresponding to the new speaker.
  • [0055]
    If, instead of providing the name of an unidentified speaker, the user desires to correct the name of a misidentified speaker, the user may optionally input the correct name in any conventional manner (act 545) (FIG. 5C). Client 150 may provide the correct name and, possibly, other identifying information, as speaker identification information, to server 140. The other identifying information may include, for example, an indication of the segment (e.g., speaker turn) to which the name corresponds, the audio (or video) data corresponding to the segment, and/or other information that may be useful for indexer 120 to identify the speaker model corresponding to the speaker.
  • [0056]
    Server 140 may determine whether confirmation of additional audio data is necessary or desired (act 550). For example, a certain amount of audio data may be needed to accurately identify the incorrectly labeled speaker model. In this case, server 140 may identify other audio segments that are close to the current segment (i.e., the speaker turn that the user identified as mislabeled) (act 555). Server 140 may use information relating to the speaker clustering performed by indexer 120 to identify these other potentially mislabeled audio segments. For each segment located, sever 140 may confirm with the user that the speaker turn is also labeled incorrectly (act 560). Server 140 may continue to obtain additional audio segments and confirm them with the user until it until some threshold is met or the user expresses a desire to cease the confirmations.
  • [0057]
    Server 140 may then provide the audio data, along with the correct name of the speaker, to indexer 120. Indexer 120 may use the audio data to identify the speaker model from which the audio data was produced. Indexer 120 may then update the name associated with the identified speaker model (act 565). For example, a mapping may exist between the names of speakers and the corresponding speaker models in statistical model 220. Indexer 120 may update the mapping to identify the correct name of the speaker for that model.
  • [0058]
    In an alternate implementation, the user knows only that the speaker is misidentified, but does not know the true identity of the speaker. For example, the user may indicate simply that the name of the speaker associated with the speaker turn is incorrect. In this case, indexer 120 may simply remove the mapping of the incorrect speaker name to the corresponding speaker model.
  • [0059]
    When indexer 120 thereafter comes across audio data corresponding to a speaker that has been identified by the user (either as a result of an unidentified or misidentified speaker turn), indexer 120 compares the audio data to the models in the speaker model pool. Indexer 120 may determine the likelihood that the audio data was created by each of the speaker models. Indexer 120 may identify the speaker model with the highest likelihood (above some threshold) as the speaker model corresponding to the audio data. Indexer 120 may then use its mapping of speaker models to speaker names to identify the speaker of that audio data by name. Indexer 120 may also update the documents in memory system 130 with the speaker name(s) identified by the user.
  • CONCLUSION
  • [0060]
    Systems and methods consistent with the present invention provide an interactive speaker identification training system. Via these systems and methods, users are permitted to assist in the speaker identification training by, for example, correctly identifying unidentified and misidentified speakers. The systems and methods may use the user-identification of unidentified speakers to train new speaker models for improved recognition results. The systems and methods may use the user-identification of misidentified speakers to update links to existing speaker models.
  • [0061]
    The foregoing description of preferred embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention.
  • [0062]
    For example, an exemplary graphical user interface has been described with regard to FIG. 6 as containing certain features consistent with the principles of the invention. It is to be understood that a graphical user interface, consistent with the present invention, may include any or all of these features or different features to facilitate user assistance in speaker identification training.
  • [0063]
    Further, a graphical user interface has been described as performing certain functions. It is to be understood that some, if not all, of these functions may be performed by client 150 or server 140.
  • [0064]
    While series of acts have been described with regard to FIGS. 5A-5C, the order of the acts may differ in other implementations consistent with the principles of the invention.
  • [0065]
    Further, certain portions of the invention have been described as “logic” that performs one or more functions. This logic may include hardware, such as an application specific integrated circuit or a field programmable gate array, software, or a combination of hardware and software.
  • [0066]
    No element, act, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. The scope of the invention is defined by the claims and their equivalents.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4879648 *19 Sep 19867 Nov 1989Nancy P. CochranSearch system which continuously displays search terms during scrolling and selections of individually displayed data sets
US4908866 *4 Feb 198513 Mar 1990Eric GoldwasserSpeech transcribing system
US5317732 *26 Apr 199131 May 1994Commodore Electronics LimitedSystem for relocating a multimedia presentation on a different platform by extracting a resource map in order to remap and relocate resources
US5404295 *4 Jan 19944 Apr 1995Katz; BorisMethod and apparatus for utilizing annotations to facilitate computer retrieval of database material
US5418716 *26 Jul 199023 May 1995Nec CorporationSystem for recognizing sentence patterns and a system for recognizing sentence patterns and grammatical cases
US5544257 *8 Jan 19926 Aug 1996International Business Machines CorporationContinuous parameter hidden Markov model approach to automatic handwriting recognition
US5559875 *31 Jul 199524 Sep 1996Latitude CommunicationsMethod and apparatus for recording and retrieval of audio conferences
US5572728 *27 Dec 19945 Nov 1996Hitachi, Ltd.Conference multimedia summary support system and method
US5613032 *2 Sep 199418 Mar 1997Bell Communications Research, Inc.System and method for recording, playing back and searching multimedia events wherein video, audio and text can be searched and retrieved
US5614940 *21 Oct 199425 Mar 1997Intel CorporationMethod and apparatus for providing broadcast information with indexing
US5684924 *19 May 19954 Nov 1997Kurzweil Applied Intelligence, Inc.User adaptable speech recognition system
US5715367 *23 Jan 19953 Feb 1998Dragon Systems, Inc.Apparatuses and methods for developing and using models for speech recognition
US5752021 *18 May 199512 May 1998Fuji Xerox Co., Ltd.Document database management apparatus capable of conversion between retrieval formulae for different schemata
US5757960 *28 Feb 199726 May 1998Murdock; Michael ChaseMethod and system for extracting features from handwritten text
US5768607 *30 Apr 199616 Jun 1998Intel CorporationMethod and apparatus for freehand annotation and drawings incorporating sound and for compressing and synchronizing sound
US5777614 *13 Oct 19957 Jul 1998Hitachi, Ltd.Editing support system including an interactive interface
US5787198 *25 Oct 199428 Jul 1998Lucent Technologies Inc.Text recognition using two-dimensional stochastic models
US5806032 *14 Jun 19968 Sep 1998Lucent Technologies Inc.Compilation of weighted finite-state transducers from decision trees
US5835667 *14 Oct 199410 Nov 1998Carnegie Mellon UniversityMethod and apparatus for creating a searchable digital video library and a system and method of using such a library
US5862259 *27 Mar 199619 Jan 1999Caere CorporationPattern recognition employing arbitrary segmentation and compound probabilistic evaluation
US5875108 *6 Jun 199523 Feb 1999Hoffberg; Steven M.Ergonomic man-machine interface incorporating adaptive pattern recognition based control system
US5960447 *13 Nov 199528 Sep 1999Holt; DouglasWord tagging and editing system for speech recognition
US5963940 *14 Aug 19965 Oct 1999Syracuse UniversityNatural language information retrieval system and method
US5970473 *31 Dec 199719 Oct 1999At&T Corp.Video communication device providing in-home catalog services
US6006184 *28 Jan 199821 Dec 1999Nec CorporationTree structured cohort selection for speaker recognition system
US6006221 *14 Aug 199621 Dec 1999Syracuse UniversityMultilingual document retrieval system and method using semantic vector matching
US6024571 *24 Apr 199715 Feb 2000Renegar; Janet ElaineForeign language communication system/device and learning aid
US6029124 *31 Mar 199822 Feb 2000Dragon Systems, Inc.Sequential, nonparametric speech recognition and speaker identification
US6029195 *5 Dec 199722 Feb 2000Herz; Frederick S. M.System for customized electronic identification of desirable objects
US6052657 *25 Nov 199718 Apr 2000Dragon Systems, Inc.Text segmentation and identification of topic using language models
US6064963 *17 Dec 199716 May 2000Opus Telecom, L.L.C.Automatic key word or phrase speech recognition for the corrections industry
US6067514 *23 Jun 199823 May 2000International Business Machines CorporationMethod for automatically punctuating a speech utterance in a continuous speech recognition system
US6067517 *2 Feb 199623 May 2000International Business Machines CorporationTranscription of speech data with segments from acoustically dissimilar environments
US6073096 *4 Feb 19986 Jun 2000International Business Machines CorporationSpeaker adaptation system and method based on class-specific pre-clustering training speakers
US6076053 *21 May 199813 Jun 2000Lucent Technologies Inc.Methods and apparatus for discriminative training and adaptation of pronunciation networks
US6088669 *28 Jan 199711 Jul 2000International Business Machines, CorporationSpeech recognition with attempted speaker recognition for speaker model prefetching or alternative speech modeling
US6112172 *31 Mar 199829 Aug 2000Dragon Systems, Inc.Interactive searching
US6151598 *4 Dec 199721 Nov 2000Shaw; Venson M.Digital dictionary with a communication system for the creating, updating, editing, storing, maintaining, referencing, and managing the digital dictionary
US6161087 *5 Oct 199812 Dec 2000Lernout & Hauspie Speech Products N.V.Speech-recognition-assisted selective suppression of silent and filled speech pauses during playback of an audio recording
US6169789 *4 Jun 19992 Jan 2001Sanjay K. RaoIntelligent keyboard system
US6185531 *9 Jan 19986 Feb 2001Gte Internetworking IncorporatedTopic indexing method
US6219640 *6 Aug 199917 Apr 2001International Business Machines CorporationMethods and apparatus for audio-visual speaker recognition and utterance verification
US6253179 *29 Jan 199926 Jun 2001International Business Machines CorporationMethod and apparatus for multi-environment speaker verification
US6308222 *30 Nov 199923 Oct 2001Microsoft CorporationTranscoding of audio data
US6332139 *8 Nov 199918 Dec 2001Mega Chips CorporationInformation communication system
US6332147 *3 Nov 199518 Dec 2001Xerox CorporationComputer controlled display system using a graphical replay device to control playback of temporal data representing collaborative activities
US6345252 *9 Apr 19995 Feb 2002International Business Machines CorporationMethods and apparatus for retrieving audio information using content and speaker information
US6347295 *26 Oct 199812 Feb 2002Compaq Computer CorporationComputer method and apparatus for grapheme-to-phoneme rule-set-generation
US6360234 *14 Aug 199819 Mar 2002Virage, Inc.Video cataloger system with synchronized encoders
US6360237 *5 Oct 199819 Mar 2002Lernout & Hauspie Speech Products N.V.Method and system for performing text edits during audio recording playback
US6373985 *12 Aug 199816 Apr 2002Lucent Technologies, Inc.E-mail signature block analysis
US6381640 *19 Feb 199930 Apr 2002Genesys Telecommunications Laboratories, Inc.Method and apparatus for automated personalization and presentation of workload assignments to agents within a multimedia communication center
US6434520 *16 Apr 199913 Aug 2002International Business Machines CorporationSystem and method for indexing and querying audio archives
US6437818 *5 May 199820 Aug 2002Collaboration Properties, Inc.Video conferencing on existing UTP infrastructure
US6463444 *14 Aug 19988 Oct 2002Virage, Inc.Video cataloger system with extensibility
US6480826 *31 Aug 199912 Nov 2002Accenture LlpSystem and method for a telephonic emotion detection that provides operator feedback
US6567980 *14 Aug 199820 May 2003Virage, Inc.Video cataloger system with hyperlinked output
US6571208 *29 Nov 199927 May 2003Matsushita Electric Industrial Co., Ltd.Context-dependent acoustic models for medium and large vocabulary speech recognition with eigenvoice training
US6602300 *3 Sep 19985 Aug 2003Fujitsu LimitedApparatus and method for retrieving data from a document database
US6604110 *31 Oct 20005 Aug 2003Ascential Software, Inc.Automated software code generation from a metadata-based repository
US6611803 *14 Dec 199926 Aug 2003Matsushita Electric Industrial Co., Ltd.Method and apparatus for retrieving a video and audio scene using an index generated by speech recognition
US6624826 *28 Sep 199923 Sep 2003Ricoh Co., Ltd.Method and apparatus for generating visual representations for audio documents
US6647383 *1 Sep 200011 Nov 2003Lucent Technologies Inc.System and method for providing interactive dialogue and iterative search functions to find information
US6654735 *8 Jan 199925 Nov 2003International Business Machines CorporationOutbound information analysis for generating user interest profiles and improving user productivity
US6708148 *9 Oct 200216 Mar 2004Koninklijke Philips Electronics N.V.Correction device to mark parts of a recognized text
US6711541 *7 Sep 199923 Mar 2004Matsushita Electric Industrial Co., Ltd.Technique for developing discriminative sound units for speech recognition and allophone modeling
US6714911 *15 Nov 200130 Mar 2004Harcourt Assessment, Inc.Speech transcription and analysis system and method
US6718303 *13 May 19996 Apr 2004International Business Machines CorporationApparatus and method for automatically generating punctuation marks in continuous speech recognition
US6718305 *17 Mar 20006 Apr 2004Koninklijke Philips Electronics N.V.Specifying a tree structure for speech recognizers using correlation between regression classes
US6728673 *9 May 200327 Apr 2004Matsushita Electric Industrial Co., LtdMethod and apparatus for retrieving a video and audio scene using an index generated by speech recognition
US6732183 *4 May 20004 May 2004Broadware Technologies, Inc.Video and audio streaming for multiple users
US6748350 *27 Sep 20018 Jun 2004Intel CorporationMethod to compensate for stress between heat spreader and thermal interface material
US6748356 *7 Jun 20008 Jun 2004International Business Machines CorporationMethods and apparatus for identifying unknown speakers using a hierarchical tree structure
US6778958 *30 Aug 200017 Aug 2004International Business Machines CorporationSymbol insertion apparatus and method
US6792409 *19 Dec 200014 Sep 2004Koninklijke Philips Electronics N.V.Synchronous reproduction in a speech recognition system
US6847961 *12 Nov 200225 Jan 2005Silverbrook Research Pty LtdMethod and system for searching information using sensor with identifier
US6877134 *29 Jul 19995 Apr 2005Virage, Inc.Integrated data and real-time metadata capture system and method
US6922691 *21 Apr 200226 Jul 2005Emotion, Inc.Method and apparatus for digital media management, retrieval, and collaboration
US6961954 *2 Mar 19981 Nov 2005The Mitre CorporationAutomated segmentation, information extraction, summarization, and presentation of broadcast news
US6973428 *24 May 20016 Dec 2005International Business Machines CorporationSystem and method for searching, analyzing and displaying text transcripts of speech after imperfect speech recognition
US6999918 *20 Sep 200214 Feb 2006Motorola, Inc.Method and apparatus to facilitate correlating symbols to sounds
US7131117 *4 Sep 200231 Oct 2006Sbc Properties, L.P.Method and system for automating the analysis of word frequencies
US7171360 *7 May 200230 Jan 2007Koninklijke Philips Electronics N.V.Background learning of speaker voices
US7257528 *13 Feb 199814 Aug 2007Zi Corporation Of Canada, Inc.Method and apparatus for Chinese character text input
US20010026377 *19 Mar 20014 Oct 2001Katsumi IkegamiImage display system, image registration terminal device and image reading terminal device used in the image display system
US20010051984 *27 Jan 199713 Dec 2001Toshihiko FukasawaCoordinative work environment construction system, method and medium therefor
US20020001261 *20 Apr 20013 Jan 2002Yoshinori MatsuiData playback apparatus
US20020010575 *2 Aug 200124 Jan 2002International Business Machines CorporationMethod and system for the automatic segmentation of an audio stream into semantic or syntactic units
US20020010916 *12 Feb 200124 Jan 2002Compaq Computer CorporationApparatus and method for controlling rate of playback of audio data
US20020049589 *27 Jun 200125 Apr 2002Poirier Darrell A.Simultaneous multi-user real-time voice recognition system
US20020059204 *10 Jul 200116 May 2002Harris Larry R.Distributed search system and method
US20020133477 *5 Mar 200119 Sep 2002Glenn AbelMethod for profile-based notice and broadcast of multimedia content
US20020184373 *21 Mar 20025 Dec 2002International Business Machines CorporationConversational networking via transport, coding and control conversational protocols
US20030051214 *6 Aug 200213 Mar 2003Ricoh Company, Ltd.Techniques for annotating portions of a document relevant to concepts of interest
US20030088414 *7 May 20028 May 2003Chao-Shih HuangBackground learning of speaker voices
US20030093580 *9 Nov 200115 May 2003Koninklijke Philips Electronics N.V.Method and system for information alerts
US20030167163 *31 Jul 20024 Sep 2003Nec Research Institute, Inc.Inferring hierarchical descriptions of a set of documents
US20040073444 *16 Nov 200115 Apr 2004Li Li PehMethod and apparatus for a financial database structure
US20050060162 *12 Jun 200117 Mar 2005Farhad MohitSystems and methods for automatic identification and hyperlinking of words or other data items and for information retrieval using hyperlinked words or data items
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US8121432 *25 Mar 200821 Feb 2012International Business Machines CorporationSystem and method for semantic video segmentation based on joint audiovisual and text analysis
US8630854 *31 Aug 201014 Jan 2014Fujitsu LimitedSystem and method for generating videoconference transcriptions
US87919775 Oct 201029 Jul 2014Fujitsu LimitedMethod and system for presenting metadata during a videoconference
US9305553 *28 Apr 20115 Apr 2016William S. MeiselSpeech recognition accuracy improvement through speaker categories
US9754593 *4 Nov 20155 Sep 2017International Business Machines CorporationSound envelope deconstruction to identify words and speakers in continuous speech
US20080065380 *12 Mar 200713 Mar 2008Kwak Keun ChangOn-line speaker recognition method and apparatus thereof
US20080175556 *25 Mar 200824 Jul 2008Chitra DoraiSystem and method for semantic video segmentation based on joint audiovisual and text analysis
US20090119103 *10 Oct 20087 May 2009Franz GerlSpeaker recognition system
US20110295603 *28 Apr 20111 Dec 2011Meisel William SSpeech recognition accuracy improvement through speaker categories
US20120053936 *31 Aug 20101 Mar 2012Fujitsu LimitedSystem and Method for Generating Videoconference Transcriptions
US20130144414 *6 Dec 20116 Jun 2013Cisco Technology, Inc.Method and apparatus for discovering and labeling speakers in a large and growing collection of videos with minimal user effort
US20140136204 *22 Oct 201315 May 2014GM Global Technology Operations LLCMethods and systems for speech systems
US20140244257 *25 Feb 201328 Aug 2014Nuance Communications, Inc.Method and Apparatus for Automated Speaker Parameters Adaptation in a Deployed Speaker Verification System
Classifications
U.S. Classification704/245
International ClassificationG06F17/00, G06F17/20, G06F17/21, G10L11/00, G10L15/00, G10L13/08, G10L15/14
Cooperative ClassificationG10L15/28, G10L15/32
Legal Events
DateCodeEventDescription
16 Oct 2003ASAssignment
Owner name: BBNT SOLUTIONS LLC, TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIU, DABEN;KUBALA, FRANCIS;REEL/FRAME:014608/0707
Effective date: 20031010
12 May 2004ASAssignment
Owner name: FLEET NATIONAL BANK, AS AGENT, MASSACHUSETTS
Free format text: PATENT & TRADEMARK SECURITY AGREEMENT;ASSIGNOR:BBNT SOLUTIONS LLC;REEL/FRAME:014624/0196
Effective date: 20040326
Owner name: FLEET NATIONAL BANK, AS AGENT,MASSACHUSETTS
Free format text: PATENT & TRADEMARK SECURITY AGREEMENT;ASSIGNOR:BBNT SOLUTIONS LLC;REEL/FRAME:014624/0196
Effective date: 20040326
2 Mar 2006ASAssignment
Owner name: BBN TECHNOLOGIES CORP., MASSACHUSETTS
Free format text: MERGER;ASSIGNOR:BBNT SOLUTIONS LLC;REEL/FRAME:017274/0318
Effective date: 20060103
Owner name: BBN TECHNOLOGIES CORP.,MASSACHUSETTS
Free format text: MERGER;ASSIGNOR:BBNT SOLUTIONS LLC;REEL/FRAME:017274/0318
Effective date: 20060103
2 Aug 2007ASAssignment
Owner name: BBNT SOLUTIONS LLC, MASSACHUSETTS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIU, DABEN;KUBALA, FRANCIS G.;REEL/FRAME:019648/0422
Effective date: 20031010
27 Oct 2009ASAssignment
Owner name: BBN TECHNOLOGIES CORP. (AS SUCCESSOR BY MERGER TO
Free format text: RELEASE OF SECURITY INTEREST;ASSIGNOR:BANK OF AMERICA, N.A. (SUCCESSOR BY MERGER TO FLEET NATIONAL BANK);REEL/FRAME:023427/0436
Effective date: 20091026