WO1989000324A1

WO1989000324A1 - Speech aid system

Info

Publication number: WO1989000324A1
Application number: PCT/GB1988/000518
Authority: WO
Inventors: David John Calder; Paul Frank Lister; Jeremy Daniel Mckendrick Watson
Original assignee: The Secretary Of State For Health
Priority date: 1987-07-03
Filing date: 1988-07-01
Publication date: 1989-01-12
Also published as: GB8715664D0

Abstract

A system is provided to enable a vocally handicapped person to produce messages in the form of synthetized speech. A soft key method is used to provide a large vocabulary bank and to enable rapid access. A user program links screen-displayed and screen-selected vocabulary to a store for speech in coded form and an erasable and programmable builder program configures part of or all of the vocabulary displayed and selected.

Description

SPEECH AID SYSTEM

This invention relates to a dynamically configured speech aid system and in particular to a programmed computer based system which enables stored vocabulary to be selected and converted into speech. The system is particularly useful for the vocally handicapped, and also for educational purposes.

Computer based systems can provide large databanks of vocabulary, phrases, and images which a vocally handicapped person can select for input into a speech synthesizer which converts the selected input into speech.

This invention is concerned with the selection of input to a synthesizer having a store of pre-programmed speech.

The known problems of speech aids are limited vocabulary and slow utterance selection. The larger the store of vocabulary the more difficult it becomes to rapidly select the chosen word or concept. Computer based systems with their large memories and rapid processing facilities have the potential for solving each of these problems. But known systems all use hard-key entry techniques, and consequently are either cumbersome or complicated. With the advent of cheap flat panel displays, such as the LCD dot matrix type used on many portable microcomputers, the possibility of using "soft key" entry techniques has become feasible. Whereas a fixed hard key entry system is limited by the dimensions of the physical device and to one level of switching, a soft key panel can be configured to have many menu levels, all screened, that theoretically give an unlimited target number. For a speech aid with the necessary memory back-up, this means a vast vocabulary is feasible, hitherto a difficult task to engineer.

According to the present invention a speech aid system has a computer, a visual display unit, a voice synthesizer characterised by having a store for speech in coded form.a user program linking screen-displayed and screen-selected vocabulary to the stored speech in coded form, and an erasable and programable builder program for configuring part or all of the vocabulary displayed and selected. Thus the invention provides a dynamically configurable speech aid system which provides instant access to a large vocabulary yet which may be displayed or represented on, and selected from, a small screen.

It therefore is as portable as the screen size permits. Combining a programmable and erasable Builder programme with a User programme provides the therapist or tutor (or occasionally the handicapped person himself) with the flexibility to suit the vocabulary and its selection to the user's need. For example, the user may be a sportsman or woman and require an increased vocabulary in that field. Alternatively, when used as an educational tool the tutor can revise and update the vocabulary used and the way in which it is selected. The use of preprogrammed speech avoids the need for text to speech synthesis where letters are typed into the machine and stored as textual strings. The latter resulting in cheaper but poorer quality output. Further the system has the vocabulary arranged by fields and stored in the memory of a computer, and the vocabulary and fields are accessed by touching a lable on a screen displayed menu of icons or alpha-numeric characters representing fields or vocabulary, which may themselves be accessed by touching a lable on another screen displayed menu. In this way the soft key display provides a menu of labels which when "touched" reveal another menu of labels, which may in turn be "touched" to display further sets of labels. In this way, a few menu decisions by the user directs him to the likely vocabulary he will need in a given field, which vocabulary he, or his therapist, is able to amend or build-up according to his needs.

Further the synthesizer of the speech aid system may be provided by phoneme synthesis or by speech stored by the Linear Predictive Coding (LPC) method. The use of pre-programmed LPC stored speech allows tViespeech output to be chosen to be similar to that of the user's voice including intonation, accent and even dialect to produce more lifelike sounding speech. It allows a range of voice types: male, female, child and is a feature often by-passed in even the most expensive systems on the market.

By means of two spreadsheets in the user program, the fields of the speech aid system may be divided between phrases and vocabulary. This allows the user to choose at an early stage whether he wants to string words together or use stored phrases. The main advantage of this approach is that a mental image of a large spread sheet is cognitively acceptable. Where there is only one directory screen pointing down to a number of adjacent pages, the user will in a short time learn the proximity of these pages and their significance to each other. Pages containing similar contextual phrases are placed adjacent to each other. Transition between the pages may be by resetting to the directory page or bi-lateral movement across the spread sheet. Pointer keys (up, down, left, right) may be used for the latter movement and there may also be a reset key for stepping back to the directory page and if necessary further back to the mode of operation required.

The phrase-based spread sheet contains user-defined semantically-linked symbols and whole phrases or word outputs. The system may comprise a standard set, but is open to customisation. This will allow the system to become very much part of the user's personal environment and take on individual characteristics. This will help in the problem of alienation which is nearly always experienced by vocally handicapped people when first trying to adapt to a new speech aid. This system adapts the machine to the user, as opposed to the reverse.

The vocabulary bank, associated with the second spread sheet, offers a large number of individual words. These may be chosen from a basic English vocabulary, for example that devised by C K Ogden. Any simple communication can be made with this "basic" vocabulary. Whilst this vocabulary would not normally be open to alteration and would be resident in the proposed system at all times, one or two pages may be left "open" to accommodate names and personalised data as all the available spread sheets area will not be covered by the basic vocabulary. The system may also be provided with a screen indicator of the field selected provided on the menu of vocabulary. This is designed to assist orientation, and to remind the user which field he has selected. A further feature is that the number of labels on the screen of the system is variable. This would normally be between 2, 4, 8, 16 or 32. This flexibility is designed to suit possible field breakdowns, eg where the field naturally breaks down into for example 4 alternatives. It also allows the programme to be set to suit the user's ability to accurately touch the right lable. Thus larger label blocks will be needed for shakier hands.

The size and shape of the labels on the screen of the system is also variable. This allows the builder to emphasise the importance of certain labels with regard to others, priority vocabulary (such as

'I want' ) would have larger labels. The shape of the labels may also have significance in user cognition. The colour of the labels may also be used to impart a given significance.

The erasable and programmable element of the builder programme in the system may be provided with a means of preventing unauthorised access. Thus the user is prevented from accidentally altering or erasing the Builder programme. Although, clearly anyone who needed to reconfigure the programme would know how to overcome the block. Furthermore the representations of the vocabulary and fields displayed on the screen may be represented by icons. The use of such images would be to suit the users cognitive abilities, and may be particularly useful in the educational field.

An embodiment of the invention will now be described by way of example only, and with reference to the accompanying drawings and tables!

Fig 1 A flowchart to illustrate the path of the outer shell of the User Programme.

Fig 2 Illustrates the path of the inner shell of the User Programme. Fig 3 Illustrates the outer shell of the Builder programme. Fig 4 Illustrates the path of the inner shell of the Builder Programme.

Fig 5 A diagrammatic representation of t soft key method. Table 1 corresponds to Fig 1 in a step-by-step tabular form. Table 2 corresponds to Fig 2 in a step-by-step tabular form. Table 3 corresponds to Fig 3 in a step-by-step tabular form. Table 4 corresponds to Fig 4 in a step-by-step tabular form. The speech aid system is operated by 2 distinct software programmes: firstly, the Builder programme, which provides the flexible programmability: secondly, the User programme, derived from the Builder programme but not amendable by the user (handicapped person) . THE USER PROGRAM

Figures 1 and 2 and tables 1 and 2 indicate the path of the program in a clockwise direction. The concept of an inner and outer shell is used for simplicity. The outer shell handles the introductory screens when the system is started and also the upper level reset requests when another spreadsheet is required or there is a change in the way the speech has to be output; ie concatenated or single word/phrase output on each key press.

Figure 1 and table 1 show that after two choices have been made (ie which spreadsheet and speech output method) and two soft keys have been touched, the program drops down to the inner shell. Pressing the reset soft key will bring the programme path up to the outer shell once again. This choice of spreadsheet and mode of output may be by-passed by setting controls in the BUILDER program. A patient who may be/confused by a top'control' screen, has therefore only to cope with /the inner shell sequence described below. Choice of operational mode and spreadsheet is therefore in the hands of the speech therapist. The capabilities of the patient will determine whether or not this feature is used. Figure 2 and table 2 the inner shell representation, is far more complex and illustrates the main function of the user program. Entry is from the top as in the other diagram. The directory record is automatically called and displayed on the screen. Two procedures occur; the plotting of the key shapes according to the key size and number held in the directory record and the labelling of each key.

The system waits for a key entry from the user and prompts by stating on the bottom line: PRESS LABEL OF PAGE REQUIRED. The user may wish to reset or choose a page key. Assuming a page is chosen with a different key number to the directory screen. The display must be cleared completely and reconstructed with that page's information. If the page format is the same as that of the directory, then it is quicker and less tiring for the user if only the symbols on the keys are changed. If the format is the same a small asterisk appears at the corner of the key previously activated indicating the position of that page on the spreadsheet directory. This is a very useful tool developed as a need to improve orientation. The point at the base of the circle is where the spreadsheet pages are "read". A number of things could happen at this point. If the symbol keys in groups of 4, 8, 16 or 32 are activated, then the voice output routine is called. Depending on the chosen mode of speech; immediate (single entry) or stored (concatenated) the system will produce speeach. If it is stored speech then the user can keep pressing speech keys and even move across to other pages and add to the store until the "speech" soft key is activated whereupon the whole string will be uttered. Another option are the four directional keys UP, DOWN, LEFT, RIGHT. These call the associated adjacent page record, clear the screen completely if necessary and fill the screen once again. The two other keys which can also be used are the "reset" key and the "speech" key as mentioned above. A reset will take us back to the entry point of this shell.

Linear Predictive Coding offers the quality of speech that is as close to an original recording as possible. It allows a range of voice types; male, female and child and regional accent. This is a requirement that is often bypassed in even the most expensive machines on the market. The TI Speech Lab is being used for developing a range of voices for use in the field. As well as LPC speech, phoneme synthesis may be used in some circumstances.

The User program is significantly simpler to use then the Builder program. Although it uses a similar layout, no alphanumeric entries are required from the user. THE BUILDER PROGRAM

Refer to Figures 3 and 4 and tables 3 and 4. Figure 3 is set out in a similar manner to Figure 1 where an outer shell is shown " interacting with an inner cycle of events. These, (inner shell) are shown in Figure 4 and table 4.

On start up, the touch fields on the screen must first be set to read touch input across the relevant key areas. Various options are possible: trigger on approach, trigger on exit, time delay before sensing (eg for a patient with motor disfunction) or software averaging (again to cope with tremmor etc) . Out test programs trigger mainly on entry and produce a reference "click" along with an inverse video pulsed flash of the key area that has registered. This procedure also occurs on the User program.

Next, a message enquiries whether the operation is merely to change the position of an existing page on the spreadsheet or change the pages themselves. If the answer is YES, registered by keying the YES soft key, the program moves to a small terminal routine which carries out this operation. The results are then filed on disc and the program ends. If a page content alteration is necessary, the program has to decide whether to change all the pages (a start from scratch) or just alter certain page/s without destroying the rest of the spreadsheet. This is established by asking the user the enter Yes/No to the statement "start from scratch?" at the base of the screen. If the answer is positive, we move down to the inner cycle of page building. If negative, we have to give the program the page we want to change. This is a number. Pages are numbered from the top left to the right bottom of the spreadsheet. The formats are as follows:

So if we want to change an eight page spreadsheet (assuming we set up the directory as eight). at the bottom left page, we are dealing with page Five. "Starting from Scratch" automatically enters the first page construction as the directory page. This is automatically set as page 33 by the system. The system also overrides the routines that normally request speech addresses when all other pages are set up. The outer shell sequence is completed when the records are filed onto external file (floppy or hard disc in the present example).

Figure 4 covers the page building operation. Each page is a record and this is gradually filled with the required information then placed into an array. The sequence starts with the user entering the number of keys required for the page under construction. On the basis of this information, the screen display is constructed with 4, 8, 16 and 32 Keys. The program then loops through the sequences on the centre ring of Figure 4. A flashing cursor indicates the key to be filled and automatically cycles from left to right and down the page. A jump is made by the cursor to the base of the page after each symbol entry. The address of the speech string is then entered and the cursor then moves back to the next key in sequence.

This operation is bypassed if the directory screen is being constructed. Voice output is not required from this screen and hence the cursor jumps straight on to the next key after symbol entry. After a page has been filled with symbols, the request is made for positional reference; page to thftleft, right, up and down. Some versions of the software do this automatically but then the flexibility of the package becomes jeopardised. It is probably more satisfactory to choose your own adjacent pages and very easy to set up.

Special note should be made of the filtering software that prevents the entry of symbols in the keys running over the edge. A "32" size key will take 9 characters before buzzing an alarm and refusing the entry. The other sizes take progressively more characters before sounding the alarm. ICONS

The programs function in the same manner as in the BUILDER/USER concept described earlier. The difference being that instead of choosing alpha-numeric key symbols at the 'building' stage, pictures are drawn directly onto the screen. The scheme allows a picture to be constructed full size on the screen, no matter what the size or number of keys there will be per page. The builder program requests a choice of.4, 8, 16 or 32 keys per page. Once this has been done the program presents a blank screen and a unique flashing cursor to indicate that the system is waiting for the drawing to commence.

Points are drawn on the screen with a finger or any blunt impli ent such as a capped pen or pencil. The cursor follows the path taken and joins each point with a line. A pulldown control menu offers 3 control options. E for erase, gives control of backtracking and redrawing defined portions. S for stop, indicates that a line ends and another is to start. F for finish, clears the screen and the text cursor waits at the bottom of the blank screen for the associated phrase code as in the conventional alpha-numeric software. The program then indicates the number of the next key and the cursor waits for the next picture to be drawn. This process is repeated automatically for the number of keys required. At this point, either an exit can be made, or another page can be constructed. If an exit is made, the page worked on is saved onto disk.

The USER programme will automatically reduce the full picture size to the right dimensions for 4, 8, 16 or 32 components per screen. An optional extra to this icon builder program is a transparent touch screen chart. As each location has a code reference, points are entered as numbers. This allows great accuracy when copying drawings and in this instance, screen touch is by-passed.

Figure 5 is an exploded view of the USER program described by figures 1 and 2 and tables 1 and 2. The screen display 10 corresponds to STEP 8 of Table 1. If a VOCABULARY or PHRASE is chosen then the appropriate directory 11 is displayed on the screen and the program enters the inner shell (STEP 14 of Table 1). If a page is chosen (STEP 4 of Table 2) the page 12 is displayed from which the user choses the desired word or phrase by moving across the screen and pressing the appropriate key (STEP 9 of Table 2). The word or phrase is then chosen from the address store 13 which enters a speech processor 14 and a voice output 15. Similarly, if SINGLE or CASCADE is chosen from screen 10, directory 16 is displayed, from which a spreadsheet 17 is chosen and displayed. The string/single output is accessed from the address store 18, fed to a speech processor 19 and speech output 20.

It can be seen that if the directory screens and spreadsheet screens have the maximum 32 keys this offers a maximum target access per spreadsheet of 1024 individual words or phrases. Whilst the above described system incorporates all the functions on soft keys displayed on the screen, hard key labels could be provided on the side of the screen of the computer to cater for 'permanent' features such as volume control, on/off switch, direction pointers, and the reset label. Although the invention has been described by way of example and possible embodiments thereof, it is to be understood that modifications or improvements may be made without departing from the scope of the invention as defined - the claims hereafter. TABLE 1

STEP 1 'Introductory screen presentation'

Two keys displayed on screen, VOCABULARY and PHRASE STEP 2 Read touch keys STEP 3 Load VOCABULARY spreadsheet or PHRASE spreadsheet from external file STEP 4 Clearscreen

STEP 5 Two new keys displayed, CASCADE and SINGLE STEP 6 Read touch keys STEP 7 Enter inner shell (Fig 2) STEP 8 'Reset Presentation'

Four keys displayed on screen, VOCABULARY and PHRASE at top of screen, CASCADE and SINGLE at base of screen. STEP 9 Read touch keys

STEP 10 If choice is CASCADE or SINGLE, enter inner shell (Fig 2) STEP 12 If choice is VOCABULARY or PHRASE, load appropriate spreadsheet STEP 13 Read touch keys STEP 14 Enter inner shell (Fig 2)

TABLE 2

STEP 1 Display directory page: two routines called: one plots the keys, the other fills them with symbols.

Read touch Ic^β-ys.

EITHER reset and exit to outer shell (Fig 1)

OR page chosen, so continue

Clearscreen if new page has different number of keys else remove symbols only

Display new keys if screen was cleared

Fill key areas with page symbol

Read touch keys

Window shift across spreadsheet, clearscreen if necessary. Load adjacent page onto screen. Four page options - UP, DOWN, LEFT,

RIGHT STEP 10* Voice output routine called by up to 32 symbol keys STEP 11 Reset key activated

* If cascade mode was chosen, then an array is filled with each speech address until the "speech output" key is touched. The whole string is then output at once.

Set up touch fields on screen

Do you want to alter adjacent page numbers?

YES. What is the page? New reference UP, DOWN, LEFT, RIGHT, file records to disc then STOP

NO

Start from scratch?

YES, Enter inner shell (Fig 3)

NO

What is the page from which to start (up to 32 pages and one directory screen)

Enter inner shell (Fig 3)

Clearscreen

Another page?

YES, Enter inner shell NO

File records onto external file (Disc .etc) STOP

TABLE 4

Enter the number of keys required: 4, 8, 16, 32

STEP 1 STEP 2 Construct keys images on screen STEP 3 Enter loop STEP 4 Position cursor offset (key) STEP 5 Read Symbol STEP 6 Place cursor at base of page STEP 7 Read speech address number STEP 8 STEP 9 Loop 4, 8, 16 or 32 times STEP 10 Exit Loop STEP 11 Enter page number left STEP 12 Enter page number right STEP 13 Adjacent pages UP, DOWN, RIGHT, LEFT STEP 14 Enter page number up STEP 15 Enter page number down

Claims

1. A speech aid system having computer ^a visual display unit, a voice synthesizer characterised by having a store for speech in coded form a user programme linking screen-displayed and screen-selected vocabulary to the stored speech in coded form and an erasable and programmable builder programme for configuring part of or all of the vocabulary displayed and selected.

2. A speech aid system as previously claimed characterised in that the vocabulary is arranged by fields and is stored in the memory of a computer and the vocabulary and fields are accessed by touching a label on a screen displayed menu of representations of the fields or vocabulary, which may themselves be accessed by touching a label on another screen displayed menu.

3. A speech aid system as claimed in claim 2 characterised in that the output speech is stored in Linear Predictive Coding form.

4. A speech aid system as claimed in claim 2 characterised in that the output speech is provided by phoneme synthesis.

5. A speech aid system as claimed in claim 2 in which the fields are divided between phrases and vocabulary.

6. A speech aid system as claimed in claim 5 characterised in that an indicator of the field selected is provided on the menu of vocabulary.

7. A speech aid system as claimed in claim characterised in that the number of labels on the screen of the system is variable.

8. A speech aid system as claimed in claim characterised in that the size and shape of the labels on the screen of the system is variable.

9. A speech aid system as claimed in claim $ characterised in that the colour of the labels is variable.

10. A speech aid system as claimed in claim *^~\ characterised in that the erasable and programmable element of the builder programme in the system is provided with a means of preventing unauthorised access.

11. A.speech aid system as claimed in claim lOcharacterised in that hard key labels are provided on the side of the screen of the computer.

12: A speech aid system as claimed in claim M characterised in that the representations of the covabulary and fields is displayed on the screen as alpha-numeric characters.

13. A speech aid system as claimed in claim II characterised in that the representations of the vocabulary and fields is displayed on the screen as icons.

SUBSTITUTESHEET