CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from U.S. Provisional Application Ser. No. 60/180,344, filed Feb. 4, 2000 entitled Voice-Activated Information Retrieval System and U.S. Provisional Application Ser. No. 60/233,068, filed Sep. 15, 2000 entitled Robust Voice Browser System and Voice Activated Device Controller.

FIELD OF THE INVENTION

The present invention relates to a robust and highly reliable system that allows users to browse web sites and retrieve information by using conversational voice commands. Additionally, the present invention allows users to control and monitor other systems and devices that are connected the Internet or any other network by using voice commands.

BACKGROUND OF THE INVENTION

Currently, three options exist for a user who wishes to gather information from a web site accessible over the Internet. The first option is to use a desktop or a laptop computer connected to a telephone line via a modem or connected to a network with Internet access. The second option is to use a Personal Digital Assistant (PDA) that has the capability of connecting to the Internet either through a modem or a wireless connection. The third option is to use one of the newly designed web-phones or web-pagers that are now being offered on the market. Although each of these options provide a user with access to the Internet to browse web sites, each of them have their own drawbacks.

Desktop computers are very large and bulky and are difficult to transport. Laptop computers solve this inconvenience, but many are still quite heavy and are inconvenient to carry. Further, laptop computers cannot be carried and used everywhere a user travels. For instance, if a user wishes to obtain information from a remote location where no electricity or communication lines are installed, it would be nearly impossible to use a laptop computer Oftentimes, information is needed on an immediate basis where a computer is not accessible. Furthermore, the use of laptop or desktop computers to access the Internet requires either a direct or a dial-up connection tan an Internet Service Provider (ISP). Oftentimes, such connections are not available when a user desires to connect to the Internet to acquire information.

The second option for remotely accessing web sites is the use of PDAs These devices also have their own set of drawbacks. First, PDAs with the ability to connect to the Internet and access web sites are not readily available. As a result, these PDAs tend to be very expensive. Furthermore, users are usually required to pay a special service fee to enable the web browsing feature of the PDA. A further disadvantage of these PDAs is that web sites must be specifically designed to allow these devices to access information on the web site. Therefore, a limited number of web sites are available that are accessible by these web-enabled PDAs. Finally, it is very common today for users to carry cell phones, however, users must also carry a separate PDA if they require the ability to gather information from various web sites Users are therefore subjected to added expenses since they must pay for both cellular telephone service and also for the web-enabling service for the PDA. This results in a very expensive alternative for the consumer.

The third alternative mentioned above is the use of web-phones or web-pagers. These devices suffer many of the same drawbacks as PDAs. First, these devices are expensive to purchase. Further, the number of web sites accessible to these devices is limited since web sites must be specifically designed to allow access by these devices. Furthermore, users are often required to pay an additional fee in order to gain wireless web access Again, this service is expensive. Another drawback of these web-phones or web-pagers is that as technology develops, the methods used by the various web sites to allow access by these devices may change. These changes may require users to purchase new web-phones or web-pagers or have the current device serviced in order to upgrade the firmware or operating system stored within the device. At the least, this would be inconvenient to users and may actually be quite expensive.

Therefore, a need exists for a system that allows users to easily access and browse the Internet from any location. Such a system would only require users to have access to any type of telephone and would not require users to subscribe to multiple services.

In the rapidly changing area of Internet applications, web sites change frequently. The design of the web site may change, the information required by the web site in order to perform searches may change, and the method of reporting search results may change. Web browsing applications that submit search requests and interpret responses from these web sites based upon expected formats will produce errors and useless responses when such changes occur. Therefore, a need exists for a system that can detect modifications to web sites and adapt to such changes in order to quickly and accurately provide the information requested by a user through a voice enabled device, such as a telephone.

When users access web sites using devices such as personal computers, delays in receiving responses are tolerated and are even expected, however, such delays are not expected when a user communicates with a telephone. Users expect communications over a telephone to occur immediately with a minimal amount of delay time A user attempting to find information using a telephone expects immediate responses to his search requests. A system that introduces too much delay between the time a user makes a request and the time of response will not be tolerated by users and will lose its usefulness. Therefore, it is important that a voice browsing system that uses telephonic communications selects web sites that provide rapid responses since speed is an important factor for maintaining the system's desirability and usability. Therefore, a need exists for a system that accesses web sites based upon their speed of operation.

SUMMARY OF THE INVENTION

It is an object of an embodiment of the present invention to allow users to gather information from web sites by using voice enabled devices, such as wireline or wireless telephones.

An additional object of an embodiment of the present invention is to provide a system and method that allows the searching and retrieving of publicly available information by controlling a web browsing server using naturally spoken voice commands.

It is an object of another embodiment of the present invention to provide a robust voice browsing system that can obtain the same information from several web sites based upon a ranking order. The ranking order is automatically adjusted if the system detects that a given web site is not functioning, is too slow, or has been modified in such a way that the requested information cannot be retrieved any longer.

A still further object of an embodiment of the present invention is to allow users to gather information from web sites from any location where a telephonic connection can be made.

Another object of an embodiment of the present invention is to allows users to browse web sites on the Internet using conversational voice commands spoken into wireless or wireline telephones or other voice enabled devices.

An additional object an embodiment of the present invention is to provide a system and method for using voice commands to control and monitor devices connected to a network.

It is an object of an embodiment of the present invention to provide a system and method which allows devices connected to a network to be controlled by conversational voice commands spoken into any voice enabled device interconnected with the same network.

The present invention relates to a system for acquiring information from sources on a network, such as the Internet. A voice browsing system maintains a database containing a list of information sources, such as web sites, connected to a network. Each of the information sources is assigned a rank number which is listed in the database along with the record for the information source. In response to a speech command received from a user, a network interface system accesses the information source with the highest rank number in order to retrieve information requested by the user.

The a preferred embodiment of the present invention allows users to access and browse web sites when they do not have access to computers with Internet access. This is accomplished by providing a voice browsing system and method that allows users to browse web sites using conversational voice commands spoken into any type of voice enabled device (i.e., any type of wireline or wireless telephone, IP phone, wireless PDA, or other wireless device). These spoken commands are then converted into data messages by a speech recognition software engine running on a user interface system. These data messages are then sent to and processed by a network interface system. This network interface system then generates the proper requests that are transmitted to the desired web site over the Internet. Responses sent from the web site are received and processed by the network interface system and then converted into an audio message via a speech synthesis engine or a pre-recorded audio concatenation application and finally transmitted to the user's voice enabled device.

A preferred embodiment of the voice browser system and method uses a web site polling and ranking methodology that allows the system to detect changes in web sites and adapt to those changes in real-time. This enables the voice browser system of a preferred embodiment to deliver highly reliable information to users over any voice enabled device. This ranking system also enables the present invention to provide rapid responses to user requests. Long delays before receiving responses to requests are not tolerated by users of voice-based systems, such as telephones When a user speaks into a telephone, an almost immediate response is expected This expectation does not exist for non-voice communications, such as email transmissions or accessing a web site using a personal computer. In such situations, a reasonable amount of transmission delay is acceptable. The ranking system of implemented by a preferred embodiment of the present invention ensures users will always receive the fastest possible response to their request.

An alternative embodiment of the present invention allows users to control and monitor the operation of a variety of household devices connected to a network using speech commands spoken into a voice enabled device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a depiction of the voice browsing system of the first embodiment of the present invention;

FIG. 2 is a block diagram of a database record used by the first preferred embodiment of the present invention;

FIG. 3 is a block diagram of a media server used by the preferred embodiment;

FIG. 4 is a block diagram of a web browsing server used by the preferred embodiment; and

FIG. 5 is a depiction of the device browsing system of the second embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A first embodiment of the present invention is a system and method for allowing users to browse information sources, such as web sites, by using naturally spoken, conversational voice commands spoken into a voice enabled device. Users are not required to learn a special language or command set in order to communicate with the voice browsing system of the present invention. Common and ordinary commands and phrases are all that is required for a user to operate the voice browsing system The voice browsing system recognizes naturally spoken voice commands and is speaker-independent, it does not have to be trained to recognize the voice patterns of each individual user. Such speech recognition systems use phonemes to recognize spoken words and not predefined voice patterns.

The first embodiment allows users to select from various categories of information and to search those categories for desired data by using conversational voice commands. The voice browsing system of the first preferred embodiment includes a user interface system referred to as a media server. The media server contains a speech recognition software engine. This speech recognition engine is used to recognize natural, conversational-voice commands spoken by the user and converts them into data messages based on the available recognition grammar. These data messages are then sent to a network interface system. In the first preferred embodiment, the network interface system is referred to as a web browsing server. The web browsing server then accesses the appropriate information source, such as a web site, to gather information requested by the user.

Responses received from the information sources are then transferred to the media server where speech synthesis engine converts the responses into audio messages that are transmitted to the user. A more detailed description of this embodiment will now be provided.

Referring to FIG. 1, a database 100 designed by Webley Systems Incorporated is connected to one or more web browsing servers 102 as well as to one or more media servers 106. The database may store information on magnetic media, such as a hard disk drive, or it may store information via other widely acceptable methods for storing data, such as optical disks. The database 100 contains a separate set of records for each web site accessible by the system. An example of a web site record is shown in FIG. 2. Each web site record 200 contains the rank number of the web site 202, the associated Uniform Resource Locator (URL) 204, and a command that enables the appropriate extraction agent 206 that is required in order to generate proper requests sent to and to format data received from the web site. The database record 200 also contains the timestamp 208 indicating the last time the web site was accessed. The extraction agent is described in more detail below. The database 100 categorizes each database record 200 according to the type of information provided by each web site. For instance, a first category of database records 200 may correspond to web sites that provide weather information. The database 100 may also contain a second category of records 200 for web sites that provide stock information. These categories may be further divided into subcategories. For instance, the weather category may contain subcategories depending upon type of weather information available to a user, such as current weather or extended forecast. Within the extended forecast subcategory, a list of web site records may be stored that provide weather information for multiple days. The use of subcategories may allow the web browsing feature to provide more accurate, relevant, and up-to-date information to the user by accessing the most relevant web site. The number of records contained in each category or subcategory is not limited. In the preferred embodiment, three web site records are provided for each category.

Table 1 below depicts two database records 200 that are used with the preferred embodiment. These records also contain a field indicating the category of the record, which is weather in each of these examples.

TABLE 1category: weatherURL: URLhttp://cgi.cnn.com/cgi-bin/weather/redirectzip_ziprank: 1command: web_dispatch.p1 weather_cnnbrowsingServer: wportallbrowsingServerBackup: wporta12dateTime: Dec 21 2000 2:15PMcategory: weatherURL: URLhttp://weather.lycos.com/wcfiveday.aspcityziprank: 2command: web_dispatch.p1 weather_lycosbrowsingServer: wportal1browsingServerBackup: wportal2dateTime: Dec 21 2000 1:45PM

The database also contains a listing of pre-recorded audio files used to create concatenated phrases and sentences Further, database 100 may contain customer profile information, system activity reports, and any other data or software servers necessary for the testing or administration of the voice browsing system.

The operation of the media servers 106 will now be discussed in relation to FIG. 3. The media servers 106 function as user interface systems. In the preferred embodiment, the media servers 106 contain a speech recognition engine 300, a speech synthesis engine 302, an Interactive Voice Response (IVR) application 304, a call processing system 306, and telephony and voice hardware 308 required to communicate with the Public Switched Telephone Network (PSTN) 116. In the preferred embodiment, each media server is based upon Intel's Dual Pentium III 730 MHz microprocessor system.

The speech recognition function is performed by a speech recognition engine 300 that converts voice commands received from the user's voice enabled device 112 (i.e., any type of wireline or wireless telephone, Internet Protocol (IP) phones, or other special wireless units) into data messages. In the preferred embodiment, voice commands and audio messages are transmitted using the PSTN 116 and data is transmitted using the TCP/IP communications protocol. However, one skilled in the art would recognize that other transmission protocols may be used for either voice or data. Other possible transmission protocols would include SIP/VoIP (Session Initiation Protocol/Voice over IP), Asynchronous Transfer Mode (ATM) and Frame Relay. A preferred speech recognition engine is developed by Nuance Communications of 1380 Willow Road, Menlo Park, Calif. 94025 (www.nuance.com). The Nuance engine capacity is measured in recognition units based on CPU type as defined in the vendor specification The natural speech recognition grammars (i.e., what a user can say that will be recognized by the speech recognition engine) were developed by Webley Systems.

Table 2 below provides a partial source code listing of the recognition grammars used by the speech recognition engine of the preferred embodiment for obtaining weather information

TABLE 2WHAT_IS the weather info information report conditions

( (like in )

UScities:n

<param1 $n.zip> <param2 $n.city> <param3$n. state>( (area code) AREA_CODE:n ) <param1 $n>( AREA_CODE:n (area code) ) <param1 $n>( (zip code) ZIP_CODE:n ) <param1 $n>( ZIP_CODE:n (zip code) ) <param1 $n>)

) <menu 194>

The media server 106 uses recognition results generated by the speech recognition engine 300 to retrieve a web site record 200 stored in the database 100 that can provide the information requested by the user. The media server 106 processes the recognition result data identifying keywords that are used to search the web site records 200 contained in the database 100 For instance, if the user's request was What is the weather in Chicago, the keywords weather and Chicago would be recognized. A web site record 200 with the highest rank number from the weather category within the database 100 would then be selected and transmitted to the web browsing server 102 along with an identifier indicating that Chicago weather is being requested.

The media servers 106 also contain a speech synthesis engine 302 that converts the data retrieved by the web browsing servers 102 into audio messags that are transmitted to the user's voice enabled device 112. A preferred speech synthesis engine is developed by Lernout and Hauspie Speech Products, 52 Third Avenue, Burlington, Mass. 01803 (www.lhsl.com).

A further description of the web browsing server 102 will be provided in relation to FIG. 4. The web browsing servers 102 provide access to any computer network such as the Internet 110 These servers are also capable of accessing databases stored on Local Area Networks (LANs) or Wide Area Networks (WANs). The web browsing servers receive responses from web sites and extract the data requested by the user. This task is also known as content extraction. The web browsing servers 102 also perform the task of periodically polling or pinging various web sites and modifying the ranking numbers of these web sites depending upon their response and speed. This polling feature is further discussed below. The web browsing server 102 is comprised of a content extraction agent 400, a content fetcher 402, a polling and ranking agent 404, and the content descriptor files 406. Each of these are software applications and will be discussed below.

Upon receiving a web site record 200 from the database 100 in response to a user request, the web browsing server 102 invokes the content extraction agent command 206 contained in the record 200. The content extraction agent 400 allows the web browsing server 102 to properly format requests and read responses provided by the web site 114 identified in the URL field 204 of the web site record 200. Each content extraction agent command 206 invokes the content extraction agent and identifies a content description file associated with the web page identified by the URL 204. This content description the directs the extraction agent where to extract data from the accessed web page and how to format a response to the user utilizing that data. For example, the content description for a web page providing weather information would indicate where to insert the city name or ZIP code in order to retrieve Chicago weather information. Additionally, the content description file for each supported URL indicates the location on the web page where the response information is provided. The extraction agent 400 uses this information to properly extract from the web page the information requested by the user.

Table 3 below contains source code for a content extraction agent 400 used by the preferred embodiment.

TABLE 3/usr/local/www/bin/sybper15$Header:/usr/local/cvsroot/webley/agents/service/web_dispatch.p1,v1.6 Dispatches all web requestshttp://wcorp.itn.net/cgi/flstatcarrierua&flight_no155&mon_abbrjul&date6&stampOhLNPdbuuE*itn/ord, itn/cb/sprint_hdhttp://cgi.cnnfn.com/flightview/rlmairlineamt&number300require config_tmp.p1; check parametersdie Usage: $0 service paramsn if $ARGV < 1;print STDERR ARGV; get parametersmy ( $service, param ) ARGV;

check servicemy %Services (weather_cnn >webget.p1 weather cnn,weather_lycos >webget.p1weather_lycos,weather_weather > webqet.p1weather_weather,weather_snap > webget.p1weather_snapweather infospace > webget.p1weather_infospace,stockQuote_yahoo > webget.p1 stock,fliqhtStatus_itn > webget.p1flight_delay,yellowPages_yahoo > yp_data.p1,yellowPages yahoo > yp data.p1,newsHeaders_newsreal > news.p1,newsArticle_newsreal > news.p1,

test parammy $date date;chop ( $date );my ( $short_date ) $date /s(w3sd1,2)s/;my %Test

weather_cnn > 60053,weather_lycos > 60053,weather_weather > 60053,weather_snap > 60053,weather_infospace > 60053,stockQuote_yahoo > msft,flightStatus_itn > ua 155 .

$short_date,

yellowPages_yahoo > tires 60015,newsHeaders_newsreal > 1,newsArticle_newsreal > 1 1,);

die $date: $0: error: no such service: $service (check thisscript) nunless $Services$service ; prepare absolute path to run other scriptsmy ( $path, $script ) $0 m{circumflex over ()}(.*/) ({circumflex over ()}/*) ; store the service to compare against datatablemy $service_stored $service; run servicewhile( ( $response $path$Services$service param) ) response failed check with test parameters$response $path$Services$service $Test$service; print test: $path$Services$service $Test$service;

if ( $response )

$service &switch_service( $service );print Wrong parameter values were supplied:

$service -paramn; die $date: $0: error: wrong parameters: $service-paramn;

else change priority and notify$service &increase_attempt( $service );

output the responseprint $response;sub increase_attempt

my ( $service ) _;my ( $service_name ) split( /_/, $service );print STDERR $date: $0: attn: changing priority for

service:$servicen;

update priority&db_query( update mcServiceRoute

.set priority ( select max( priority )

frommcServiceRoute

.where service $service_name) 1,.date getdate( ),.attempt attempt 1.where route $script $service );

print ---$routen; find new routemy $route &db_query( select route frommcServiceRoute

.where service $service name.and attempt < 5.order bypriority )

> 0 route ;

&db_query( update mcServiceRoute

.set attempt 0.where route $script $service )if ( $route eq $script $serviceor $route eq $script $service_stored ) ;

( $service_name, $service ) split( /s/, $route );die $date: $0: error: no route for the service:

$service (addmore) n

unless $service;return $service;

sub switch service

my ( $service ) _;my ( $service_name ) split( /_/, $service );print STDERR $date: $0: attn: changing priority for

service:$servicen;

update priority&db_query( update mcServiceRoute

.set priority ( select max( priority )

frommcServiceRoute

.where service $service_name) 1,.date getdate( ).where route $script $service );

print ---$routen;

find new routemy $route &db_query( select route from

mcServiceRoute

.where service

$service name

.and attempt < 5.order by

priority )

> 0 route ;

die $date: $0: error: there is the only service:

$route (addmore) n

if ( $route eq $script $serviceor $route eq $script $service_stored );

$service_name, $service ) split( /s/, $route );die $date: $0: error: no route for the service:

$service (addmore)n

unless $service;

return $service;

Table 4 below contains source code of the content fetcher 402 used with the content extraction agent 400 to retrieve information from a web site.

TABLE 4/usr/local/www/bin/sybper15-T -w $Header:/usr/local/cvsroot/webley/agents/service/webget.p1,v 1.4 Agent to get info from the web. Parameters: service_name service_parameters, i.e. stockmsft or weather60645 Configuration stored in files service_name.ini if this file is absent the configuration is received frommcServices table This script provides autoupdate to datatable if the .inifile is newer.$debug 1;use URI::URL;use LWP::UserAgent;use HTTP::Request::Common;use Vail::VarList;use Sybase::CTlib;use HTTP::Cookies;print Sybase::CTlib $DB_USR, $DB_PWD, $DB_SRV;;open( STDERR, >>$0.log ) if $debug; open( STDERR, >&STDOUT );$logdate; $response ./url.p1http://cgi.cnn.com/cgi-bin/weather/redirectzip 60605;$response pwd;print STDERR pwd$responsen;$response 1s;print STDERR 1s$responsen;chop( $log );$log . pwd .pwd;chop( $log );$debug2 1;my $service shift;$log . $service:. join( :, ARGV ) . n;print STDERR $iog if $debug;$response ./ur1.p1http://cgi.cnn.com/cgi-bin/weather/redirectzip60605;my ini &read_ini( $service );chop( ini );my $section ;do $section &process_section( $section ) while $section;$response ./ur1.p1http://cgi.cnn.com/cgi-bin/weather/redirectzip60605;exit;sub read_ini

my ( $service ) _;my ini ( ); first, try to read file$0 m{circumflex over ()}(.*/) {circumflex over ()}/*;$service $1 . $service;if ( open( INT, $service.ini ) )

ini (<INI>);return ini unless ( $DB_SRV ); update datatablemy $file_time time int( ( M $service.ini )

* 24 *3600 );print time $file_timen;

my $dbh new Sybase::CTlib $DB_USR, $DB_PWD,

$DB_SRV;

unless ( $dbh ) print STDERR webget.p1: Cannot connect to

dataserver $DB_SRV:$DB_USR:$DB_PWDn;

return ini;my row_refs $dbh>ct_sql( select lastUpdate

frommcServices where service $service, undef, 1 );

if ( $dbh>RC CS_FAIL ) print STDERR webget.p1: DB select from

mcServicesfailedn;

return ini;unless ( defined row_refs have to insertmy ( ini_escaped ) map ( my$x $_ ) s///g;$x;ini;$dbh>ct_sq1( insert mcServices values(

$service,ini_escaped, $file time ) );